My Map Has More Than 50 Million Nodes

Ollj · February 2003

<div class="IPBDescription">thats 154.000% of HL`s maximum, btw.</div> I get a werid error with my RAD (1.7p)
everything runs fine till it writes the final .bsp wich is (if you trust that report) 1.2 gig big and RAD crashes.
It takes less than 3 hours to compile.
the error is reproduceable.

Can someone try an explanation?
<span style='font-size:27pt;line-height:100%'><b>WTF?</b></span>

Ollj · February 2003

to show that the map is all fine (apart from 105% of max lightdata) i show the Vis sumup:

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>CODE</b> </td></tr><tr><td id='CODE'>
at the end of vis everything is fine.
here the vis report:
models 122/400 7808/25600 (30.5%)
planes 24166/65535 483320/1310700 (36.9%)
vertexes 12345/65535 148140/786420 (18.8%)
nodes 5317/32767 127608/786408 (16.2%)
texinfos 6487/32767 259480/1310680 (19.8%)
faces 9870/65535 197400/1310700 (15.1%)
clipnodes 26558/32767 212464/262136 (81.1%)
leaves 2846/8192 79688/229376 (34.7%)
marksurfaces 11989/65535 23978/131070 (18.3%)
surfedges 45276/512000 181104/2048000 ( 8.8%)
edges 22874/256000 91496/1024000 ( 8.9%)
texdata [variable] 3924/4194304 ( 0.1%)
lightdata [variable] 0/4194304 ( 0.0%)
visdata [variable] 98150/2097152 ( 4.7%)
entdata [variable] 11929/524288 ( 2.3%)</td></tr></table><span class='postcolor'>

it does not write a final bsp after the RAD. when i start hl after compile i play the fullbright .bsp without Vis ( = pre-.bps).

Greedo · February 2003

As far as I know, the node count shouldn't change afer CSG. Sounds to me like you're probably using a RAD command line switch incorrectly, or your hlrad.exe is corrupt.

j0e · February 2003

just remove 50,463,000 nodes <img src='http://www.unknownworlds.com/forums/html/emoticons/biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif'>

Lord_Requiem · February 2003

Corrupt hlrad or command line switches used incorrectly about sums it up, that would be the first thing I look at.

2nd thing I'd look at would be that i didnt have 1 or more brushes shattered from using carve... you naughty person!

Cagey · February 2003

Ollj, are you running opt_plns before hlrad? If you are, it may be the cause of the problem -- the optimizer should always be run after everything else since it strips some information from the map, and hlrad crashed my sample map after running it (I posted a warning to the compiler tools thread about this, but I may need to make it more prominent). If you're not running the optimizer first, I'm not sure what the problem could be, but I'll look into other possibilities since the change from 1.7 to 1.7p is my responsibility (although the change was so minor it shouldn't cause this corruption).

Cagey · February 2003

Did some quick looking at the source code... 105% of lighting capacity would have unpredictable results, possibly including the error you're seeing. The light data is stored in a static byte array -- if there are too many light samples, the code currently keeps writing past the end of the array... the buffer overflow may be corrupting your node data and causing the tools to think you have 50 million nodes. There ought to be a warning about this added to the compiler tools--it should stop the compiler and throw an error instead of making you wait almost 3 hours to find out you're over the limit.

The first way to combat this would be to lower the number of light samples--the command line allows setting several variables that can help with this. The other way to fix this is to simply increase the maximum allowed light data. I've recompiled the tools with a higher limit on the light size (4MB -> 6MB) since the in-source comments say the limit is arbitrary, but you'll need to playtest in software mode (preferably on several older machines) to make sure it's going to be compatible with everybody's machine.

Ollj -- If you want to try out the higher limit, PM me your email, I'll give you a URL to download and beta test the new build (1.7p2). I don't want to widely release tool code that may break the game.

EDIT: after looking at the code some more, I've found that the light data size is calculated to be (sum for all faces of ((# of samples on a face)*(# of light styles on the face)). If you're using a lot of dynamic lighting, especially where there's 2-3 different light styles per face, another way to fix the problem would be to reduce the number of light styles on faces -- reduce the number of dynamic lights in your map and it should drop you below the 100% mark without raising the limit.

gaggle · February 2003

Wow Ollj.. that's.. er.. interesting timing on your behalf. I have a big floating WTF! over my head, and I was heading to the forum to write pretty much exactly what you've written.

Yes, I too suffer suddenly from this.. ah.. strange bug.

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>CODE</b> </td></tr><tr><td id='CODE'>
Object names Objects/Maxobjs Memory / Maxmem Fullness
------------ --------------- --------------- --------
models 242/400 15488/25600 (60.5%)
planes 29788/65535 595760/1310700 (45.5%)
vertexes 20450/65535 245400/786420 (31.2%)
nodes -2054850689/32767 -2071776280/786408 (-6271098.0%)
texinfos 276/32767 11040/1310680 ( 0.8%)
faces 15111/65535 302220/1310700 (23.1%)
clipnodes 30028/32767 240224/262136 (91.6%)
leaves 5458/8192 152824/229376 (66.6%)
marksurfaces 19177/65535 38354/131070 (29.3%)
surfedges 69497/512000 277988/2048000 (13.6%)
edges 35566/256000 142264/1024000 (13.9%)
texdata [variable] 1104/4194304 ( 0.0%)
lightdata [variable] 4265670/4194304 (101.7%)
visdata [variable] 117112/2097152 ( 5.6%)
entdata [variable] 44994/524288 ( 8.6%)
25 textures referenced
=== Total BSP file data space used: -2065325838 bytes ===
Error: File read failure
</td></tr></table><span class='postcolor'>

I'm also, as can be seen by the planes maximum, using XP-Cagey's compilertools. I haven't run the optimizer.
For what it's worth, that's about 6 or 7 hours of compiling right out the window. Nothing bitter in that at this stage as it wasn't the focus of this compile, I'm just mentioning that in an attempt to win over the most sympathy. You know.. in case any naked ladies are around looking for someone to comfort.

.. but I digress.. <img src='http://www.natural-selection.org/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'>

My HLRAD settings this time were:
-sparse
-chop 32
-texchop 32
-coring 0.2
-notexscale
-bounce 4
-smooth 80

(one guy on this forum will now go "whoop, he's basically using my suggested settings")
I belive, but I didn't keep a keen track of it, that I've compiled with -chop 64 -texchop 64 as well (which essentially drops the compiletime for HLRAD to 30 or so minutes..), but it still crapped out on the nodes.
I'll give that a try right now just to make sure.

Oh I got the following error earlier on last night, but it seems to've gone away now, instead having been replaced with the nodes-error. I don't know if it's related at all. It was when Half-Life was loading the level, then bam, it crashed with an OK dialogbox saying "miptex >= loadmodel->numtextures".

I'm just mentioning that because of it's temporal proximity to the node-error, for now I suppose we just ignore it unless either others also experience it, or I can pinpoint a pattern. No point going after unreproduceable bugs <img src='http://www.natural-selection.org/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'> .. I wonder if it was with -chop and -texchop at 64 I got that error..

Anyway, I hope we can solve this one way or another.

Cagey · February 2003

gaggle -- The common element in your compiles seems to be going over the 100% mark (4 MB) on the light data -- I'm emailing you the beta link for the latest build that has a higher limit (6 MB).

gaggle · February 2003

Thanks XP, I'm firing up the compile now. First a -chop -texchop 64 one, then an ordinary -32 one.

I've just finished a compile where the only change was -choop and -texchop 64, -noopaque turned on, and -notexscale turned off. Surprisingly it compiled successfully, HLRAD didn't crash. Even more surprisingly, the stats now reports no nodes-useage at all.

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>CODE</b> </td></tr><tr><td id='CODE'>
Object names Objects/Maxobjs Memory / Maxmem Fullness
------------ --------------- --------------- --------
models 242/400 15488/25600 (60.5%)
planes 29788/65535 595760/1310700 (45.5%)
vertexes 20449/65535 245388/786420 (31.2%)
nodes 0/32767 0/786408 ( 0.0%)
texinfos 276/32767 11040/1310680 ( 0.8%)
faces 15111/65535 302220/1310700 (23.1%)
clipnodes 30029/32767 240232/262136 (91.6%)
leaves 5458/8192 152824/229376 (66.6%)
marksurfaces 19172/65535 38344/131070 (29.3%)
surfedges 69494/512000 277976/2048000 (13.6%)
edges 35565/256000 142260/1024000 (13.9%)
texdata [variable] 1104/4194304 ( 0.0%)
lightdata [variable] 4290381/4194304 (102.3%)
visdata [variable] 117093/2097152 ( 5.6%)
entdata [variable] 44994/524288 ( 8.6%)
25 textures referenced
=== Total BSP file data space used: 6475104 bytes ===
1421.00 seconds elapsed [23m 41s]
</td></tr></table><span class='postcolor'>

And maybe at this point it's less surprising that I'm getting the miptex>= error if I try and load the level.
Actually Half-Life crashes if I just run HL.exe from the commandline instructing it to load NS and the level. But if I start HL up per commandline, then do a "gamedir ns" switch in the console, and then loading my map, theeen it displays the miptex error dialogbox.

But that's not really interesting. The point is that while HLRAD didn't crash this time it still didn't create a valid map.
Here goes with the recompile with new toolset.

ImaTarget · February 2003

maybe a dumb question, but did you try to just delete some lights for a test?

Revenant · February 2003

when i get weird compiler erros i use two versions. if ur getting weird stuff on 1.7p then try using 1.6 (i dont use 1.7p btw). just use an older version than zhlt or the original hl tools <img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'> then report ur findings

gaggle · February 2003

Bad news so far, the compile finished successfully, although there is an oddity in the lightdata stats?...:

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>CODE</b> </td></tr><tr><td id='CODE'>
Object names Objects/Maxobjs Memory / Maxmem Fullness
------------ --------------- --------------- --------
models 0/400 0/25600 ( 0.0%)
planes 29788/65535 595760/1310700 (45.5%)
vertexes 0/65535 0/786420 ( 0.0%)
nodes 0/32767 0/786408 ( 0.0%)
texinfos 276/32767 11040/1310680 ( 0.8%)
faces 0/65535 0/1310700 ( 0.0%)
clipnodes 0/32767 0/262136 ( 0.0%)
leaves 0/8192 0/229376 ( 0.0%)
marksurfaces 0/65535 0/131070 ( 0.0%)
surfedges 0/512000 0/2048000 ( 0.0%)
edges 0/256000 0/1024000 ( 0.0%)
texdata [variable] 0/4194304 ( 0.0%)
lightdata [variable] 0/6291456 ( 0.0%)
visdata [variable] 0/2097152 ( 0.0%)
entdata [variable] 0/524288 ( 0.0%)
0 textures referenced
=== Total BSP file data space used: 606800 bytes ===
</td></tr></table><span class='postcolor'>

But the level does not load successfully. First time I loaded it (all this in windowed software-mode btw) I could walk for a few seconds before it crashed.
Subsequent attempts yielded instant crashing as the console slides up after having loaded the level.
No specific errors displayed, it quits with a default Windows General Protection Fault thing.

I'll do a full recompile tomorrow using the -chop -texchop 32 settings, just to make sure. I can't do it today before having to shut down the computer anyway.

In the mean time I'll try a quick compile with the light-entities removed, as ImaTarget suggests.. dumb question it's not I don't think.. it could just be that simple <img src='http://www.natural-selection.org/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'>
Although the 125 lights I have in the level so far would be a tiny amount compared to what I'll end up needing to properly light the whole thing...

XP, please advise if I should continue using the 1.7p2 <i>beta</i>-beta-build, or if I should switch back to 1.7p? I'll at least do the full compile with the p2 tools, but if there's anything else I can try just let me know.

Greedo · February 2003

Since the only commonality between the errors here seems to be Cagey's build of ZHLT, I would suggest simply going back to the last Merl's build, which doesn't have the artificially increased limits that may be causing these problems.

gaggle · February 2003

Greedo yeah, that's the test I'll run right after compiling without lightentities using XP's 1.7p build.

No lightentities with the 1.7p2 build yielded another compilable map that HL crashes on right after loading it.

Revenant · February 2003

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'>Since the only commonality between the errors here seems to be Cagey's build of ZHLT, I would suggest simply going back to the last Merl's build, which doesn't have the artificially increased limits that may be causing these problems. </td></tr></table><span class='postcolor'>

I already said that

Cagey · February 2003

$10 says the original problem with 1.7p is a buffer overflow on lightdata and Merl's 1.7 exhibits identical behavior since you're not over the old limits <img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'>... as a matter of fact, if you do a binary diff on the bsp files generated by each tool set, they ought to be identical (if planes < 32K) since I didn't change any algorithms between 1.7 and 1.7p, just upped a #define.

If 125 lights are causing over 4MB of light data to be generated, there's a problem.

I really need to go back to 1.7p2 now to see what I stepped on--sorry about that gaggle :\.

EDIT: /me prepares to eat crow if I'm wrong on this one.

gaggle · February 2003

I won't bother checking with Merl's original build (unless someone asks) as the level is working now. I've done a quick compile using XP's 1.7p, with all the lightentities removed (about 110 of them, the rest of the light comes from two texturelights). It compiles fine, the lightdata reports about.. yikes, 50% full? That's a little extreme.. I wonder what's going on with that..

Anyway, so that's working. Hence the thought about just removing light seems to've been a valid one indeed. I'll be prodding and pushing my level in the next hours, to see if I can figure out what a hell is making the lightdata limit being reached so fast.

Here are the log parts:
</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>CODE</b> </td></tr><tr><td id='CODE'>
VIS:
Object names Objects/Maxobjs Memory / Maxmem Fullness
------------ --------------- --------------- --------
models 242/400 15488/25600 (60.5%)
planes 29788/65535 595760/1310700 (45.5%)
vertexes 20453/65535 245436/786420 (31.2%)
nodes 9083/32767 217992/786408 (27.7%)
texinfos 276/32767 11040/1310680 ( 0.8%)
faces 15113/65535 302260/1310700 (23.1%)
clipnodes 30026/32767 240208/262136 (91.6%)
leaves 5460/8192 152880/229376 (66.7%)
marksurfaces 19177/65535 38354/131070 (29.3%)
surfedges 69507/512000 278028/2048000 (13.6%)
edges 35572/256000 142288/1024000 (13.9%)
texdata [variable] 1104/4194304 ( 0.0%)
lightdata [variable] 2309445/4194304 (55.1%)
visdata [variable] 117179/2097152 ( 5.6%)
entdata [variable] 36559/524288 ( 7.0%)
25 textures referenced
=== Total BSP file data space used: 4704021 bytes ===
1579.48 seconds elapsed [26m 19s]

HLRAD:
15113 faces
Create Patches : 87934 base patches
0 opaque faces
838323 square feet [120718616.00 square inches]
16 direct lights

Transfer Lists : 53186978 : 53.19M transfers
Indices : 31579116 : 30.12M bytes
Data : 212747912 : 202.89M bytes
</td></tr></table><span class='postcolor'>

I was thinking maybe someone would go "OOH MI GAWWDD that can't be!" at some of these numbers, as I really don't have anything to compare with (can't remember the various numbers from my last Half-Life level). At any rate, if there are some of these numbers that seems seriously out of whack feel free to scream about it.

PhilipK · February 2003

Hmm.. Might be d_scale settings?

Ollj · February 2003

-

watch_me_die · February 2003

gaggle when you ran the map and it crashes as soon as the console slides up, were you using sv_cheats 1?

I had a compile of my map that would run fine until i set sv_cheats, then when I slid the console back up it would crash ;/

gaggle · February 2003

Watch, yes I did. I'll see what happens if I don't do that. Just gotta compile compile first

My Map Has More Than 50 Million Nodes

Comments