Get The Latest Compile Tools

Cagey · February 2003

Specifically, it does a modified mark-and-sweep garbage collection of planes before writing the result back to the file.

In layman's terms, the algorithm works like this:

1) read the file into memory
2) make an array of integers that is the same size as the array of planes on the map
3) peek at every face on the map, and make a mark on the integer array that corresponds to the plane being used (all faces have a plane that they reference by index, so if face #928 uses plane #124, I mark the integer array at #124 to say that plane has been used). To keep track of which plane is which, I use an increasing count for each match (so if it's the 10th plane I've marked, it's #10 in the new ordering... if it's used again, which is common, it stays #10 in each use).
4) repeat step #3, but look at nodes instead of faces
5) repeat step #3, but look at clipnodes instead of faces
6) using the integer array as a reference, copy all of the used planes to temporary storage (in this step they are rearranged to match the order they were marked)
7) copy the sorted planes back to the planes array (now used planes take up the front of the array)
8) tell the tools that the number of planes on the map is actually the number of used planes, so it ignores the garbage at the back of the array
9) copy the map information back into the file

The garbage at the back can be ignored in step #8 due to the way the tools allocate memory--each array is created at startup to be the maximum possible size, and then they aren't ever resized. To keep track of actual data sizes, the tools use a second variable that keeps track of the count in the array. I don't need to resize the array, just fix the size variable.

Cagey · February 2003

I guess I answered, "what does it do" literally in the previous post, but here's why it makes a difference:

The hlcsg and hlbsp steps that are responsible for creating level geometry (planes, nodes, clipnodes) have a "fill" step where the outside of the world is stripped away from the node lists. This is one reason why maps that leak have significantly higher r_speeds even if you don't run vis on the leakproof copy. Many of the faces on each brush are also thrown out if they aren't part of the final visible hull--stuff like faces that are flush with another brush that is part of the worldspawn.

Although faces and nodes have always been removed by the tools, the planes that help to define them (by saying which way each face or node portal points) are never removed even if all faces and nodes that use the plane are removed. Many planes are artifacts of faces and nodes that were already removed by the tools. The extra information provided by the removed planes is required by the tools, so the optimizer shouldn't be run if you're going to do extra lighting work or compile with --onlyents to tweak your map, but the game itself only references the planes when looking up face and node information -- if there isn't a face or node that needs it, a plane is just taking up space when running the game.

Hopefully the explanation above combined with the literal "whatsitdo?" will give you enough background to consider using the optimizer.

FuzzDad · February 2003

Well...you are entirely too smart. <img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'> Thanks...I have a map in dod beta 4 and I think this helps in the bsp size reduction and r_speed arenas...though I'm not smart enough to understand why my r_speeds got a little better other than to think the engine was attempting to draw some of the planes pre-optimization that got removed after optimization that it didn;t need to draw anymore...but the map looks fine (no gaping holes...lol) and seems to run a bit smoother now so whatever intelligent magic was involved I appreciate it.

quazilin · February 2003

I have some doubt that this optimizer is so good. I compiled my map with those tools and dragged my bsp and it optimized it but I noticed that the planes were ok and r_speeds lower but I think the map lagged much more than ever before maybe FPS was dropped. or what I am doing wrong.

ImaTarget · February 2003

hmm should not happen. just test it a bit. compile the map without optimizing it, run around and note r_speeds and FPS, then optimize it and check again. that simple <img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'> if you can proof there is a problem all the better so cagey has the time to fix it.

Cadaver · February 2003

Hmmm, I just tried using the optimizer, having added a load of extra detail to a map that was already pushing the (original) planes limit; it compiled OK with the new compile tools, but when it came to optimizing, it crashed (Windows XP reported a problem and closed it down) :/

Here's the dialog box that came up, along with the compile/optimising info:

<img src='http://www.planethalflife.com/cadaver/images/origin/planes_crash.jpg' border='0' alt='user posted image'>

Any ideas about what could be wrong - could it be because the clipnodes limit is also perilously full? Or perhaps it's because of too many entities or something (I managed to run the map in NS, despite having gone over the regular planes limit without optimising, and it only seemed to be func_illusionary entities that were b0rky)? My map properties are here, in case it's any use:

<img src='http://www.planethalflife.com/cadaver/images/origin/map_properties.jpg' border='0' alt='user posted image'>

Any advice/help will be much appreciated, as I don't want to have to scrap my new biodome brushwork to get back below the planes limit :/

Jedediah · February 2003

I noticed that opt_plns.exe crashes if it can't find the file you specify.. might wanna check to make sure it's correct

Cadaver · February 2003

I'm doing it by dragging the .bsp file onto the optimizer program, and they're both in the same directory, so I don't think it's that.

WolfWings · February 2003

Windows XP can cause spurious crashes at random times with a LOT of command-line tools, so it may just be Windows XP trying to 'protect' your system from the utility. On the other hand, maybe the optimizer doesn't support larger than 32k arrays? *grins mischeviously* Would be amusing at the very least.

Cagey · February 2003

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (WolfWings @ Feb 4 2003, 06:47 PM)</td></tr><tr><td id='QUOTE'> On the other hand, maybe the optimizer doesn't support larger than 32k arrays? *grins mischeviously* Would be amusing at the very least.  </td></tr></table><span class='postcolor'> 
That would be pretty funny <img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'>. The hard limit for the optimizer is set to 64K planes, same as the 1.7p build of the tools... I just hardcoded the display to be relative to 32K since that's the point where the game starts acting strange.

Cadaver, would you be willing to let me use your bsp to try to duplicate the crash? PM me if you'd like me to check it over.

I'm on Win2000 here, so it could be a difference in the OS, but that wouldn't be my first inclination. The fact that the map information is displayed before the program dies means that it's reading the file from disk successfully. I'm guessing the crash is probably happening at either step 2 (see my earlier post in this thread for my description of the steps involved) when I'm attempting to allocate a large array to the stack or step 6 when I'm allocating a large chunk of additional memory. Since it's not displaying the optimized plane count, it's definately crashing before step 8.

If anybody else is having crash issues, I'd like to get as detailed a bug report as you can give... Just PM me the details and I'll see if I can track down the problems. I'll fix the missing file crash ASAP, and post v1.2 of opt_plns.exe here.

Cagey · February 2003

Version 1.2 of opt_plns.exe is now available, fixing the missing file crash and also fixing a bug which may cause data corruption at plane #0. This is a recommended upgrade for anybody who has downloaded a previous version.

EDIT: By missing file crash I was referring to the crash on bsp file not found. I should say what I mean more often <img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'> Apologies about any confusion, and as venomus notes in the next post, the .Net framework DLLs are still required.

<span style='color:red'>EDIT: this version has been superceded by version 1.4, which is available on the 5th page of this topic. Version 1.4 removes texdata, visdata, and lightdata limits and contains cumulative bugfixes from 1.2 and 1.3.</span>

venomus · February 2003

Hi, I'm still trying to get the tools work so I can resurrect an ancient map of mine, abandoned when it became too complex. I'll post more details on these problems later.

However, I'm just pointing out that although version 1.2 of opt_plns fixes the missing file crash, the modified tools still seem to require that file. If anyone is looking for the the two dlls needed and is running Windows XP, try looking in WINDOWS\Microsoft.NET\Framework\v1.0 . I just searched my hard drive for the files. Then copy the files (msvcr70.dll and msvcp70.dll) to your WINDOWS\system or WINDOWS\system32 folder (for me it was the latter).

venomus · February 2003

Also, this is just a minor inconvenience, opt_plns requires you to press a key after it finishes, which seems to screw up my batch files. At the moment the batch file looks something like this:

hlcsg.exe bigmap.map
hlbsp.exe bigmap.map
hlvis.exe bigmap.map
hlrad.exe bigmap.map
opt_plns.exe bigmap.bsp
copy bigmap.bsp C:\HL\valve\maps
C:\HL\hl.exe +map bigmap.bsp

The sequence stops after I have to press the key. Can the batch be scripted to avoid this, or the program changed to have the key pressing optional or something?

Cagey · February 2003

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (venomus @ Feb 5 2003, 08:04 AM)</td></tr><tr><td id='QUOTE'> The sequence stops after I have to press the key. Can the batch be scripted to avoid this, or the program changed to have the key pressing optional or something?  </td></tr></table><span class='postcolor'> 
I'll add a command line switch to make the key press optional in the next public release (which should happen very soon, possibly later today. I've asked Cadaver to double-check a bugfix for his problem before I make it public).

venomus · February 2003

OK, turns out the map I was trying to resurrect also had too many clipnodes, but for a variety of reasons this was hard to detect. Still I have noticed as much as 4/5 planes were culled on some other maps I tried the tool on, and clipnodes can be economised relatively easily. These tools are excellent <img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' style='vertical-align:middle' alt='smile.gif'> .

Cagey · February 2003

Version 1.3 is now available... this version fixes a crash bug (discovered by Cadaver) that may occur on maps that are over the original 32K plane limit, and also introduces a command line switch, -nopause, that turns off the keypress prompt when the program finishes. The -nopause option was requested for using opt_plns in a batch script.

If you haven't had any problems with 1.2, you can wait before upgrading. People using 1.1 should upgrade to 1.3 to fix the plane #0 bug corrected in 1.2.

<span style='color:red'>EDIT: this version has been superceded by version 1.4, which is available on the 5th page of this topic. Version 1.4 removes texdata, visdata, and lightdata limits and contains cumulative bugfixes from 1.2 and 1.3. </span>

LumpN · February 2003

i found that there is an error when trying to compile a map with -subdivide 256. so i coded this tool which fixes this problem. just
run it before the compile process starts like "fixmap mymap" and there wont be any errors

now you can truely reduce your w_poly a lot as you can see here:

<a href='http://www.jobvdsande.com/~lumpn/de/tutorials/speeds.htm' target='_blank'>http://www.jobvdsande.com/~lumpn/de/tutori...ials/speeds.htm</a>

gnah i know it's german, sorry folks but the text just sais nothing else than i did here, just look at the images

/me lumps you!

Revenant · February 2003

hmm, seems like there is gonna get lots of tools soon, maybe code LumpNs program into the end of XP-Cageys optplns.exe(If thats the right exe name)?

venomus · February 2003

One more request (thanks for implementing the last one so quickly). Could you make it so the opt_plns output can be appended to mapname.log?

LumpN · February 2003

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (Revenant @ Feb 6 2003, 06:38 AM)</td></tr><tr><td id='QUOTE'> maybe code LumpNs program into the end of XP-Cageys optplns.exe  </td></tr></table><span class='postcolor'> 
sorry that wont work, it has to be coded into hlcsg.exe because you have to use it BEFORE compiling. i'll ask merl about that

Klamath · February 2003

Could anyone be more specific about how much this reduced r_speeds on their map? It sounds like it might be worthwhile even if you aren't having a max planes problem.

Revenant · February 2003

KLAMATH: it depends on the amount of planes in your maps, and in areas in particular.

[EDIT] Personally I am not using these tools for ns_fusion. The reason is I will not go over the planes limit and the fact that its still buggy. I am most likely to use it for my next map, as this is a larger map and is likely to exceed the planes limit [EDIT]

WolfWings · February 2003

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (LumpN @ Feb 5 2003, 03:31 PM)</td></tr><tr><td id='QUOTE'> i found that there is an error when trying to compile a map with -subdivide 256. so i coded this tool which fixes this problem. just
run it before the compile process starts like "fixmap mymap" and there wont be any errors

now you can truely reduce your w_poly a lot as you can see here:

<a href='http://www.jobvdsande.com/~lumpn/de/tutorials/speeds.htm' target='_blank'>http://www.jobvdsande.com/~lumpn/de/tutori...ials/speeds.htm</a>

gnah i know it's german, sorry folks but the text just sais nothing else than i did here, just look at the images

/me lumps you!  </td></tr></table><span class='postcolor'> 
Using the newer <b>-subdivide</b> option breaks software mode.

Please do not use this feature for NS maps, as the Developers, and Flayra, have stated repeatedly that they wish for NS to be supported in Software mode again, once Valve is able to fix bugs in the HL codebase to allow this.

Using the <b>-subdivide</b> option available in the newer MHLT set breaks this, though it allows for on average a 2.1x-4.5x (peak 3-9x) reduction in overall r_speeds on large surfaces in OpenGL mode at it's maximum value of 512. Yes, as much as a 9x reduction in a best-case scenario where a 512x512 square goes from 18 polygons to 2 polygons, while a 513x513 square goes from 18 to 8, etc, etc.

It does this by breaking the 'rule' of flat walls being broken every 240 units if longer than 256 units. Instead, they are broken every X units if longer then 256 units, with X defaulting to 240 and being changeable with the <b>-subdivide</b> option. This is incompatable with software mode, much like software mode cannot handle more than 4 megabytes of textures, while OpenGL or DirectX mode have no such limitations.

LumpN · February 2003

</span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (WolfWings @ Feb 7 2003, 12:47 PM)</td></tr><tr><td id='QUOTE'> Using the newer <b>-subdivide</b> option breaks software mode.

Please do not use this feature for NS maps, as the Developers, and Flayra, have stated repeatedly that they wish for NS to be supported in Software mode again, once Valve is able to fix bugs in the HL codebase to allow this.

This is incompatable with software mode, much like software mode cannot handle more than 4 megabytes of textures, while OpenGL or DirectX mode have no such limitations.  </td></tr></table><span class='postcolor'> 
well, i'm not really sure about it, but i think you are wrong, we tested maps compiled with 256 on dedicated server, server, opengl, d3d and software. there werent <b>any</b> problems.

maybe an error occurs when using an old 4mb voodoo card with 3dfx mini-gl drivers but that's only speculation too.

could you test it please? afaik there are no fatal errors anymore

Ollj · February 2003

if you have this +maxplanes tool and you know how to map with that detail you dond need to subdivide.
If you use subdivide on this tool, you stupid.

Axehandler · February 2003

Here is how I have My batch file Layed out for NS. Includes Cagy's opt_plns.exe tool for final output.
This is done under Win 2000. By all means not a COMPLETE Batch... just simple choice based Batchfile.
and could really make it alot smaller <img src='http://www.unknownworlds.com/forums/html/emoticons/wink.gif' border='0' style='vertical-align:middle' alt='wink.gif'>

%'s are output from WorldCraft. to the bathfile. <img src='http://www.unknownworlds.com/forums/html/emoticons/biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif'> <img src='http://www.unknownworlds.com/forums/html/emoticons/biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif'>

Call the Bathfile instead of the tools. still setup the tools thou.

have these for Param's in Wolrdcraft/hammer

$path\$file $bspdir $file

%1 = $path\$file
%2 = $bspdir
%3 = $file
FileName : Compile-HLNS.Bat

----------
@Echo off
ECHO %1
ECHO
ECHO 1. Fast Compile (Fastest)
ECHO 2. Fast Vis No Rad (Fast)
ECHO 3. Fast Vis With Rad (Slow)
ECHO 4. FULL Compile (Slowest)
ECHO 5. Quit
@CHOICE /C:12345 /N /T:2,5 Please choose a menu option.
@IF ERRORLEVEL == 5 GOTO finished
@IF ERRORLEVEL == 4 GOTO Slowest
@IF ERRORLEVEL == 3 GOTO slow
@IF ERRORLEVEL == 2 GOTO fast
@IF ERRORLEVEL == 1 GOTO fastest

:slowest
d:
E:\Mapping\ZHLT\hlcsg.exe -high %1
E:\Mapping\ZHLT\hlbsp.exe -high -chart %1
E:\Mapping\ZHLT\hlvis.exe -high -full %1
E:\Mapping\ZHLT\hlrad.exe -bounce 10 -high -sparse -extra %1
GOTO finished

:slow
d:
E:\Mapping\ZHLT\hlcsg.exe -high %1
E:\Mapping\ZHLT\hlbsp.exe -high -chart %1
E:\Mapping\ZHLT\hlvis.exe -high -fast %1
E:\Mapping\ZHLT\hlrad.exe -bounce 10 -high -sparse %1
GOTO finished

:fast
d:
E:\Mapping\ZHLT\hlcsg.exe -high %1
E:\Mapping\ZHLT\hlbsp.exe -high -chart %1
E:\Mapping\ZHLT\hlvis.exe -high -fast %1
GOTO finished

:fastest
d:
E:\Mapping\ZHLT\hlcsg.exe -high %1
E:\Mapping\ZHLT\hlbsp.exe -high -chart %1
GOTO finished

:finished
E:\Mapping\ZHLT\opt_plns %1.bsp
copy %1.bsp %2\%3.bsp
copy %1.pts %2\%3.pts
ECHO *** FINISHED ***
Pause

------

Hope that helps someone <img src='http://www.unknownworlds.com/forums/html/emoticons/wink.gif' border='0' style='vertical-align:middle' alt='wink.gif'>

Axe

Token · February 2003

Very cool exelent work!!!...I just finnally got the compile to run I was missing A .dll Other than the one mentioned (MSVCP70.dll) was teh one i was missing. I was cofused for a bit <img src='http://www.unknownworlds.com/forums/html/emoticons/wink.gif' border='0' style='vertical-align:middle' alt='wink.gif'>

Got it here>>>> <a href='http://www.dll-files.com/dllindex/dll-files.shtml?msvcp70' target='_blank'>http://www.dll-files.com/dllindex/dll-file...s.shtml?msvcp70</a>

Im using your optimizer on my Dod_isle map <img src='http://www.unknownworlds.com/forums/html/emoticons/biggrin.gif' border='0' style='vertical-align:middle' alt='biggrin.gif'>

builderhicks · February 2003

OK call me a **** but hey we all have to start somewhere.
Right, dl'd the tools, optimizer and DLL put the tools in place of my hlcsg and hlrad hit the run button in wc then hit go with the first 2 boxes checked (i just want to c the hlcsg work) hit the ok button and nothing happens <img src='http://www.unknownworlds.com/forums/html/emoticons/confused.gif' border='0' style='vertical-align:middle' alt='confused.gif'><img src='http://www.unknownworlds.com/forums/html/emoticons/confused.gif' border='0' style='vertical-align:middle' alt='confused.gif'> am i doing something wrong.
Ive compiled loads of times and never had this.
<img src='http://www.unknownworlds.com/forums/html/emoticons/sad.gif' border='0' style='vertical-align:middle' alt='sad.gif'>

** Executing...
** Command: Change Directory
** Parameters: "C:\Sierra\Half-Life"

** Executing...
** Command: C:\VALVEH~1\hlcsg.exe
** Parameters: "c:\valve hammer editor\maps\alpine5" -estimate -wadinclude #######

this is as far as it gets. I've dl'd the custom 2.5.3 1.7 tools but nothing
It dont even crash it just stops as tho its finished

builderhicks · February 2003

Oh and another thing i got a MAX MIPTEX error is this to do with the size of my wad or the size/amounts of the textures in the map?<img src='http://www.unknownworlds.com/forums/html/emoticons/confused.gif' border='0' style='vertical-align:middle' alt='confused.gif'>
I know it dont have anything to do with MAX MAP_PLANES but i got the error after i deleted some detail to get rid of the MAX_PLANES error i use zoners tools (as u can tell DOH!) how does the -texdata work to put the 4mg memory up to 8mg.
ty

Token · February 2003

just a quick question ... is the new zoners 1.7 .CSG compatable with the null tex(merles_custom_tools)? im just trying to troubleshoot, BSP keeps crashing with the modified tools..when i use my origianl tools i get max planes error. I am going to mess around with my map delete some of my most recent additions see if it helps.... any advice welcome

Get The Latest Compile Tools

Comments