Get The Latest Compile Tools

2456720

Comments

  • CageyCagey Ex-Unknown Worlds Programmer Join Date: 2002-11-15 Member: 8829Members, Retired Developer, NS1 Playtester, Constellation Posts: 1,751
    edited February 2003
    Specifically, it does a modified mark-and-sweep garbage collection of planes before writing the result back to the file.

    In layman's terms, the algorithm works like this:

    1) read the file into memory
    2) make an array of integers that is the same size as the array of planes on the map
    3) peek at every face on the map, and make a mark on the integer array that corresponds to the plane being used (all faces have a plane that they reference by index, so if face #928 uses plane #124, I mark the integer array at #124 to say that plane has been used). To keep track of which plane is which, I use an increasing count for each match (so if it's the 10th plane I've marked, it's #10 in the new ordering... if it's used again, which is common, it stays #10 in each use).
    4) repeat step #3, but look at nodes instead of faces
    5) repeat step #3, but look at clipnodes instead of faces
    6) using the integer array as a reference, copy all of the used planes to temporary storage (in this step they are rearranged to match the order they were marked)
    7) copy the sorted planes back to the planes array (now used planes take up the front of the array)
    8) tell the tools that the number of planes on the map is actually the number of used planes, so it ignores the garbage at the back of the array
    9) copy the map information back into the file

    The garbage at the back can be ignored in step #8 due to the way the tools allocate memory--each array is created at startup to be the maximum possible size, and then they aren't ever resized. To keep track of actual data sizes, the tools use a second variable that keeps track of the count in the array. I don't need to resize the array, just fix the size variable.
    XP-Cagey

    Recommended Reading: NS Mapping Guidelines | Mapping Forum FAQ
    Nostalgia: Power Cells Thread
  • CageyCagey Ex-Unknown Worlds Programmer Join Date: 2002-11-15 Member: 8829Members, Retired Developer, NS1 Playtester, Constellation Posts: 1,751
    edited February 2003
    I guess I answered, "what does it do" literally in the previous post, but here's why it makes a difference:

    The hlcsg and hlbsp steps that are responsible for creating level geometry (planes, nodes, clipnodes) have a "fill" step where the outside of the world is stripped away from the node lists. This is one reason why maps that leak have significantly higher r_speeds even if you don't run vis on the leakproof copy. Many of the faces on each brush are also thrown out if they aren't part of the final visible hull--stuff like faces that are flush with another brush that is part of the worldspawn.

    Although faces and nodes have always been removed by the tools, the planes that help to define them (by saying which way each face or node portal points) are never removed even if all faces and nodes that use the plane are removed. Many planes are artifacts of faces and nodes that were already removed by the tools. The extra information provided by the removed planes is required by the tools, so the optimizer shouldn't be run if you're going to do extra lighting work or compile with --onlyents to tweak your map, but the game itself only references the planes when looking up face and node information -- if there isn't a face or node that needs it, a plane is just taking up space when running the game.

    Hopefully the explanation above combined with the literal "whatsitdo?" will give you enough background to consider using the optimizer.
    XP-Cagey

    Recommended Reading: NS Mapping Guidelines | Mapping Forum FAQ
    Nostalgia: Power Cells Thread
  • FuzzDadFuzzDad Join Date: 2003-02-03 Member: 13052Members Posts: 2
    Well...you are entirely too smart. smile.gif Thanks...I have a map in dod beta 4 and I think this helps in the bsp size reduction and r_speed arenas...though I'm not smart enough to understand why my r_speeds got a little better other than to think the engine was attempting to draw some of the planes pre-optimization that got removed after optimization that it didn;t need to draw anymore...but the map looks fine (no gaping holes...lol) and seems to run a bit smoother now so whatever intelligent magic was involved I appreciate it.
  • quazilinquazilin Join Date: 2002-11-25 Member: 9880Members, Contributor, NS2 Playtester, Squad Five Blue Posts: 383 Fully active user
    I have some doubt that this optimizer is so good. I compiled my map with those tools and dragged my bsp and it optimized it but I noticed that the planes were ok and r_speeds lower but I think the map lagged much more than ever before maybe FPS was dropped. or what I am doing wrong.
  • ImaTargetImaTarget Join Date: 2002-11-01 Member: 3415Members Posts: 192
    hmm should not happen. just test it a bit. compile the map without optimizing it, run around and note r_speeds and FPS, then optimize it and check again. that simple smile.gif if you can proof there is a problem all the better so cagey has the time to fix it.
    Teamplay is essential, it gives them someone else to shoot at!
  • CadaverCadaver Creator of original ns_origin Join Date: 2002-04-17 Member: 466Members, NS1 Playtester, Contributor Posts: 66 Fully active user
    Hmmm, I just tried using the optimizer, having added a load of extra detail to a map that was already pushing the (original) planes limit; it compiled OK with the new compile tools, but when it came to optimizing, it crashed (Windows XP reported a problem and closed it down) :/

    Here's the dialog box that came up, along with the compile/optimising info:

    user posted image

    Any ideas about what could be wrong - could it be because the clipnodes limit is also perilously full? Or perhaps it's because of too many entities or something (I managed to run the map in NS, despite having gone over the regular planes limit without optimising, and it only seemed to be func_illusionary entities that were b0rky)? My map properties are here, in case it's any use:

    user posted image

    Any advice/help will be much appreciated, as I don't want to have to scrap my new biodome brushwork to get back below the planes limit :/
  • JedediahJedediah Join Date: 2003-01-27 Member: 12847Members Posts: 94
    I noticed that opt_plns.exe crashes if it can't find the file you specify.. might wanna check to make sure it's correct
  • CadaverCadaver Creator of original ns_origin Join Date: 2002-04-17 Member: 466Members, NS1 Playtester, Contributor Posts: 66 Fully active user
    I'm doing it by dragging the .bsp file onto the optimizer program, and they're both in the same directory, so I don't think it's that.
  • WolfWingsWolfWings NS_Nancy Resurrectionist Join Date: 2002-11-02 Member: 4416Members Posts: 595
    Windows XP can cause spurious crashes at random times with a LOT of command-line tools, so it may just be Windows XP trying to 'protect' your system from the utility. On the other hand, maybe the optimizer doesn't support larger than 32k arrays? *grins mischeviously* Would be amusing at the very least.
    user posted image
  • CageyCagey Ex-Unknown Worlds Programmer Join Date: 2002-11-15 Member: 8829Members, Retired Developer, NS1 Playtester, Constellation Posts: 1,751
    QUOTE (WolfWings @ Feb 4 2003, 06:47 PM)
    On the other hand, maybe the optimizer doesn't support larger than 32k arrays? *grins mischeviously* Would be amusing at the very least.

    That would be pretty funny smile.gif. The hard limit for the optimizer is set to 64K planes, same as the 1.7p build of the tools... I just hardcoded the display to be relative to 32K since that's the point where the game starts acting strange.

    Cadaver, would you be willing to let me use your bsp to try to duplicate the crash? PM me if you'd like me to check it over.

    I'm on Win2000 here, so it could be a difference in the OS, but that wouldn't be my first inclination. The fact that the map information is displayed before the program dies means that it's reading the file from disk successfully. I'm guessing the crash is probably happening at either step 2 (see my earlier post in this thread for my description of the steps involved) when I'm attempting to allocate a large array to the stack or step 6 when I'm allocating a large chunk of additional memory. Since it's not displaying the optimized plane count, it's definately crashing before step 8.

    If anybody else is having crash issues, I'd like to get as detailed a bug report as you can give... Just PM me the details and I'll see if I can track down the problems. I'll fix the missing file crash ASAP, and post v1.2 of opt_plns.exe here.
    XP-Cagey

    Recommended Reading: NS Mapping Guidelines | Mapping Forum FAQ
    Nostalgia: Power Cells Thread
  • CageyCagey Ex-Unknown Worlds Programmer Join Date: 2002-11-15 Member: 8829Members, Retired Developer, NS1 Playtester, Constellation Posts: 1,751
    edited February 2003
    Version 1.2 of opt_plns.exe is now available, fixing the missing file crash and also fixing a bug which may cause data corruption at plane #0. This is a recommended upgrade for anybody who has downloaded a previous version.

    EDIT: By missing file crash I was referring to the crash on bsp file not found. I should say what I mean more often smile.gif Apologies about any confusion, and as venomus notes in the next post, the .Net framework DLLs are still required.

    EDIT: this version has been superceded by version 1.4, which is available on the 5th page of this topic. Version 1.4 removes texdata, visdata, and lightdata limits and contains cumulative bugfixes from 1.2 and 1.3.
    Post edited by Unknown User on
    XP-Cagey

    Recommended Reading: NS Mapping Guidelines | Mapping Forum FAQ
    Nostalgia: Power Cells Thread
  • venomusvenomus Join Date: 2002-11-16 Member: 8951Members Posts: 353
    Hi, I'm still trying to get the tools work so I can resurrect an ancient map of mine, abandoned when it became too complex. I'll post more details on these problems later.

    However, I'm just pointing out that although version 1.2 of opt_plns fixes the missing file crash, the modified tools still seem to require that file. If anyone is looking for the the two dlls needed and is running Windows XP, try looking in WINDOWS\Microsoft.NET\Framework\v1.0 . I just searched my hard drive for the files. Then copy the files (msvcr70.dll and msvcp70.dll) to your WINDOWS\system or WINDOWS\system32 folder (for me it was the latter).
  • venomusvenomus Join Date: 2002-11-16 Member: 8951Members Posts: 353
    Also, this is just a minor inconvenience, opt_plns requires you to press a key after it finishes, which seems to screw up my batch files. At the moment the batch file looks something like this:

    hlcsg.exe bigmap.map
    hlbsp.exe bigmap.map
    hlvis.exe bigmap.map
    hlrad.exe bigmap.map
    opt_plns.exe bigmap.bsp
    copy bigmap.bsp C:\HL\valve\maps
    C:\HL\hl.exe +map bigmap.bsp

    The sequence stops after I have to press the key. Can the batch be scripted to avoid this, or the program changed to have the key pressing optional or something?
  • CageyCagey Ex-Unknown Worlds Programmer Join Date: 2002-11-15 Member: 8829Members, Retired Developer, NS1 Playtester, Constellation Posts: 1,751
    QUOTE (venomus @ Feb 5 2003, 08:04 AM)
    The sequence stops after I have to press the key. Can the batch be scripted to avoid this, or the program changed to have the key pressing optional or something?

    I'll add a command line switch to make the key press optional in the next public release (which should happen very soon, possibly later today. I've asked Cadaver to double-check a bugfix for his problem before I make it public).
    XP-Cagey

    Recommended Reading: NS Mapping Guidelines | Mapping Forum FAQ
    Nostalgia: Power Cells Thread
  • venomusvenomus Join Date: 2002-11-16 Member: 8951Members Posts: 353
    OK, turns out the map I was trying to resurrect also had too many clipnodes, but for a variety of reasons this was hard to detect. Still I have noticed as much as 4/5 planes were culled on some other maps I tried the tool on, and clipnodes can be economised relatively easily. These tools are excellent smile.gif .
  • CageyCagey Ex-Unknown Worlds Programmer Join Date: 2002-11-15 Member: 8829Members, Retired Developer, NS1 Playtester, Constellation Posts: 1,751
    edited February 2003
    Version 1.3 is now available... this version fixes a crash bug (discovered by Cadaver) that may occur on maps that are over the original 32K plane limit, and also introduces a command line switch, -nopause, that turns off the keypress prompt when the program finishes. The -nopause option was requested for using opt_plns in a batch script.

    If you haven't had any problems with 1.2, you can wait before upgrading. People using 1.1 should upgrade to 1.3 to fix the plane #0 bug corrected in 1.2.

    EDIT: this version has been superceded by version 1.4, which is available on the 5th page of this topic. Version 1.4 removes texdata, visdata, and lightdata limits and contains cumulative bugfixes from 1.2 and 1.3.
    Post edited by Unknown User on
    XP-Cagey

    Recommended Reading: NS Mapping Guidelines | Mapping Forum FAQ
    Nostalgia: Power Cells Thread
  • LumpNLumpN Join Date: 2002-10-30 Member: 1725Members, Subnautica Developer Posts: 131 Advanced user
    i found that there is an error when trying to compile a map with -subdivide 256. so i coded this tool which fixes this problem. just
    run it before the compile process starts like "fixmap mymap" and there wont be any errors

    now you can truely reduce your w_poly a lot as you can see here:

    http://www.jobvdsande.com/~lumpn/de/tutori...ials/speeds.htm

    gnah i know it's german, sorry folks but the text just sais nothing else than i did here, just look at the images

    /me lumps you!
  • RevenantRevenant Join Date: 2003-01-13 Member: 12249Members Posts: 444
    hmm, seems like there is gonna get lots of tools soon, maybe code LumpNs program into the end of XP-Cageys optplns.exe(If thats the right exe name)?
    . Revenant / KillerKungFuFerrit

    Contributing Level Designer For Morbid Inc. and Vampire Slayer Modifications
  • venomusvenomus Join Date: 2002-11-16 Member: 8951Members Posts: 353
    One more request (thanks for implementing the last one so quickly). Could you make it so the opt_plns output can be appended to mapname.log?
  • LumpNLumpN Join Date: 2002-10-30 Member: 1725Members, Subnautica Developer Posts: 131 Advanced user
    QUOTE (Revenant @ Feb 6 2003, 06:38 AM)
    maybe code LumpNs program into the end of XP-Cageys optplns.exe

    sorry that wont work, it has to be coded into hlcsg.exe because you have to use it BEFORE compiling. i'll ask merl about that
  • KlamathKlamath Join Date: 2003-02-06 Member: 13186Members Posts: 1
    edited February 2003
    Could anyone be more specific about how much this reduced r_speeds on their map? It sounds like it might be worthwhile even if you aren't having a max planes problem.
  • RevenantRevenant Join Date: 2003-01-13 Member: 12249Members Posts: 444
    edited February 2003
    KLAMATH: it depends on the amount of planes in your maps, and in areas in particular.

    [EDIT] Personally I am not using these tools for ns_fusion. The reason is I will not go over the planes limit and the fact that its still buggy. I am most likely to use it for my next map, as this is a larger map and is likely to exceed the planes limit [EDIT]
    . Revenant / KillerKungFuFerrit

    Contributing Level Designer For Morbid Inc. and Vampire Slayer Modifications
  • WolfWingsWolfWings NS_Nancy Resurrectionist Join Date: 2002-11-02 Member: 4416Members Posts: 595
    QUOTE (LumpN @ Feb 5 2003, 03:31 PM)
    i found that there is an error when trying to compile a map with -subdivide 256. so i coded this tool which fixes this problem. just
    run it before the compile process starts like "fixmap mymap" and there wont be any errors

    now you can truely reduce your w_poly a lot as you can see here:

    http://www.jobvdsande.com/~lumpn/de/tutori...ials/speeds.htm

    gnah i know it's german, sorry folks but the text just sais nothing else than i did here, just look at the images

    /me lumps you!

    Using the newer -subdivide option breaks software mode.

    Please do not use this feature for NS maps, as the Developers, and Flayra, have stated repeatedly that they wish for NS to be supported in Software mode again, once Valve is able to fix bugs in the HL codebase to allow this.

    Using the -subdivide option available in the newer MHLT set breaks this, though it allows for on average a 2.1x-4.5x (peak 3-9x) reduction in overall r_speeds on large surfaces in OpenGL mode at it's maximum value of 512. Yes, as much as a 9x reduction in a best-case scenario where a 512x512 square goes from 18 polygons to 2 polygons, while a 513x513 square goes from 18 to 8, etc, etc.

    It does this by breaking the 'rule' of flat walls being broken every 240 units if longer than 256 units. Instead, they are broken every X units if longer then 256 units, with X defaulting to 240 and being changeable with the -subdivide option. This is incompatable with software mode, much like software mode cannot handle more than 4 megabytes of textures, while OpenGL or DirectX mode have no such limitations.
    user posted image
  • LumpNLumpN Join Date: 2002-10-30 Member: 1725Members, Subnautica Developer Posts: 131 Advanced user
    QUOTE (WolfWings @ Feb 7 2003, 12:47 PM)
    Using the newer -subdivide option breaks software mode.

    Please do not use this feature for NS maps, as the Developers, and Flayra, have stated repeatedly that they wish for NS to be supported in Software mode again, once Valve is able to fix bugs in the HL codebase to allow this.

    This is incompatable with software mode, much like software mode cannot handle more than 4 megabytes of textures, while OpenGL or DirectX mode have no such limitations.

    well, i'm not really sure about it, but i think you are wrong, we tested maps compiled with 256 on dedicated server, server, opengl, d3d and software. there werent any problems.

    maybe an error occurs when using an old 4mb voodoo card with 3dfx mini-gl drivers but that's only speculation too.

    could you test it please? afaik there are no fatal errors anymore
  • OlljOllj our themepark-stalking nightmare Fade Join Date: 2002-12-12 Member: 10696Members Posts: 3,844
    if you have this +maxplanes tool and you know how to map with that detail you dond need to subdivide.
    If you use subdivide on this tool, you stupid.
    "What are we going to do tonight, Brain?"
    "The same thing we do every night, Pinky, trying to launch Steam."

    I like:
    user posted image
  • AxehandlerAxehandler Join Date: 2003-01-08 Member: 12038Members Posts: 50
    Here is how I have My batch file Layed out for NS. Includes Cagy's opt_plns.exe tool for final output.
    This is done under Win 2000. By all means not a COMPLETE Batch... just simple choice based Batchfile.
    and could really make it alot smaller wink.gif

    %'s are output from WorldCraft. to the bathfile. biggrin.gif biggrin.gif

    Call the Bathfile instead of the tools. still setup the tools thou.

    have these for Param's in Wolrdcraft/hammer

    $path\$file $bspdir $file

    %1 = $path\$file
    %2 = $bspdir
    %3 = $file
    FileName : Compile-HLNS.Bat



    @Echo off
    ECHO %1
    ECHO
    ECHO 1. Fast Compile (Fastest)
    ECHO 2. Fast Vis No Rad (Fast)
    ECHO 3. Fast Vis With Rad (Slow)
    ECHO 4. FULL Compile (Slowest)
    ECHO 5. Quit
    @CHOICE /C:12345 /N /T:2,5 Please choose a menu option.
    @IF ERRORLEVEL == 5 GOTO finished
    @IF ERRORLEVEL == 4 GOTO Slowest
    @IF ERRORLEVEL == 3 GOTO slow
    @IF ERRORLEVEL == 2 GOTO fast
    @IF ERRORLEVEL == 1 GOTO fastest

    :slowest
    d:
    E:\Mapping\ZHLT\hlcsg.exe -high %1
    E:\Mapping\ZHLT\hlbsp.exe -high -chart %1
    E:\Mapping\ZHLT\hlvis.exe -high -full %1
    E:\Mapping\ZHLT\hlrad.exe -bounce 10 -high -sparse -extra %1
    GOTO finished

    :slow
    d:
    E:\Mapping\ZHLT\hlcsg.exe -high %1
    E:\Mapping\ZHLT\hlbsp.exe -high -chart %1
    E:\Mapping\ZHLT\hlvis.exe -high -fast %1
    E:\Mapping\ZHLT\hlrad.exe -bounce 10 -high -sparse %1
    GOTO finished

    :fast
    d:
    E:\Mapping\ZHLT\hlcsg.exe -high %1
    E:\Mapping\ZHLT\hlbsp.exe -high -chart %1
    E:\Mapping\ZHLT\hlvis.exe -high -fast %1
    GOTO finished

    :fastest
    d:
    E:\Mapping\ZHLT\hlcsg.exe -high %1
    E:\Mapping\ZHLT\hlbsp.exe -high -chart %1
    GOTO finished

    :finished
    E:\Mapping\ZHLT\opt_plns %1.bsp
    copy %1.bsp %2\%3.bsp
    copy %1.pts %2\%3.pts
    ECHO *** FINISHED ***
    Pause




    Hope that helps someone wink.gif

    Axe
  • TokenToken Join Date: 2003-02-08 Member: 13262Members Posts: 2
    edited February 2003
    Very cool exelent work!!!...I just finnally got the compile to run I was missing A .dll Other than the one mentioned (MSVCP70.dll) was teh one i was missing. I was cofused for a bit wink.gif

    Got it here>>>> http://www.dll-files.com/dllindex/dll-file...s.shtml?msvcp70

    Im using your optimizer on my Dod_isle map biggrin.gif
  • builderhicksbuilderhicks Join Date: 2003-02-08 Member: 13263Members Posts: 7
    edited February 2003
    OK call me a **** but hey we all have to start somewhere.
    Right, dl'd the tools, optimizer and DLL put the tools in place of my hlcsg and hlrad hit the run button in wc then hit go with the first 2 boxes checked (i just want to c the hlcsg work) hit the ok button and nothing happens confused.gifconfused.gif am i doing something wrong.
    Ive compiled loads of times and never had this.
    sad.gif

    ** Executing...
    ** Command: Change Directory
    ** Parameters: "C:\Sierra\Half-Life"


    ** Executing...
    ** Command: C:\VALVEH~1\hlcsg.exe
    ** Parameters: "c:\valve hammer editor\maps\alpine5" -estimate -wadinclude #######

    this is as far as it gets. I've dl'd the custom 2.5.3 1.7 tools but nothing
    It dont even crash it just stops as tho its finished
    Post edited by Unknown User on
  • builderhicksbuilderhicks Join Date: 2003-02-08 Member: 13263Members Posts: 7
    Oh and another thing i got a MAX MIPTEX error is this to do with the size of my wad or the size/amounts of the textures in the map?confused.gif
    I know it dont have anything to do with MAX MAP_PLANES but i got the error after i deleted some detail to get rid of the MAX_PLANES error i use zoners tools (as u can tell DOH!) how does the -texdata work to put the 4mg memory up to 8mg.
    ty
  • TokenToken Join Date: 2003-02-08 Member: 13262Members Posts: 2
    just a quick question ... is the new zoners 1.7 .CSG compatable with the null tex(merles_custom_tools)? im just trying to troubleshoot, BSP keeps crashing with the modified tools..when i use my origianl tools i get max planes error. I am going to mess around with my map delete some of my most recent additions see if it helps.... any advice welcome
Sign In or Register to comment.