Entdat Optimiser Tool

FragBait0FragBait0 Join Date: 2003-12-16 Member: 24431Members
edited November 2004 in Mapping Forum
<div class="IPBDescription">Cut your entity data in half</div> Ah well just thought some of you people might like to try this out:
<a href='http://www.chatbear.com/board.plm?a=viewthread&t=52,1080647136,18491&id=634830&b=590&v=flatold' target='_blank'>http://www.chatbear.com/board.plm?a=viewth...b=590&v=flatold</a>

Or you can grab it directly at:
<a href='http://users.tpg.com.au/dplevi/opt_entdata_0.9.rar' target='_blank'>http://users.tpg.com.au/dplevi/opt_entdata_0.9.rar</a>

Go ahead and post any bugs or suggestions you have to the VERC thread, or here for that matter. If you have to e-mail me: tlevi@tpg.com.au
Off Topic Disagree Agree Awesome

Comments

  • BlackPlagueBlackPlague Join Date: 2004-02-02 Member: 25990Banned
    you know us, no screenies, no downloads. you dont even tell us what it IS to begin with... please describe this program <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' /><!--endemo-->
    Off Topic Disagree Agree Awesome
  • SpacerSpacer Invented dogs Join Date: 2003-05-02 Member: 16008Members
    ....
    It's an optimiser, plague <!--emo&???--><img src='http://www.unknownworlds.com/forums/html//emoticons/confused-fix.gif' border='0' style='vertical-align:middle' alt='confused-fix.gif' /><!--endemo-->.
    Off Topic Disagree Agree Awesome
  • JezpuhJezpuh Join Date: 2003-04-03 Member: 15157Banned
    edited September 2004
    <!--QuoteBegin-BlackPlague+Aug 21 2004, 04:53 PM--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (BlackPlague @ Aug 21 2004, 04:53 PM)</td></tr><tr><td id='QUOTE'><!--QuoteEBegin--> you know us, no screenies, no downloads. you dont even tell us what it IS to begin with... please describe this program <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' /><!--endemo--> <!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->
    Do you just say that every time you see a link?
    Off Topic Disagree Agree Awesome
  • BlackPlagueBlackPlague Join Date: 2004-02-02 Member: 25990Banned
    no? i didnt see any description for the program. so i asked for one <!--emo&:p--><img src='http://www.unknownworlds.com/forums/html//emoticons/tounge.gif' border='0' style='vertical-align:middle' alt='tounge.gif' /><!--endemo--> and i also wanted some screenies ^_^
    Off Topic Disagree Agree Awesome
  • MendaspMendasp I touch maps in inappropriate places Valencia, Spain Join Date: 2002-07-05 Member: 884Members, NS1 Playtester, Contributor, Constellation, NS2 Playtester, Squad Five Gold, NS2 Map Tester, Reinforced - Shadow, WC 2013 - Shadow, Retired Community Developer
    I'll give a description:

    It optimizes entity data in bsp files.
    Off Topic Disagree Agree Awesome
  • BlackPlagueBlackPlague Join Date: 2004-02-02 Member: 25990Banned
    :O thank the allmighty mandasp for his wisdom!!! hes the greatest! lol, i tried this program out, its pretty good.
    Off Topic Disagree Agree Awesome
  • Some_tall_guy1Some_tall_guy1 Join Date: 2003-05-22 Member: 16601Members
    So would this optimizer reduce lag from the entities?
    Off Topic Disagree Agree Awesome
  • ReeseReese Join Date: 2003-05-08 Member: 16143Members
    STG: in that it would remove entites that are unnecessary, yes. I don't exactly know the specifics on this but i'm guessing that some of the methods used to store entity data are a little bloated. More than likely this program goes through and checks for entities that can be entirely removed, or that were duplicated for some reason. (The forum post that Fragbait linked to mentioned something about duplicate entities)

    I don't know if beyond that it does anything to reduce the system stress caused by an entity. I think in the end there are just going to be some things that are impossible to do at high framerates/less lag without access to the rendering code.
    Off Topic Disagree Agree Awesome
  • HibameHibame Join Date: 2003-11-16 Member: 22974Members, Reinforced - Shadow
    How do I know it wont mess up my code doors >.<
    Off Topic Disagree Agree Awesome
  • ReeseReese Join Date: 2003-05-08 Member: 16143Members
    You don't know that it won't. That said just make a backup of the bsp and if it does you can just delete the non-working bsp. Don't even have to recompile at that point.
    Off Topic Disagree Agree Awesome
  • FragBait0FragBait0 Join Date: 2003-12-16 Member: 24431Members
    edited August 2004
    If you do screw up your map badly simply do a -onlyents in CSG and you get it all back. Or you can try restoring the .ent backup with ripents.

    <!--QuoteBegin--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'><!--QuoteEBegin--> you know us, no screenies, no downloads. you dont even tell us what it IS to begin with... please describe this program<!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->
    Sorry about the lack of description - I have typed it out a few times and I thought linkage to a 7 page thread at VERC would be enough.
    As for screenies....its just a command-line app like the compile tools.
    Download? I provided a link there and clearly said it was a direct download...

    Well Ill write a basic summary -
    In half-life BSPs you have a chunck of it for entities (duh). They appear much like in your .map file (open it in notepad if you dont know what im talking about) and just like in the map file all the properties with default values are kept in the bsp. This wastes a lot of space. Also kept in the bsp are the lights. Now why do you need them after RAD? Unless the lights are named (switchable) or they have a style of some sort (flicker, pulse etc) then they get taken out the entdata.

    Before anyone kills me over this im just going to advise you all that it is one giant hack all the way. That dosen't mean its not safe but its an optimiser...
    <!--QuoteBegin--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'><!--QuoteEBegin-->Optimizers cheat, and sometimes they get caught cheating.<!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->
    <a href='http://www.perl.com/pub/a/2004/08/18/onion.html?page=3' target='_blank'>http://www.perl.com/pub/a/2004/08/18/onion.html?page=3</a>

    <!--QuoteBegin--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'><!--QuoteEBegin-->I don't know if beyond that it does anything to reduce the system stress caused by an entity. I think in the end there are just going to be some things that are impossible to do at high framerates/less lag without access to the rendering code.<!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->
    The lag - I have no idea if it actually will give less lag in your maps but it does remove useless light entities and info_null. You need to be very carefull what you use info_null for because this program removes any occurance without thinking. I took info_null to be used only as a target for light_spots in ZHLT so assuming thats how you use them then you wont have any problems because of it.

    So here is what the output looks like anyway:
    <!--QuoteBegin--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'><!--QuoteEBegin-->opt_entdata, version 0.7 Beta by FragBait0,
    Submit detailed bug reports and suggestions to FragBait0: tlevi@tpg.com.au.
    Parts based on code by Zoner.
    --- BEGIN opt_entdata ---

    Command line options: g:\games\HL\spirit\maps\frag_dm1 -simplify -delbackup -ripent -verbose

    Setting from: g:\hldev\programs\vhe\tools\fragsperl\setting_default.db
    Reading from: g:\hldev\programs\vhe\tools\fragsperl\propertylist.db
    Reading from: g:\hldev\programs\vhe\tools\fragsperl\propertylist_spirit.db
    Reading from: g:\hldev\programs\vhe\tools\fragsperl\propertylist_zhlt.db
    Reading from: g:\hldev\programs\vhe\tools\fragsperl\propertylist_custom.db
    Using ripent: "g:\hldev\programs\vhe\tools\fragsperl\..\xhlt\ripent.exe" -export g:\games\HL\spirit\maps\frag_dm1.bsp
    Using ripent: "g:\hldev\programs\vhe\tools\fragsperl\..\xhlt\ripent.exe" -import g:\games\HL\spirit\maps\frag_dm1.bsp

    Reading file..Removing unrequired entities..Striping ent properties..
    Removing Lights..Optimising Dynamic Lights..Fixing bad paths..
    Symplifying names..Modding Line Terminator..Writing file..
    Renaming files..Deleting Backup..

    4129 lines in input file,
    1692 lines in output file,
    Striped a list of 334 unique properties, replaced 0 things.
    65642 Bytes reduced to 26266 Bytes.
    Difference of -39376 Bytes, roughly 40% of original size.

    ---- END opt_entdata ----<!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->

    That's a lot of text really but I do have verbose mode on there. There is another option, -noscrspam for those that use the hammer compile button. You get very little output with that. As you can see there signifigant reductions in entdata sizes can occur. This map takes about 5 seconds to be optimised. On my shiny athlon 3000+.
    That sounds a little bit crap and it is because the program is written in perl. Interperted. Slow.


    <!--QuoteBegin--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'><!--QuoteEBegin-->29679 Bytes reduced to 5877 Bytes.
    Difference of -23802 Bytes, roughly 19.8% of original size.<!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->
    Thats another map of thine but almost every ent is a light.

    Ah and i screwed up the topic title. I guess a moderator has to fix that because I can't find a button for it here <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' /><!--endemo-->
    Off Topic Disagree Agree Awesome
  • tommydtommyd Creator of ns_tanith, co_pulse and co_faceoff Join Date: 2002-01-24 Member: 40Members, NS1 Playtester, Contributor, NS2 Developer
    hmmm, i didnt think light ents actually cause any server 'lag' at all... i don't think they are 'tracked' by the server - much like info_loc's and info_player_starts. this might just end up reducing map size, which isn't a bad thing. i don't know much about the zoner tools and stuff like that, so i could be wrong.
    Off Topic Disagree Agree Awesome
  • OneEyedOneEyed Join Date: 2003-03-14 Member: 14493Members
    edited August 2004
    Actually I think every entity gets used by the servers, because you can have the clients with an entirely blank entity list in their .bsp, and yet have the server's entity list full of the entities in the .bsp which its data gets transfered to clients.

    As for the lights actually causing any lag, maybe if HL has a problem lagging after to many entities, regardless of what type, which I doubt, but not sure if its been declared.
    Off Topic Disagree Agree Awesome
  • prsearleprsearle Join Date: 2002-11-01 Member: 2365Members, Constellation
    All the entities that aren't counted towards the NS entity limit are removed as soon as they spawn. So unnamed/non-flickering lights don't cause any server or network lag, but they do unnecesarily bulk up your BSP.
    Off Topic Disagree Agree Awesome
  • DrathDrath Join Date: 2002-11-12 Member: 8310Members, NS1 Playtester, Contributor
    so will this give any benefit besides reducing bsp filesize?
    Off Topic Disagree Agree Awesome
  • prsearleprsearle Join Date: 2002-11-01 Member: 2365Members, Constellation
    Well, it may slightly reduce the loading times as the engine doesn't have to parse/load all the redundant entities and then immediately remove them, but I'm not sure whether you'd notice it. If it removes redundant dynamic lights (I think it does), then it will save a few entities if you have multiple switchable lights with the same name.
    Off Topic Disagree Agree Awesome
  • FragBait0FragBait0 Join Date: 2003-12-16 Member: 24431Members
    edited August 2004
    I think you guys have figured it all out really....to be honest i dont actually know. It all started out as a pet project to learn perl. Once I finished I made a thread on VERC...

    Anyways yeah they just bloat your BSP and I save about 5-10-20k or so on my maps. Your milage will vary. The thing about entdata is it varies a lot between mappers, maps and mods. There are few 'rules' about what is going to be in there in terms of bloatage. With planes you see a 2/3 reduction or maybe only 1/2. Its not a lot. I check one of my other maps and it had 9% left. Then I checked the RMF....only 4 ents other than lights.

    <!--QuoteBegin--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'><!--QuoteEBegin--> If it removes redundant dynamic lights (I think it does), then it will save a few entities if you have multiple switchable lights with the same name.<!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->

    I thought I coded it that way but I was looking at one of my map's entdata the other day and it didnt seem to have been done.

    Ah well thats why we have version numbers!
    Bring on 0.8...

    EDIT: by "its not a lot" regarding the planes optimiser I meant that there is not a huge amount of variation in percentages removed compared to my percentages removed by my tool. XP-Cagey's plane optimiser is an excellent tool and I didn't intend to say it was not effective.
    Off Topic Disagree Agree Awesome
  • FragBait0FragBait0 Join Date: 2003-12-16 Member: 24431Members
    I've just put up an alpha of 0.8. It does remove all duplicate dynamic lights now so give it a try. Also introduced is an origin remover - although only for ents that are safe - it does add another "anti-decompile" aspect. And then there is the usual tweak and bugfix stuff.

    Enjoy. Smash somthing. Post.
    Off Topic Disagree Agree Awesome
  • FragBait0FragBait0 Join Date: 2003-12-16 Member: 24431Members
    0.8 has become the current "release" version.

    Have a shot with it. I've added a DB for NS now too so your percentages might go down a little more. And like I said it removes identical dynamic lights now so you save a few entities.
    Off Topic Disagree Agree Awesome
  • FragBait0FragBait0 Join Date: 2003-12-16 Member: 24431Members
    I've uploaded 0.9 so I might as well announce it here.

    0.8-0.9 Changlog entries
    Alpha 1
    -Modified stripnulents to get more speed
    -Modified the status updates for stripnulents (better amount on smaller maps)
    -Fix for changelevel crashes (HL couldn't find landmarks if you used -simplify, it is now safe to do so)
    -Changed info from bytes to kB.
    -Fix to dynamic light removal.
    -Wad path optimiser now eats the .wad extension as suggested by Wolf.
    -Modified file loading to work off one database file.
    Alpha 2
    -Fix for spirit special trigger methods in -simplify.
    -Fix for spirit's myentity(whatever) stuff. (untested)
    Beta
    -Small speed boost for stripnulents
    -Fixed missing line on second run of opt_entdata. (still need to use -nolinemod on first run)
    -Fixed stupid issue with -settings, its also -setting now although -settings will still work.
    -Added "skin" "-1" property

    Enjoy.. or find a mod to lock the thread. <!--emo&:p--><img src='http://www.unknownworlds.com/forums/html//emoticons/tounge.gif' border='0' style='vertical-align:middle' alt='tounge.gif' /><!--endemo-->
    Off Topic Disagree Agree Awesome
  • Meat_PopsicleMeat_Popsicle Join Date: 2003-08-09 Member: 19254Members, Constellation
    gd work, ill have to try this on the ns training map which uses a ridiculous amount of ents
    Off Topic Disagree Agree Awesome
  • GiGaBiTeGiGaBiTe Join Date: 2003-10-07 Member: 21489Members
    edited November 2004
    woo, i tried it and i get a ?? bad crc32 error ?? from ns

    lol..
    Off Topic Disagree Agree Awesome
  • FragBait0FragBait0 Join Date: 2003-12-16 Member: 24431Members
    edited November 2004
    GiGaBiTe: Logs. Details. Copy of entdata (before/after). Steamed NS or WON? <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' /><!--endemo-->

    And yes I'm looking into HL2.

    EDIT: And I'm not getting topic update emails so excuse any laggy responses..
    Off Topic Disagree Agree Awesome
  • JezpuhJezpuh Join Date: 2003-04-03 Member: 15157Banned
    <!--QuoteBegin-FragBait0+Nov 9 2004, 04:34 PM--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (FragBait0 @ Nov 9 2004, 04:34 PM)</td></tr><tr><td id='QUOTE'><!--QuoteEBegin--> GiGaBiTe: Logs. Details. Copy of entdata (before/after). Steamed NS or WON? <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' /><!--endemo-->

    And yes I'm looking into HL2.

    EDIT: And I'm not getting topic update emails so excuse any laggy responses.. <!--QuoteEnd--> </td></tr></table><div class='postcolor'> <!--QuoteEEnd-->
    WON's dead ;x~
    Off Topic Disagree Agree Awesome
  • NEO_PhyteNEO_Phyte We need shirtgons&#33; Join Date: 2003-12-16 Member: 24453Members, Constellation
    <!--QuoteBegin-Jezpuh+Nov 9 2004, 09:02 AM--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (Jezpuh @ Nov 9 2004, 09:02 AM)</td></tr><tr><td id='QUOTE'><!--QuoteEBegin--> <!--QuoteBegin-FragBait0+Nov 9 2004, 04:34 PM--></div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (FragBait0 @ Nov 9 2004, 04:34 PM)</td></tr><tr><td id='QUOTE'><!--QuoteEBegin--> GiGaBiTe: Logs. Details. Copy of entdata (before/after). Steamed NS or WON? <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' /><!--endemo-->

    And yes I'm looking into HL2.

    EDIT: And I'm not getting topic update emails so excuse any laggy responses.. <!--QuoteEnd--></td></tr></table><div class='postcolor'><!--QuoteEEnd-->
    WON's dead ;x~ <!--QuoteEnd--> </td></tr></table><div class='postcolor'> <!--QuoteEEnd-->
    but it is still possible to run NS through the WON version, you are just limited to LANing with friends
    Off Topic Disagree Agree Awesome
  • FragBait0FragBait0 Join Date: 2003-12-16 Member: 24431Members
    All true but you can still map in it.
    I do. But that'd be because i don't have HL on steam at all <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' /><!--endemo-->
    Off Topic Disagree Agree Awesome
  • FragBait0FragBait0 Join Date: 2003-12-16 Member: 24431Members
    For the cutting edge people using the latest ZHLT 3 beta 4 I've added support for the new ripent options.

    1.0 Alpha 1
    -Added -noinfo, -chart and -lightdata options for passing to ripent. (ZHLT 3 required)

    <a href='http://users.tpg.com.au/dplevi/opt_entdata_1.0a1.rar' target='_blank'>http://users.tpg.com.au/dplevi/opt_entdata_1.0a1.rar</a>


    GiGaBiTe: ....bug or no bug??
    Off Topic Disagree Agree Awesome
Sign In or Register to comment.