Entdat Optimiser Tool

FragBait0 · August 2004

<div class="IPBDescription">Cut your entity data in half</div> Ah well just thought some of you people might like to try this out:
<a href='http://www.chatbear.com/board.plm?a=viewthread&t=52,1080647136,18491&id=634830&b=590&v=flatold' target='_blank'>http://www.chatbear.com/board.plm?a=viewth...b=590&v=flatold</a>

Or you can grab it directly at:
<a href='http://users.tpg.com.au/dplevi/opt_entdata_0.9.rar' target='_blank'>http://users.tpg.com.au/dplevi/opt_entdata_0.9.rar</a>

Go ahead and post any bugs or suggestions you have to the VERC thread, or here for that matter. If you have to e-mail me: tlevi@tpg.com.au

BlackPlague · August 2004

you know us, no screenies, no downloads. you dont even tell us what it IS to begin with... please describe this program <img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' />

Spacer · August 2004

....
It's an optimiser, plague <img src='http://www.unknownworlds.com/forums/html//emoticons/confused-fix.gif' border='0' style='vertical-align:middle' alt='confused-fix.gif' />.

Jezpuh · August 2004

</div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (BlackPlague @ Aug 21 2004, 04:53 PM)</td></tr><tr><td id='QUOTE'> you know us, no screenies, no downloads. you dont even tell us what it IS to begin with... please describe this program <img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' /> </td></tr></table><div class='postcolor'>
Do you just say that every time you see a link?

BlackPlague · August 2004

no? i didnt see any description for the program. so i asked for one <img src='http://www.unknownworlds.com/forums/html//emoticons/tounge.gif' border='0' style='vertical-align:middle' alt='tounge.gif' /> and i also wanted some screenies ^_^

Mendasp · August 2004

I'll give a description:

It optimizes entity data in bsp files.

BlackPlague · August 2004

:O thank the allmighty mandasp for his wisdom!!! hes the greatest! lol, i tried this program out, its pretty good.

Some_tall_guy1 · August 2004

So would this optimizer reduce lag from the entities?

Reese · August 2004

STG: in that it would remove entites that are unnecessary, yes. I don't exactly know the specifics on this but i'm guessing that some of the methods used to store entity data are a little bloated. More than likely this program goes through and checks for entities that can be entirely removed, or that were duplicated for some reason. (The forum post that Fragbait linked to mentioned something about duplicate entities)

I don't know if beyond that it does anything to reduce the system stress caused by an entity. I think in the end there are just going to be some things that are impossible to do at high framerates/less lag without access to the rendering code.

Hibame · August 2004

How do I know it wont mess up my code doors >.<

Reese · August 2004

You don't know that it won't. That said just make a backup of the bsp and if it does you can just delete the non-working bsp. Don't even have to recompile at that point.

FragBait0 · August 2004

If you do screw up your map badly simply do a -onlyents in CSG and you get it all back. Or you can try restoring the .ent backup with ripents.

</div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'> you know us, no screenies, no downloads. you dont even tell us what it IS to begin with... please describe this program</td></tr></table><div class='postcolor'>
Sorry about the lack of description - I have typed it out a few times and I thought linkage to a 7 page thread at VERC would be enough.
As for screenies....its just a command-line app like the compile tools.
Download? I provided a link there and clearly said it was a direct download...

Well Ill write a basic summary -
In half-life BSPs you have a chunck of it for entities (duh). They appear much like in your .map file (open it in notepad if you dont know what im talking about) and just like in the map file all the properties with default values are kept in the bsp. This wastes a lot of space. Also kept in the bsp are the lights. Now why do you need them after RAD? Unless the lights are named (switchable) or they have a style of some sort (flicker, pulse etc) then they get taken out the entdata.

Before anyone kills me over this im just going to advise you all that it is one giant hack all the way. That dosen't mean its not safe but its an optimiser...
</div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'>Optimizers cheat, and sometimes they get caught cheating.</td></tr></table><div class='postcolor'>
<a href='http://www.perl.com/pub/a/2004/08/18/onion.html?page=3' target='_blank'>http://www.perl.com/pub/a/2004/08/18/onion.html?page=3</a>

</div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'>I don't know if beyond that it does anything to reduce the system stress caused by an entity. I think in the end there are just going to be some things that are impossible to do at high framerates/less lag without access to the rendering code.</td></tr></table><div class='postcolor'>
The lag - I have no idea if it actually will give less lag in your maps but it does remove useless light entities and info_null. You need to be very carefull what you use info_null for because this program removes any occurance without thinking. I took info_null to be used only as a target for light_spots in ZHLT so assuming thats how you use them then you wont have any problems because of it.

So here is what the output looks like anyway:
</div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'>opt_entdata, version 0.7 Beta by FragBait0,
Submit detailed bug reports and suggestions to FragBait0: tlevi@tpg.com.au.
Parts based on code by Zoner.
--- BEGIN opt_entdata ---

Command line options: g:\games\HL\spirit\maps\frag_dm1 -simplify -delbackup -ripent -verbose

Setting from: g:\hldev\programs\vhe\tools\fragsperl\setting_default.db
Reading from: g:\hldev\programs\vhe\tools\fragsperl\propertylist.db
Reading from: g:\hldev\programs\vhe\tools\fragsperl\propertylist_spirit.db
Reading from: g:\hldev\programs\vhe\tools\fragsperl\propertylist_zhlt.db
Reading from: g:\hldev\programs\vhe\tools\fragsperl\propertylist_custom.db
Using ripent: "g:\hldev\programs\vhe\tools\fragsperl\..\xhlt\ripent.exe" -export g:\games\HL\spirit\maps\frag_dm1.bsp
Using ripent: "g:\hldev\programs\vhe\tools\fragsperl\..\xhlt\ripent.exe" -import g:\games\HL\spirit\maps\frag_dm1.bsp

Reading file..Removing unrequired entities..Striping ent properties..
Removing Lights..Optimising Dynamic Lights..Fixing bad paths..
Symplifying names..Modding Line Terminator..Writing file..
Renaming files..Deleting Backup..

4129 lines in input file,
1692 lines in output file,
Striped a list of 334 unique properties, replaced 0 things.
65642 Bytes reduced to 26266 Bytes.
Difference of -39376 Bytes, roughly 40% of original size.

---- END opt_entdata ----</td></tr></table><div class='postcolor'>

That's a lot of text really but I do have verbose mode on there. There is another option, -noscrspam for those that use the hammer compile button. You get very little output with that. As you can see there signifigant reductions in entdata sizes can occur. This map takes about 5 seconds to be optimised. On my shiny athlon 3000+.
That sounds a little bit crap and it is because the program is written in perl. Interperted. Slow.

</div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'>29679 Bytes reduced to 5877 Bytes.
Difference of -23802 Bytes, roughly 19.8% of original size.</td></tr></table><div class='postcolor'>
Thats another map of thine but almost every ent is a light.

Ah and i screwed up the topic title. I guess a moderator has to fix that because I can't find a button for it here <img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' />

tommyd · August 2004

hmmm, i didnt think light ents actually cause any server 'lag' at all... i don't think they are 'tracked' by the server - much like info_loc's and info_player_starts. this might just end up reducing map size, which isn't a bad thing. i don't know much about the zoner tools and stuff like that, so i could be wrong.

OneEyed · August 2004

Actually I think every entity gets used by the servers, because you can have the clients with an entirely blank entity list in their .bsp, and yet have the server's entity list full of the entities in the .bsp which its data gets transfered to clients.

As for the lights actually causing any lag, maybe if HL has a problem lagging after to many entities, regardless of what type, which I doubt, but not sure if its been declared.

prsearle · August 2004

All the entities that aren't counted towards the NS entity limit are removed as soon as they spawn. So unnamed/non-flickering lights don't cause any server or network lag, but they do unnecesarily bulk up your BSP.

Drath · August 2004

so will this give any benefit besides reducing bsp filesize?

prsearle · August 2004

Well, it may slightly reduce the loading times as the engine doesn't have to parse/load all the redundant entities and then immediately remove them, but I'm not sure whether you'd notice it. If it removes redundant dynamic lights (I think it does), then it will save a few entities if you have multiple switchable lights with the same name.

FragBait0 · August 2004

I think you guys have figured it all out really....to be honest i dont actually know. It all started out as a pet project to learn perl. Once I finished I made a thread on VERC...

Anyways yeah they just bloat your BSP and I save about 5-10-20k or so on my maps. Your milage will vary. The thing about entdata is it varies a lot between mappers, maps and mods. There are few 'rules' about what is going to be in there in terms of bloatage. With planes you see a 2/3 reduction or maybe only 1/2. Its not a lot. I check one of my other maps and it had 9% left. Then I checked the RMF....only 4 ents other than lights.

</div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'> If it removes redundant dynamic lights (I think it does), then it will save a few entities if you have multiple switchable lights with the same name.</td></tr></table><div class='postcolor'>

I thought I coded it that way but I was looking at one of my map's entdata the other day and it didnt seem to have been done.

Ah well thats why we have version numbers!
Bring on 0.8...

EDIT: by "its not a lot" regarding the planes optimiser I meant that there is not a huge amount of variation in percentages removed compared to my percentages removed by my tool. XP-Cagey's plane optimiser is an excellent tool and I didn't intend to say it was not effective.

FragBait0 · September 2004

I've just put up an alpha of 0.8. It does remove all duplicate dynamic lights now so give it a try. Also introduced is an origin remover - although only for ents that are safe - it does add another "anti-decompile" aspect. And then there is the usual tweak and bugfix stuff.

Enjoy. Smash somthing. Post.

FragBait0 · September 2004

0.8 has become the current "release" version.

Have a shot with it. I've added a DB for NS now too so your percentages might go down a little more. And like I said it removes identical dynamic lights now so you save a few entities.

FragBait0 · November 2004

I've uploaded 0.9 so I might as well announce it here.

0.8-0.9 Changlog entries
Alpha 1
-Modified stripnulents to get more speed
-Modified the status updates for stripnulents (better amount on smaller maps)
-Fix for changelevel crashes (HL couldn't find landmarks if you used -simplify, it is now safe to do so)
-Changed info from bytes to kB.
-Fix to dynamic light removal.
-Wad path optimiser now eats the .wad extension as suggested by Wolf.
-Modified file loading to work off one database file.
Alpha 2
-Fix for spirit special trigger methods in -simplify.
-Fix for spirit's myentity(whatever) stuff. (untested)
Beta
-Small speed boost for stripnulents
-Fixed missing line on second run of opt_entdata. (still need to use -nolinemod on first run)
-Fixed stupid issue with -settings, its also -setting now although -settings will still work.
-Added "skin" "-1" property

Enjoy.. or find a mod to lock the thread. <img src='http://www.unknownworlds.com/forums/html//emoticons/tounge.gif' border='0' style='vertical-align:middle' alt='tounge.gif' />

Meat_Popsicle · November 2004

gd work, ill have to try this on the ns training map which uses a ridiculous amount of ents

GiGaBiTe · November 2004

woo, i tried it and i get a ?? bad crc32 error ?? from ns

lol..

FragBait0 · November 2004

GiGaBiTe: Logs. Details. Copy of entdata (before/after). Steamed NS or WON? <img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' />

And yes I'm looking into HL2.

EDIT: And I'm not getting topic update emails so excuse any laggy responses..

Jezpuh · November 2004

</div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (FragBait0 @ Nov 9 2004, 04:34 PM)</td></tr><tr><td id='QUOTE'> GiGaBiTe: Logs. Details. Copy of entdata (before/after). Steamed NS or WON? <img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' />

And yes I'm looking into HL2.

EDIT: And I'm not getting topic update emails so excuse any laggy responses..  </td></tr></table><div class='postcolor'> 
WON's dead ;x~

NEO_Phyte · November 2004

</div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (Jezpuh @ Nov 9 2004, 09:02 AM)</td></tr><tr><td id='QUOTE'> </div><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> (FragBait0 @ Nov 9 2004, 04:34 PM)</td></tr><tr><td id='QUOTE'> GiGaBiTe: Logs. Details. Copy of entdata (before/after). Steamed NS or WON? <img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' />

And yes I'm looking into HL2.

EDIT: And I'm not getting topic update emails so excuse any laggy responses.. </td></tr></table><div class='postcolor'>
WON's dead ;x~  </td></tr></table><div class='postcolor'> 
but it is still possible to run NS through the WON version, you are just limited to LANing with friends

FragBait0 · November 2004

All true but you can still map in it.
I do. But that'd be because i don't have HL on steam at all <img src='http://www.unknownworlds.com/forums/html//emoticons/smile-fix.gif' border='0' style='vertical-align:middle' alt='smile-fix.gif' />

FragBait0 · November 2004

For the cutting edge people using the latest ZHLT 3 beta 4 I've added support for the new ripent options.

1.0 Alpha 1
-Added -noinfo, -chart and -lightdata options for passing to ripent. (ZHLT 3 required)

<a href='http://users.tpg.com.au/dplevi/opt_entdata_1.0a1.rar' target='_blank'>http://users.tpg.com.au/dplevi/opt_entdata_1.0a1.rar</a>

GiGaBiTe: ....bug or no bug??

Entdat Optimiser Tool

Comments