267: FPS quality, part 2: Files and the critical path

matsomatso Master of PatchesMembers, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer Join Date: 2002-11-05 Member: 7000Posts: 1,554 mod
edited July 2014 in NS2 General Discussion
The second part of the NS2 frame time quality pretty much comes down to incomplete asset setup.

Essentially, all assets (models, textures, animation, fonts, sounds, materials, shaders etc) should have been loaded and compiled/prepared before you start playing the game so that if you need an asset while the program is on the critical path (ie working on the next frame), it is instantly available.

That's what NS2 is doing during that long Loading/Precaching phase. Or rather "should" be doing.

When looking at hitchy "p_logall" logs in the PerfAnylyzer, I could see that many of the had a of File::Open lingering in the neighbourhood. Unfortunately, there was no way of seeing WHAT file was opened in the profiler.

Once I had engine access, I added a "file critical path" logging - ie, you could tell the filesystem when it was on a critical path so it could log file opens there (there is a lot of files being opened in non-critical paths (texture streaming among other things), so logging all file accesses would be tedious).

After looking at that, it turned out that there was a lot of files being opened in the wrong place, and for lots of different reasons.


- Fonts were opened up on demand, and the FontManager kept track of how many were using a font ... and releasing it when the usage count went to zero.

But sometimes your HUD display pops up an element that is the only users of a font - which means you take a hit loading the font from the filesystem AND compiling it; an average of 5-10ms. Then the element fades out ... and the FontManager promptly releases the font, preparing for another 5-10ms hitch the next time that HUD element needs to be shown.

- Materials are short files ~100bytes or so. They were considered too small to be precached... which is fine if the file is in the filesystem buffers or on an SSD, but if not - 5-20ms hitch for each (http://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics).

- Cinematics is kinda like a movie containing sounds, models (with materials and textures) and a script; there are droves of them in the game, doing everything from waterfalls to railgun splatting.

Turns out that cinematics precached their textures. Nothing else. Which meant that the first time a cinematic was played for you, it would start out by compiling any models that wasn't present already. Starting by loading them from the filesystem... and a cinematic can contain upto 20 models. And models take time to compile... hi, 100+ ms hitch.

As cinematics control stuff like blink effects, blood spray etc ... guess what happens the first time you are in combat - or the first time you meet a new lifeform or marine weapon.

- The GUI system is used for the HUD display, and it's ... weird. All the rest of the lua code is loaded when the client starts, but the GUI code is loaded on demand. So the first time you become a jetpack marine or an Exo or an alien, the gui script is compiled.

Not so bad, really - a one time hitch the first time you become a new lifeform isn't really anything to worry about.

However, it interferes badly with how you precache things in NS2. You do that by doing a compile time statement that you will be using an asset, so as the lua code is being loaded, the NS2 client is being told what assets it needs.

Works well when the lua code is loaded during startup... actually, it ONLY works during that time, as the precaching system does not like being used DURING the game ... then it's not _pre_ caching, really.

So no assets used only by the GUI elements are precached... and there are 422 files in "ui/*" just waiting to hitch your frame.

- The only weirder part than the GUI system is actually a GUI subsystem called GUIViews ..this allows you to paint on a texture, and originally (like, back in the oughties) this was done in its own Flash virtual machine. Adding Flash just to tell you how many bullets were in the rifle was a silly idea, so Flash got kicked out and replaced by a Lua VM.

Unfortunately, it wasn't replaced by using the already existing Lua VM. It was replaced by an individual Lua VM. One for every display. So a standard marine has one Lua VM dedicated to his rifle bullet display and one for his pistol bullet display. Oh, and one for the builder display. And another for the welder, if he has one.

This isn't really as bad as it sounds - Lua VMs are small and cheap. The problem here is that the displays are dynamically created and destroyed - if you drop your rifle, the Lua VM is destroyed. When you pick it up again, a new Lua VM is created - AND loaded up with code ... which is read from the filesystem.

And of course, the Lua code used in these separate lua VM don't even have access to the precaching system (not that they could have used it anyhow ....) so any textures or fonts used in them will of course not have been precached.

So if you wonder why picking up today's first shotgun off the floor can freeze you for 100+ ms, now you know.


Anyhow, most of these things (the most critical stuff) are fixed in 267, resulting in a much smoother experience.

The rest are scheduled for 268.

Ain't gonna be no hitching on CDT's watch.
Member of CDT, Senior Spark Engine Hacker
DecoySupaFredMuckyMcFlyMcGlaspieWyzcrakRoobubbameatmachinemoultanoVetinariDestherBlrgCannon_FodderAUSMouseLaosh'RaCalegoIeptBarakatWalsajoshhhSoul_RiderCmdrKeenDC_DarklingCarNagE1shonanLokiNominousSantaClawsxen32IronHorsevartijaMrFangsBeigeAlertMigeradionautkmgNeokenNarfwakcaperpDictator93Ricezsoccerguy243twilitebluecoineNeoQuaker1NousWandererJibrail0niDarakianLocklearSaltlickMiniH0wied4rkAlfRadimaXrkfgfirepowercypher0sixWarmongertummy_yummyHamlet
«1345

Comments

  • TurbineTurbine Members Join Date: 2012-09-13 Member: 159160Posts: 172 Fully active user
    Sounds good, I didn't realise so many people in-game were having the same issues. The earlier this is released, the less damage done. A month has come since the last patch. *optimistically sad face*
    senthy
  • HoliHoli SwedenMembers Join Date: 2014-01-28 Member: 193600Posts: 19 Fully active user
    Awesome read! Thanks for telling us :) Now.. Back to waiting :(
  • MuckyMcFlyMuckyMcFly Members, Reinforced - Supporter, Reinforced - Shadow Join Date: 2012-03-19 Member: 148982Posts: 616 Fully active user
    Great work. ^:)^


    [Cult]MartyMcFly - Yo!Clan Regular - Player of Real-Lifeâ„¢
    (\__/)
    (='.'=)
    (")_(")
  • CyberKunCyberKun Members, Reinforced - Shadow Join Date: 2013-02-02 Member: 182733Posts: 336 Fully active user
    While this is an amazing piece of work for CDT, it is also a massive shame for the old devs. This should not have happened.
    Cyber_kun: Not going to invade again
    Cyber_kun: Well, least not in the area that requires you to run forever
    Lolgorgeous: np i appreciate you wasting 30 minutes of my time anyway

    ~~Dark Souls~~
    RapGodd4rkAlfwulf
  • CyberKunCyberKun Members, Reinforced - Shadow Join Date: 2013-02-02 Member: 182733Posts: 336 Fully active user
    Asraniel wrote: »
    Also, only judge those changes once you actually play with them. They are not the silver bullet to performance problems. Yes, some computers will benefit from them, but others wont and lot of work/research will have to be done to help them also.

    I've been having micro-hitches for a very long time and this is the exact fix that is needed for myself and quite a few other players. The fact that it took this long to get to the bottom of it is a problem. I know that work can always be improved, and part of that is that there is less and less obvious things to be fixed. The micro-hitching though? Too common for it to be found now. It is not like it is a small typo in this case.

    Even still, I should have wrote the original post in a better manner. "This should not have happened", should be, "this should have been fixed way before.". I'm trying to think of a more common issue that harms the game that has been around for as long. I know IronHorse can find something.
    Cyber_kun: Not going to invade again
    Cyber_kun: Well, least not in the area that requires you to run forever
    Lolgorgeous: np i appreciate you wasting 30 minutes of my time anyway

    ~~Dark Souls~~
    d4rkAlf
  • RoobubbaRoobubba Who you gonna call?Members, Reinforced - Shadow, WC 2013 - Shadow Join Date: 2003-01-06 Member: 11930Posts: 3,191 Fully active user
    I think this is great - the care and attention NS2 needs at this point in its lifecycle. Agree with Asraniel, no shame on UWE at all for this, it's just fantastic to see progress being made now that the opportunity has presented itself in the CDT being granted access to the engine as well as the LUA code.

    How much it will help almost doesn't matter - any improvement is a good thing. Stuttering is something that pretty much anyone on any system can experience in NS2, so any step to address this is brilliant :)

    Thank you @matso for the very clear information on these posts! More! :)
    For all your gorge busting needs.
    It is very strange how some1 who spend so much time makeing videos to help mans, can fall and take miror image of dark ages bourgeoisie, outdated set of belifs
    How True.
  • Cannon_FodderAUSCannon_FodderAUS Brisbane, AUMembers, Squad Five Blue, Squad Five Silver, Reinforced - Shadow Join Date: 2013-06-23 Member: 185664Posts: 877 Advanced user
    3 cheers for Matso... Hip Hip Horray, Hip Hip Horray, Hip Hip Horray. Holy crap, I can't imagine how long it would have taken to work all this out.
    Author of a ePaper manual for NS2, now also a steam guide
    Co-wrote the Marine EXO Suit Protocol
    CalegoBeigeAlertNeoQuaker1tummy_yummy
  • CalegoCalego Members, NS2 Map Tester Join Date: 2013-01-24 Member: 181848Posts: 763 Advanced user
    3 cheers for Matso... Hip Hip Horray, Hip Hip Horray, Hip Hip Horray. Holy crap, I can't imagine how long it would have taken to work all this out.
    All Hail Matso! And anyone else who has worked on this!
  • ShotoSelectionShotoSelection Members Join Date: 2005-01-23 Member: 37558Posts: 17 Fully active user
    I just want to say, great work. It seems like these issues were not very easy to find and I appreciate all the hard work and effort you guys are putting into the game. Anything that makes an NS player's experience and performance that much better is well worth it imo. Thank you!
  • MendaspMendasp I touch maps in inappropriate places Valencia, SpainMembers, NS1 Playtester, Contributor, Constellation, NS2 Playtester, Squad Five Gold, NS2 Map Tester, Reinforced - Shadow, WC 2013 - Shadow, Retired Community Developer Join Date: 2002-07-05 Member: 884Posts: 4,177 Advanced user
  • joshhhjoshhh Milwaukee, WIMembers, NS2 Playtester, NS2 Map Tester, Reinforced - Supporter, Reinforced - Shadow, WC 2013 - Shadow, Subnautica Playtester Join Date: 2011-06-21 Member: 105717Posts: 1,437 Advanced user
    You're a wizard, Harry Matso!
    BeigeAlertObraxisNarfwakWasabiOne
  • DC_DarklingDC_Darkling Members, Constellation, Squad Five Blue, Squad Five Silver Join Date: 2003-07-10 Member: 18068Posts: 4,381 Advanced user
    Well done. (last sentence made me laugh Matso)

    Also I would not bugger uwe to hard about not finding these bugs sooner. Proving there was a bug for many was surely hard enough, and thats usually step one in fixing a bug.. right? :p
    NS2
    Old/resigned Dutch Translation Lead. Give feedback about the translation, or help improve by clicking: Current NS2 Dutch translation project. Dutch players apply!
    NS2 Server Maintenance Manual
    WorkshopBackup Manual (WSB Server)
    Ingame: DCDarkling
    'DCDs Steam Workshop', for your NS2 X-Hairs and Viewmodel Weapon mods
    Viewmodels are the ones you hold in your hand, not the one shown in the world!

    Old NS2 Stuff:
    Marine Commander Guide NS2
    Alien Commander Guide NS2
    NS2 Mentor Program
    Old NS1 stuff:
    'Infested Pack v0.6' for models & props
    Darklings Guide to Commanding Version 1.001
  • IeptBarakatIeptBarakat The most difficult name to speak ingame. Members, Constellation, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Diamond, Reinforced - Shadow Join Date: 2009-07-10 Member: 68107Posts: 2,679 Advanced user
    whens the patch? whens the patch? whens the patch?

    After reading through this, reading each of the 'ready for 267' posts in the CDT Public Tracker, I can easily say this patch should greatly improve my overall enjoyment of the game.

    Now I know this is no silver bullet and some things will have to come later, but anything that can reduce the amount of hitching NS2 does with my old Radeon HD 6770 GPU and 7200RPM HDD will probably cause me to play it a lot more than I have been.

    The "Smooth PhaseGate exit" fix in the Public Tracker has me excited too, I've had to play on the lowest settings because how bad the hitching can get overall. (To compensate for lost frames.)
    d4rkAlf
  • RockyMarcRockyMarc Members, Reinforced - Shadow Join Date: 2009-11-24 Member: 69519Posts: 474 Fully active user
    Sounds bloody awesome! Time to get out the Commodore64 again and crank up some NS2 on it!
    :P
    RapGod
  • soccerguy243soccerguy243 Members, WC 2013 - Supporter Join Date: 2012-12-22 Member: 175920Posts: 515 Advanced user
    edited July 2014
    now these are something's i'd pay (again) for!
  • Cannon_FodderAUSCannon_FodderAUS Brisbane, AUMembers, Squad Five Blue, Squad Five Silver, Reinforced - Shadow Join Date: 2013-06-23 Member: 185664Posts: 877 Advanced user
    RockyMarc wrote: »
    Sounds bloody awesome! Time to get out the Commodore64 again and crank up some NS2 on it!
    :P

    With the tape deck!
    Author of a ePaper manual for NS2, now also a steam guide
    Co-wrote the Marine EXO Suit Protocol
  • RockyMarcRockyMarc Members, Reinforced - Shadow Join Date: 2009-11-24 Member: 69519Posts: 474 Fully active user
    *With the tape deck!* - Groan - 30 minutes of loading for an 8bit vertical scrolling shooter. The stuff of nightmares.

    Had so much patience with those.. Now if software takes longer then 10 seconds to load I start to rage :(
    MrFangs
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributorMembers, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts Join Date: 2010-05-08 Member: 71669Posts: 8,194 admin
    edited July 2014
    RockyMarc wrote: »
    Now if software takes longer then 10 seconds to load I start to rage :(

    How do you cope with battlefield games? Yeesh.. takes me like 5 minutes to load
    (besides the atrocious lensflare / visual obstruction/ low skill ceiling gameplay)
    QUOTE (Techercizer @ Feb 3 2012, 10:47 AM) »
    Every time you ask for troubleshooting without providing system info, ATI adds a rendering bug for an upcoming game.

    When you feel you need to be rude or angry about a game, just read these links and remember what role you are playing:
    http://en.wikipedia.org/wiki/Online_disinhibition_effect
    http://www.eldergame.com/2008/06/taming-the-forum-tiger/
    Jekt
  • BensonBenson Members, Reinforced - Shadow, WC 2013 - Shadow Join Date: 2012-03-07 Member: 148303Posts: 1,296 Advanced user
    I think Matso may be one of the only people who has more awsomes than agrees XD
    If we are to be damned, let us be damned for what we really are. - Capt. Jean Luc Picard
  • Cannon_FodderAUSCannon_FodderAUS Brisbane, AUMembers, Squad Five Blue, Squad Five Silver, Reinforced - Shadow Join Date: 2013-06-23 Member: 185664Posts: 877 Advanced user
    IronHorse wrote: »
    RockyMarc wrote: »
    Now if software takes longer then 10 seconds to load I start to rage :(

    How do you cope with battlefield games? Yeesh.. takes me like 5 minutes to load
    (besides the atrocious lensflare / visual obstruction/ low skill ceiling gameplay)

    Easy, Don't play BF games!! I got BF BC3 free, and I loaded it up once. then deleted it. NS2 so much better.
    Author of a ePaper manual for NS2, now also a steam guide
    Co-wrote the Marine EXO Suit Protocol
  • HamletHamlet Members, Reinforced - Shadow Join Date: 2008-08-17 Member: 64837Posts: 373 Advanced user
    I wonder what would be the ideal (minimum) amount of VRAM for NS2 to run properly.
    2 GB? 3 GB?
  • RoobubbaRoobubba Who you gonna call?Members, Reinforced - Shadow, WC 2013 - Shadow Join Date: 2003-01-06 Member: 11930Posts: 3,191 Fully active user
    Benson wrote: »
    I think Matso may be one of the only people who has more awsomes than agrees XD

    that would be Mendasp. ;)
    For all your gorge busting needs.
    It is very strange how some1 who spend so much time makeing videos to help mans, can fall and take miror image of dark ages bourgeoisie, outdated set of belifs
    How True.
    Benson
  • Dictator93Dictator93 Members, Reinforced - Shadow Join Date: 2008-12-21 Member: 65833Posts: 657 Fully active user
    Hamlet wrote: »
    I wonder what would be the ideal (minimum) amount of VRAM for NS2 to run properly.
    2 GB? 3 GB?

    After the update... it should probably be anything around 1GB being sufficient for high textures I hope. Proper streaming should allow for almost any resolution of textures to be viewed up close... it is just their quality while moving around or in the distance that should be compromised.

    Higher amounts of VRAM should allow for more temporal stability in texture quality IMO, not better textures when viewed up close (unless the gulf in texture RAM is huge like 6GB)
  • matsomatso Master of Patches Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer Join Date: 2002-11-05 Member: 7000Posts: 1,554 mod
    edited July 2014
    asdf
    Dictator93 wrote: »
    Hamlet wrote: »
    I wonder what would be the ideal (minimum) amount of VRAM for NS2 to run properly.
    2 GB? 3 GB?

    After the update... it should probably be anything around 1GB being sufficient for high textures I hope. Proper streaming should allow for almost any resolution of textures to be viewed up close... it is just their quality while moving around or in the distance that should be compromised.

    Higher amounts of VRAM should allow for more temporal stability in texture quality IMO, not better textures when viewed up close (unless the gulf in texture RAM is huge like 6GB)

    1.6-1.8 Gb allocated is the about the high limit with high textures; depends a bit on the map.

    So a 2Gb card do not have any problem with running high textures.

    Medium textures have half the dimension as high (you drop one MipMap level), so they take up 1/4 of the high textures (and low has 1/16th of high).

    As you have some (say 100Mb) of other stuff, that means medium textures hits about 600-700Mb and low 300Mb, about.

    And yea, the texture streamer actually works well enough that you can run on high textures with a 1-1.5Gb card (if you have less than that, used medium textures).

    If you have a dual core, the extra work from the streamer might lower your fps by 1-2% though.

    Member of CDT, Senior Spark Engine Hacker
    Dictator93d4rkAlf
  • CCTEECCTEE Members, Reinforced - Shadow Join Date: 2013-06-20 Member: 185634Posts: 772 Advanced user
    Benson wrote: »
    I think Matso may be one of the only people who has more awsomes than agrees XD

    A privilige reserved for the really usefull or really useless forum members.
    grrrr im beast
    meatmachine
Sign In or Register to comment.