267: FPS quality, part 2: Files and the critical path

matsomatso Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
edited July 2014 in NS2 General Discussion
The second part of the NS2 frame time quality pretty much comes down to incomplete asset setup.

Essentially, all assets (models, textures, animation, fonts, sounds, materials, shaders etc) should have been loaded and compiled/prepared before you start playing the game so that if you need an asset while the program is on the critical path (ie working on the next frame), it is instantly available.

That's what NS2 is doing during that long Loading/Precaching phase. Or rather "should" be doing.

When looking at hitchy "p_logall" logs in the PerfAnylyzer, I could see that many of the had a of File::Open lingering in the neighbourhood. Unfortunately, there was no way of seeing WHAT file was opened in the profiler.

Once I had engine access, I added a "file critical path" logging - ie, you could tell the filesystem when it was on a critical path so it could log file opens there (there is a lot of files being opened in non-critical paths (texture streaming among other things), so logging all file accesses would be tedious).

After looking at that, it turned out that there was a lot of files being opened in the wrong place, and for lots of different reasons.


- Fonts were opened up on demand, and the FontManager kept track of how many were using a font ... and releasing it when the usage count went to zero.

But sometimes your HUD display pops up an element that is the only users of a font - which means you take a hit loading the font from the filesystem AND compiling it; an average of 5-10ms. Then the element fades out ... and the FontManager promptly releases the font, preparing for another 5-10ms hitch the next time that HUD element needs to be shown.

- Materials are short files ~100bytes or so. They were considered too small to be precached... which is fine if the file is in the filesystem buffers or on an SSD, but if not - 5-20ms hitch for each (http://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics).

- Cinematics is kinda like a movie containing sounds, models (with materials and textures) and a script; there are droves of them in the game, doing everything from waterfalls to railgun splatting.

Turns out that cinematics precached their textures. Nothing else. Which meant that the first time a cinematic was played for you, it would start out by compiling any models that wasn't present already. Starting by loading them from the filesystem... and a cinematic can contain upto 20 models. And models take time to compile... hi, 100+ ms hitch.

As cinematics control stuff like blink effects, blood spray etc ... guess what happens the first time you are in combat - or the first time you meet a new lifeform or marine weapon.

- The GUI system is used for the HUD display, and it's ... weird. All the rest of the lua code is loaded when the client starts, but the GUI code is loaded on demand. So the first time you become a jetpack marine or an Exo or an alien, the gui script is compiled.

Not so bad, really - a one time hitch the first time you become a new lifeform isn't really anything to worry about.

However, it interferes badly with how you precache things in NS2. You do that by doing a compile time statement that you will be using an asset, so as the lua code is being loaded, the NS2 client is being told what assets it needs.

Works well when the lua code is loaded during startup... actually, it ONLY works during that time, as the precaching system does not like being used DURING the game ... then it's not _pre_ caching, really.

So no assets used only by the GUI elements are precached... and there are 422 files in "ui/*" just waiting to hitch your frame.

- The only weirder part than the GUI system is actually a GUI subsystem called GUIViews ..this allows you to paint on a texture, and originally (like, back in the oughties) this was done in its own Flash virtual machine. Adding Flash just to tell you how many bullets were in the rifle was a silly idea, so Flash got kicked out and replaced by a Lua VM.

Unfortunately, it wasn't replaced by using the already existing Lua VM. It was replaced by an individual Lua VM. One for every display. So a standard marine has one Lua VM dedicated to his rifle bullet display and one for his pistol bullet display. Oh, and one for the builder display. And another for the welder, if he has one.

This isn't really as bad as it sounds - Lua VMs are small and cheap. The problem here is that the displays are dynamically created and destroyed - if you drop your rifle, the Lua VM is destroyed. When you pick it up again, a new Lua VM is created - AND loaded up with code ... which is read from the filesystem.

And of course, the Lua code used in these separate lua VM don't even have access to the precaching system (not that they could have used it anyhow ....) so any textures or fonts used in them will of course not have been precached.

So if you wonder why picking up today's first shotgun off the floor can freeze you for 100+ ms, now you know.


Anyhow, most of these things (the most critical stuff) are fixed in 267, resulting in a much smoother experience.

The rest are scheduled for 268.

Ain't gonna be no hitching on CDT's watch.
«1345

Comments

  • TurbineTurbine Join Date: 2012-09-13 Member: 159160Members
    Sounds good, I didn't realise so many people in-game were having the same issues. The earlier this is released, the less damage done. A month has come since the last patch. *optimistically sad face*
  • HoliHoli Sweden Join Date: 2014-01-28 Member: 193600Members
    Awesome read! Thanks for telling us :) Now.. Back to waiting :(
  • MuckyMcFlyMuckyMcFly Join Date: 2012-03-19 Member: 148982Members, Reinforced - Supporter, Reinforced - Shadow
  • CyberKunCyberKun Join Date: 2013-02-02 Member: 182733Members, Reinforced - Shadow
    While this is an amazing piece of work for CDT, it is also a massive shame for the old devs. This should not have happened.
  • CyberKunCyberKun Join Date: 2013-02-02 Member: 182733Members, Reinforced - Shadow
    Asraniel wrote: »
    Also, only judge those changes once you actually play with them. They are not the silver bullet to performance problems. Yes, some computers will benefit from them, but others wont and lot of work/research will have to be done to help them also.

    I've been having micro-hitches for a very long time and this is the exact fix that is needed for myself and quite a few other players. The fact that it took this long to get to the bottom of it is a problem. I know that work can always be improved, and part of that is that there is less and less obvious things to be fixed. The micro-hitching though? Too common for it to be found now. It is not like it is a small typo in this case.

    Even still, I should have wrote the original post in a better manner. "This should not have happened", should be, "this should have been fixed way before.". I'm trying to think of a more common issue that harms the game that has been around for as long. I know IronHorse can find something.
  • RoobubbaRoobubba Who you gonna call? Join Date: 2003-01-06 Member: 11930Members, Reinforced - Shadow, WC 2013 - Shadow
    I think this is great - the care and attention NS2 needs at this point in its lifecycle. Agree with Asraniel, no shame on UWE at all for this, it's just fantastic to see progress being made now that the opportunity has presented itself in the CDT being granted access to the engine as well as the LUA code.

    How much it will help almost doesn't matter - any improvement is a good thing. Stuttering is something that pretty much anyone on any system can experience in NS2, so any step to address this is brilliant :)

    Thank you @matso for the very clear information on these posts! More! :)
  • Cannon_FodderAUSCannon_FodderAUS Brisbane, AU Join Date: 2013-06-23 Member: 185664Members, Squad Five Blue, Squad Five Silver, Reinforced - Shadow
    3 cheers for Matso... Hip Hip Horray, Hip Hip Horray, Hip Hip Horray. Holy crap, I can't imagine how long it would have taken to work all this out.
  • CalegoCalego Join Date: 2013-01-24 Member: 181848Members, NS2 Map Tester
    3 cheers for Matso... Hip Hip Horray, Hip Hip Horray, Hip Hip Horray. Holy crap, I can't imagine how long it would have taken to work all this out.
    All Hail Matso! And anyone else who has worked on this!
  • ShotoSelectionShotoSelection Join Date: 2005-01-23 Member: 37558Members
    I just want to say, great work. It seems like these issues were not very easy to find and I appreciate all the hard work and effort you guys are putting into the game. Anything that makes an NS player's experience and performance that much better is well worth it imo. Thank you!
  • MendaspMendasp I touch maps in inappropriate places Valencia, Spain Join Date: 2002-07-05 Member: 884Members, NS1 Playtester, Contributor, Constellation, NS2 Playtester, Squad Five Gold, NS2 Map Tester, Reinforced - Shadow, WC 2013 - Shadow, Retired Community Developer
    Matso is the coolest guy in the universe.
  • joshhhjoshhh Milwaukee, WI Join Date: 2011-06-21 Member: 105717Members, NS2 Playtester, NS2 Map Tester, Reinforced - Supporter, Reinforced - Shadow, WC 2013 - Shadow, Subnautica Playtester
    You're a wizard, Harry Matso!
  • DC_DarklingDC_Darkling Join Date: 2003-07-10 Member: 18068Members, Constellation, Squad Five Blue, Squad Five Silver
    Well done. (last sentence made me laugh Matso)

    Also I would not bugger uwe to hard about not finding these bugs sooner. Proving there was a bug for many was surely hard enough, and thats usually step one in fixing a bug.. right? :p
  • IeptBarakatIeptBarakat The most difficult name to speak ingame. Join Date: 2009-07-10 Member: 68107Members, Constellation, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Diamond, Reinforced - Shadow
    whens the patch? whens the patch? whens the patch?

    After reading through this, reading each of the 'ready for 267' posts in the CDT Public Tracker, I can easily say this patch should greatly improve my overall enjoyment of the game.

    Now I know this is no silver bullet and some things will have to come later, but anything that can reduce the amount of hitching NS2 does with my old Radeon HD 6770 GPU and 7200RPM HDD will probably cause me to play it a lot more than I have been.

    The "Smooth PhaseGate exit" fix in the Public Tracker has me excited too, I've had to play on the lowest settings because how bad the hitching can get overall. (To compensate for lost frames.)
  • RockyMarcRockyMarc Join Date: 2009-11-24 Member: 69519Members, Reinforced - Shadow
    Sounds bloody awesome! Time to get out the Commodore64 again and crank up some NS2 on it!
    :P
  • soccerguy243soccerguy243 Join Date: 2012-12-22 Member: 175920Members, WC 2013 - Supporter
    edited July 2014
    now these are something's i'd pay (again) for!
  • Cannon_FodderAUSCannon_FodderAUS Brisbane, AU Join Date: 2013-06-23 Member: 185664Members, Squad Five Blue, Squad Five Silver, Reinforced - Shadow
    RockyMarc wrote: »
    Sounds bloody awesome! Time to get out the Commodore64 again and crank up some NS2 on it!
    :P

    With the tape deck!
  • OnosFactoryOnosFactory New Zealand Join Date: 2008-07-16 Member: 64637Members
    *With the tape deck!* - Groan - 30 minutes of loading for an 8bit vertical scrolling shooter. The stuff of nightmares.

  • RockyMarcRockyMarc Join Date: 2009-11-24 Member: 69519Members, Reinforced - Shadow
    *With the tape deck!* - Groan - 30 minutes of loading for an 8bit vertical scrolling shooter. The stuff of nightmares.

    Had so much patience with those.. Now if software takes longer then 10 seconds to load I start to rage :(
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    edited July 2014
    RockyMarc wrote: »
    Now if software takes longer then 10 seconds to load I start to rage :(

    How do you cope with battlefield games? Yeesh.. takes me like 5 minutes to load
    (besides the atrocious lensflare / visual obstruction/ low skill ceiling gameplay)
  • BensonBenson Join Date: 2012-03-07 Member: 148303Members, Reinforced - Shadow, WC 2013 - Shadow
    I think Matso may be one of the only people who has more awsomes than agrees XD
  • Cannon_FodderAUSCannon_FodderAUS Brisbane, AU Join Date: 2013-06-23 Member: 185664Members, Squad Five Blue, Squad Five Silver, Reinforced - Shadow
    IronHorse wrote: »
    RockyMarc wrote: »
    Now if software takes longer then 10 seconds to load I start to rage :(

    How do you cope with battlefield games? Yeesh.. takes me like 5 minutes to load
    (besides the atrocious lensflare / visual obstruction/ low skill ceiling gameplay)

    Easy, Don't play BF games!! I got BF BC3 free, and I loaded it up once. then deleted it. NS2 so much better.
  • HamletHamlet Join Date: 2008-08-17 Member: 64837Members, Reinforced - Shadow
    I wonder what would be the ideal (minimum) amount of VRAM for NS2 to run properly.
    2 GB? 3 GB?
  • RoobubbaRoobubba Who you gonna call? Join Date: 2003-01-06 Member: 11930Members, Reinforced - Shadow, WC 2013 - Shadow
    Benson wrote: »
    I think Matso may be one of the only people who has more awsomes than agrees XD

    that would be Mendasp. ;)
  • Dictator93Dictator93 Join Date: 2008-12-21 Member: 65833Members, Reinforced - Shadow
    Hamlet wrote: »
    I wonder what would be the ideal (minimum) amount of VRAM for NS2 to run properly.
    2 GB? 3 GB?

    After the update... it should probably be anything around 1GB being sufficient for high textures I hope. Proper streaming should allow for almost any resolution of textures to be viewed up close... it is just their quality while moving around or in the distance that should be compromised.

    Higher amounts of VRAM should allow for more temporal stability in texture quality IMO, not better textures when viewed up close (unless the gulf in texture RAM is huge like 6GB)
  • matsomatso Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
    edited July 2014
    asdf
    Dictator93 wrote: »
    Hamlet wrote: »
    I wonder what would be the ideal (minimum) amount of VRAM for NS2 to run properly.
    2 GB? 3 GB?

    After the update... it should probably be anything around 1GB being sufficient for high textures I hope. Proper streaming should allow for almost any resolution of textures to be viewed up close... it is just their quality while moving around or in the distance that should be compromised.

    Higher amounts of VRAM should allow for more temporal stability in texture quality IMO, not better textures when viewed up close (unless the gulf in texture RAM is huge like 6GB)

    1.6-1.8 Gb allocated is the about the high limit with high textures; depends a bit on the map.

    So a 2Gb card do not have any problem with running high textures.

    Medium textures have half the dimension as high (you drop one MipMap level), so they take up 1/4 of the high textures (and low has 1/16th of high).

    As you have some (say 100Mb) of other stuff, that means medium textures hits about 600-700Mb and low 300Mb, about.

    And yea, the texture streamer actually works well enough that you can run on high textures with a 1-1.5Gb card (if you have less than that, used medium textures).

    If you have a dual core, the extra work from the streamer might lower your fps by 1-2% though.

  • CCTEECCTEE Join Date: 2013-06-20 Member: 185634Members, Reinforced - Shadow
    Benson wrote: »
    I think Matso may be one of the only people who has more awsomes than agrees XD

    A privilige reserved for the really usefull or really useless forum members.
Sign In or Register to comment.