Framerate Benchmarking
ScardyBob
ScardyBob Join Date: 2009-11-25 Member: 69528Forum Admins, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow
ScardyBob Join Date: 2009-11-25 Member: 69528Forum Admins, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow
<div class="IPBDescription">Quantifying the impact of specific changes</div>Since client FPS performance is a regular topic for NS2, I figured I’d try to do some benchmarking to see which changes do and do not affect fps. I used fraps to do the benchmarking and used the following steps for consistency:
<ul><li>Create a listen server on ns2_summit and put ‘cheats 1’ into console to start the round</li><li>Join marines and start on top of marine start command station</li><li>Run a loop from marine start, to heliport, to alien start, to surface, and back to marine start (~90s with sprinting)</li></ul>
My computer specs and default settings are the following:
CPU: Intel i5 2500k (OCd to 4.3GHz)
GPU: AMD Radeon HD6950
RAM: 4GB
Resolution: 1920x1080
Graphics: High
Windowed
Console settings:
<ul><li>r_shadows true</li><li>r_atmospherics false</li><li>r_bloom true</li><li>r_flash true</li><li>r_shadowsfade true</li><li>r_instancing true</li><li>r_gui true</li><li>r_mode lit</li></ul>
Two caveats apply:
<ol type='1'><li>This benchmark does not represent actual gameplay, but I expect the general trends in fps improvements to be similar to an actual match</li><li>My setup is pretty powerful and I’m unsure how other hardware will handle the different improvements (for example, I believe my CPU is the bottleneck, but that might not be true for someone with a powerful CPU, but poor GPU).</li></ol>
Here is my average fps (as recorded by fraps) and increases/decreases in fps I achieved with each single or combined change (in order from best to worst).
<a href="http://imgur.com/5Qyfa" target="_blank"><img src="http://i.imgur.com/5Qyfal.jpg" border="0" class="linked-image" /></a>
Basically, I think you can group the single changes into 3 categories:
<ol type='1'><li>Notable FPS improvements (>2.5% increase, in order from best to worst)
<ul><li>r_mode unlit</li><li>r_flash false</li><li>r_shadows false</li><li>r_bloom false</li><li>Running at full screen</li><li>r_instancing false</li></ul></li><li>Negligible FPS improvements (<2.5%, in order from best to worst)
<ul><li>Drop resolution to 1280x720</li><li>Drop resolution to 1600x900</li><li>Drop resolution to 800x600</li><li>r_gui false</li><li>Drop graphics to medium</li><li>Drop graphics to awful</li></ul></li><li>FPS decrease (in order from least to most decrease)
<ul><li>r_shadowsfade false</li><li>r_atmospherics true</li></ul></li></ol>
Of course, if you combine these options you can get a noticeable FPS improvement (>10%) (I show two options here just using console commands). However, I wouldn’t recommend using r_gui false or r_mode unlit, since the former gets rid of your GUI (no crosshairs, no menus, no comm view) and the latter basically gets rid of all lighting effects (i.e. the game looks terrible without the proper lighting).
The tl;dr version is that you can get a modest improvement (probably ~10-20%, depending on hardware) in framerate by disabling the major rendering options (r_mode unlit, r_shadows false, r_instancing false, r_bloom false, and r_flash false). All other changes don't seem to do much.
<ul><li>Create a listen server on ns2_summit and put ‘cheats 1’ into console to start the round</li><li>Join marines and start on top of marine start command station</li><li>Run a loop from marine start, to heliport, to alien start, to surface, and back to marine start (~90s with sprinting)</li></ul>
My computer specs and default settings are the following:
CPU: Intel i5 2500k (OCd to 4.3GHz)
GPU: AMD Radeon HD6950
RAM: 4GB
Resolution: 1920x1080
Graphics: High
Windowed
Console settings:
<ul><li>r_shadows true</li><li>r_atmospherics false</li><li>r_bloom true</li><li>r_flash true</li><li>r_shadowsfade true</li><li>r_instancing true</li><li>r_gui true</li><li>r_mode lit</li></ul>
Two caveats apply:
<ol type='1'><li>This benchmark does not represent actual gameplay, but I expect the general trends in fps improvements to be similar to an actual match</li><li>My setup is pretty powerful and I’m unsure how other hardware will handle the different improvements (for example, I believe my CPU is the bottleneck, but that might not be true for someone with a powerful CPU, but poor GPU).</li></ol>
Here is my average fps (as recorded by fraps) and increases/decreases in fps I achieved with each single or combined change (in order from best to worst).
<a href="http://imgur.com/5Qyfa" target="_blank"><img src="http://i.imgur.com/5Qyfal.jpg" border="0" class="linked-image" /></a>
Basically, I think you can group the single changes into 3 categories:
<ol type='1'><li>Notable FPS improvements (>2.5% increase, in order from best to worst)
<ul><li>r_mode unlit</li><li>r_flash false</li><li>r_shadows false</li><li>r_bloom false</li><li>Running at full screen</li><li>r_instancing false</li></ul></li><li>Negligible FPS improvements (<2.5%, in order from best to worst)
<ul><li>Drop resolution to 1280x720</li><li>Drop resolution to 1600x900</li><li>Drop resolution to 800x600</li><li>r_gui false</li><li>Drop graphics to medium</li><li>Drop graphics to awful</li></ul></li><li>FPS decrease (in order from least to most decrease)
<ul><li>r_shadowsfade false</li><li>r_atmospherics true</li></ul></li></ol>
Of course, if you combine these options you can get a noticeable FPS improvement (>10%) (I show two options here just using console commands). However, I wouldn’t recommend using r_gui false or r_mode unlit, since the former gets rid of your GUI (no crosshairs, no menus, no comm view) and the latter basically gets rid of all lighting effects (i.e. the game looks terrible without the proper lighting).
The tl;dr version is that you can get a modest improvement (probably ~10-20%, depending on hardware) in framerate by disabling the major rendering options (r_mode unlit, r_shadows false, r_instancing false, r_bloom false, and r_flash false). All other changes don't seem to do much.
Comments
On our server (HBZ) this option set back to default by the server, because of that.
On our server (HBZ) this option set to default by the server, because of that.<!--QuoteEnd--></div><!--QuoteEEnd-->
I used to like the idea of hiding in darkness serving as camouflage. However, I found out that it does not work well in multiplayer FPS games, simply because often, lighting is ultimately a malleable client-side effect. Creating more ambush locations with LOS blocking obstacles is a much better solution. That has been mentioned many times, and I have come to terms with that idea.
In NS2, shadows may still be possible to redeem darkness from becoming irrelevant. For example, under default lighting (r_mode lit), the shadow of a Skulk hidden behind a obstacle could easily betray the Skulk. Whereas under r_mode unlit, it is not rendered.
omg there are actually people with 70+ fps. *drool*
Got 20fps that drops to 5 whenever there is action going on.
In NS2, shadows may still be possible to redeem darkness from becoming irrelevant. For example, under default lighting (r_mode lit), the shadow of a Skulk hidden behind a obstacle could easily betray the Skulk. Whereas under r_mode unlit, it is not rendered.<!--QuoteEnd--></div><!--QuoteEEnd-->
The case with the skulk behind a crate betrayed by its own shadow is true but very specific and does not excuse the use of r_mode unlit. Its cheap to have full bright rooms no matter what. Also keep in mind that apiens lose the advantage of killing powernodes. With this command rines can aim just as fine in an unpowered room as if it was powered. Also the lifefOrms have way better contrast to the background which makes aiming easier again.
However... I agree with you about the problematic of hiding in shadows, when these are clientside determined.
Dolce & Lebowski
While FPS gains are still a valuable way of visualising data to the end user, you should technically describe performance gains in terms of milliseconds (1/FPS). E.g. (1/70)-(1/80)= decrease in time taken to render one frame. Then convert to ms. Compare this to a gain from 20 FPS to 25 FPS and you can see that top-end gains are far more neglible.
Beware of referring to percentage increases when dealing with framerate as it's not linear.
Before each test you should close all other programs, in case performance in the game for your setup is indeed CPU limited.
So in order to have more accurate info, make sure you restart NS2 before loading the map and doing your tests.
You also have to make sure that you don't somehow look out of the map, so avoid moving in ready room (type j1) and in marine start until the game is really ready, because otherwise some lag when running into teamjoin or looking around after joining might show the map void for a splitsecond and ruin your fps somehow.
@dePARA
I agree and I think UWE should eventually make it a cheat command. However, if you're <10 fps, it seems to be the single best command to increase your fps. However, I think people should try using all of the other commands (r_shadows, r_instancing, r_bloom, and r_flash) first and only resort to r_mode unlit if their fps is still unplayable. Quite simply, anyone using r_mode unlit right now to get a competitive advantage is only hurting themselves later when UWE fixes the issue.
@subshadow
I have a (very) powerful system :) I was actually hoping my results would help people like you with less powerful systems to show which commands/changes are the best to improve fps. I'd recommend turning off r_shadows, r_bloom, r_flash, and r_instancing and seeing how much it helps. Only resort to r_mode unlit if you're still having low framerate issues.
@Crispy
Good points! I want to try this benchmarking again on my laptop (which is much, much less powerful) and those are things I need to be more careful about considering. I did have background programs running when I did the benchmarking (fraps, of course, but a few others I probably didn't need) that may have effected the results. However, I did this benchmarking in one sitting, so it should still be consistent. As such, the results should be viewed as relative rather than absolute (i.e. r_mode unlit might not give a +4.66 fps increase for all setups/conditions, but I expect it to give a greater increase than r_instancing false).
@pSyk0mAn
I tried to be as particular as possible to ensure that I didn't have those issues, but got a representative selection of summit (which is why I did a loop through most of the major sections). I understand the spot issue too. For example, r_atmospherics true only seemed to be an issue in alien start (although those god rays are awesome and I can't wait till their optimized :)). Since I was just testing various framerate improvement changes, the map choice was basically arbitrary (summit is the most popular right now). I just needed to be consistent (same map and path each time).
Personally, the biggest fps gain I've had was when I changed resolution to 800*600. It's not pretty, but it rarely drops below 30 fps. Strangely, I get some huge fps-drops at Alien Start in ns_summit. Anyone else having this?
<ol type='1'><li>This benchmark does not represent actual gameplay, but I expect the general trends in fps improvements to be similar to an actual match</li><li>My setup is pretty powerful and I’m unsure how other hardware will handle the different improvements (for example, I believe my CPU is the bottleneck, but that might not be true for someone with a powerful CPU, but poor GPU).</li></ol><!--QuoteEnd--></div><!--QuoteEEnd-->
meh, the first one is a bit of a dealbreaker, atleast on my system.
i had a
i7 920 @ 4,5
6gb 1333mhz ddr3 ram
580GTX
Yet sometimes the game still has its lagspikes, mostly when in combat with a lot of players, even when I'm playing on low ping servers (sub 40ms).
Also often have a lagspike when im alien and i jump over the railing in vent to marine spawn on summit.
So honestly, if your playing with <10/<20 fps to begin with your screwed anyway =/ unless its only my system, but that seems unlikely.
i had a
i7 920 @ 4,5
6gb 1333mhz ddr3 ram
580GTX
Yet sometimes the game still has its lagspikes, mostly when in combat with a lot of players, even when I'm playing on low ping servers (sub 40ms).
Also often have a lagspike when im alien and i jump over the railing in vent to marine spawn on summit.
So honestly, if your playing with <10/<20 fps to begin with your screwed anyway =/ unless its only my system, but that seems unlikely.<!--QuoteEnd--></div><!--QuoteEEnd-->
Are you sure its a framerate issue and not a server issue? Unless you're in the 10-20fps range, its likely the server is the bottleneck (i.e. the server tickrate drops to <10 ticks/s). The effect looks the same as low framerate (i.e. a lagspike), but its totally unrelated to your framerate. To check this in-game, type r_stats (shows framerate) and net_stats (shows server rate = server tick rate) in console. When a lagspike occurs, you can check your fps and the server rate to see which one is lower and, therefore, the more likely culprit of the lagspike.
Also often have a lagspike when im alien and i jump over the railing in vent to marine spawn on summit.<!--QuoteEnd--></div><!--QuoteEEnd-->
This is mainly server side. Although FPS drop does happen in combat, it's more the calculations that the server has to do when people are shooting each other or in the same place that seems to spike calculations. You might have seen people talk about tickrate, this is a much better indication of server quality than ping. Ping is almost always just where the server is situated, for example a server 1000 km away, will have a much higher ping than one 10 km away.
The lagspike when you jump over the railing is probably because when you do, you load the entire of marine start, and a little bit of the corridor towards flight control. And there are a LOT of structures in marine start when you tend to jump over that railing. It happens slightly slower when going down the stairs, so that's less noticable
On our server (HBZ) this option set back to default by the server, because of that.<!--QuoteEnd--></div><!--QuoteEEnd-->
For the record, this is also considered a cheat by clan 156, and members are encouraged not to use it even during casual play (so it isn't a crutch). And it is totally forbidden during scrims, of course.
Some other tips:
Allow the framerate to recover after accessing the console before recording static FPS results. The framerate dips quite significantly while the console is overlaying the game (probably because it does so with only partial opacity).
It would be better for performance metrics to be taken on a lower-spec system (the min-spec and target spec systems, ideally). Occasionally you'll want to look at how higher spec systems are running to ensure you're utilising both/all GPUs efficiently, but you can make better judgements on where to optimise from testing on your min-spec.
E.g. here are my results* compared to yours:
<!--c1--><div class='codetop'>CODE</div><div class='codemain'><!--ec1-->Setting | Framerate | Increase (frames/s) | Decrease (ms)
====================================================================
default 72.35 0 0
r_mode unlit 77.01 4.66 0.84
r_flash false 76.29 3.94 0.71
r_shadows false 75.61 3.26 0.60
r_bloom false 75.52 3.17 0.58
r_instancing false 74.36 2.01 0.37
--------------------------------------------------------------------
default 30 0 0
r_mode unlit 33 3 3.03
r_flash false 34 4 3.92
r_shadows false 33 3 3.03
r_bloom false 34 4 3.92
r_instancing false 35 5 4.76<!--c2--></div><!--ec2-->
For a solid 60 FPS the frame shouldn't take any longer than 16.6 milliseconds. You can see that, for r_mode unlit, an increase of 4.66 FPS at ~70 FPS is a saving of 0.83ms, whereas the same setting achieves an increase of 3 FPS at ~30 FPS, which is a saving of 3.03ms. You can see that the 5 settings listed are costing over 16.6ms (although I'm not sure if any of those settings would be subsets of r_mode unlit).
* Note my results are not accurate because I didn't average them over time, I just sat still. I was also running at a different resolution, but the results don't need to be too accurate to illustrate the point I'm making.
Finally, gameplay will not only impact the CPU (more stuff happening) but also the GPU (effects, decals, more models on screen). However, metrics taken from actual gameplay can vary wildly from player to player and game to game. Without demos, it would require a scripted map and bots to simulate actual gameplay in order to get accurate results between builds, which is quite a length to go to.
onProcessMove()
This little function seems to be called more than once per frame (which is unnecessary I believe) and doing so is causing a lot of lag, in fact, based on my tests I dropped 20fps (70 -> 50) when looking at non-complex geometry. Fortunately, onProcessMove is run solely clientside (this is also run serverside, but slightly differently) so multiple other players does not incur additional fps penalties as far as I can tell.
-----
Cinematics (or rather cinematics added dynamically, ie. not built into the map)
Rifle shooting, bullet ricochets, flames, spores, blink, etc. All contribute to epic lagz. Again, in my tests it caused upwards of 10fps drops. This is exasperated by online play because there are often multiple marines firing weapons and lerks gassing them out and fades blinking in and out all the time. This could cause very many new cinematics to be created causing a heavy impact on performance.
I wonder if these commonly used cinematics are pre-cached, and if not, would that make a difference?
----
+1, this should only be allowed through using a value of "1" for cheats on a server.
Why do you need that when you have a minimap showing all the enemies, should be forbidden too :)
It doesn't show all the enemies at all times. I don't rely on little kinks in the system to use to my advantage in order to get kills. I rarely open the mini-map and I rarely use the Alien View. I want to develop skill through using the bare minimum, so when NS2 develops a competitive side, the features that people are (in my opinion) exploiting now are disabled, I won't see a drop in my performance.
This little function seems to be called more than once per frame (which is unnecessary I believe) and doing so is causing a lot of lag, in fact, based on my tests I dropped 20fps (70 -> 50) when looking at non-complex geometry. Fortunately, onProcessMove is run solely clientside (this is also run serverside, but slightly differently) so multiple other players does not incur additional fps penalties as far as I can tell.<!--QuoteEnd--></div><!--QuoteEEnd-->
OnProcessMove is run on both the client and the server. Calling it multiple times per frame on the client is necessary for client-side prediction (you can turn this off by typing "prediction false" at the console and you'll see why this is important!) OnProcessMove does take a lot of time -- and is something ongoing optimization will focus on -- but it really is a major part of the game logic and therefore should be one of the biggest time sinks.
omg, I can't +1 this anymore. TF2 has "cl_autohelp" as well as many other attributes (yes, I KNOW HL Source engine is heavily developed, I am using it at a reference). I also feel that the console should only come down 50% of your screen, not 100%. Most of all, I would really like to re-position where my net_stats position up on my screen. Having it overlay my map location is rather annoying. I wish we had the ability to relocate our console using the x/y positions.
(I know I won't get a response but I'll give it a shot!)
Any idea on when something like this will be implemented? I'm in no rush, but I would be interested in knowing if it's even planned.
Since there is no timedemo ability, or at least a demo recording ability like other engines (AFAIK).
Would it be worthwhile for me to create a bit of software that controls NS2 for you to ensure that it does the same test every time?
Perhaps make it a wrap around fraps and do all of the leg work for you?
That way people can just download and run it to compare benchmarks?
I actually work in software testing automation so it's right up my alley.
I understand the importance of comparing apples to apples.
Ideally it would be fantastic to actually code some kind of demo recording ability in lua and submit it.
But that will probably take me far too long to maintain interest heh.
Since there is no timedemo ability, or at least a demo recording ability like other engines (AFAIK).
Would it be worthwhile for me to create a bit of software that controls NS2 for you to ensure that it does the same test every time?
Perhaps make it a wrap around fraps and do all of the leg work for you?
That way people can just download and run it to compare benchmarks?
I actually work in software testing automation so it's right up my alley.
I understand the importance of comparing apples to apples.
Ideally it would be fantastic to actually code some kind of demo recording ability in lua and submit it.
But that will probably take me far too long to maintain interest heh.<!--QuoteEnd--></div><!--QuoteEEnd-->
If you create it, I will test it bud! :)
Seconded.
The benchmarking I did was a bit tedious, so anything to automate the process would be greatly appreciated.
- Bug reporting (infinitely useful)
- Performance and compatibility testing (use the same demo on all hardware setups to get accurate comparisons)
- Official media footage (your time is precious, not to mention you can use this to prep media for existing and potential investors)
- Fanmade, viral media footage (free advertising!)
- Cheat/exploit investigation (the community can self-police, putting less pressure on post-sales support)
- Competitive gaming (skill/frag vids and clan match footage are another form of advertising, also important for referee review)
Real competitive gaming can wait until the game is feature complete, and kinda balanced out.
And performance tests would be better with an ingame benchmark...
1. Fraps is not accessible by default to all NS2 players. An integrated demo feature would be. Less barrier to entry = more bugs and more free advertising. Downloading Fraps to capture a demo you <i>know</i> is good is more worthwhile than Frapsing every game you play 'just in case'. You'd happily demo every game you play as a matter of course, but would you really go to the effort of opening Fraps every time you play?
2. Following on from the same point, a demo is more instantaneous. Someone's cheating? Go into the console and start recording. With Fraps you'd need to Alt-tab, boot up Fraps (if you have it) and by the time you're rolling the moment may have passed. Same goes for bugs that may only occur during certain, short-lived scenarios.
3. A demo leaves a much smaller footprint on the game than Fraps running in the background (or other forms of capture for that matter). A demo being played back after the fact leaves an even smaller footprint (because the players in a demo are just actors, they're not sending any more data than what actually happened, as opposed to the server/client having to spend time working out who should do what and in what order). Essentially recording a demo will have a lesser impact on performance than live video capture.
4. A demo is a much smaller file than Fraps (think about a hundredth of the size). This makes it quicker to upload/download and it takes up less space in storage.
5. A demo requires no special compression techniques/knowledge by the end-user to reduce size to something practical. Again, barrier to entry.
6. Fraps cannot substitute as a benchmarking tool. A video is a video is a video. A demo will give different results on different machines. It puts in commands and the results are channelled through the hardware, to a degree (networking is a bit different).
7. A demo can (depending on how it's been set up) allow live debugging of a scene. You can play back a demo, pause it at a specific point and enter debug commands to better understand what's going on. A video capture is non-interactive.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->Real competitive gaming can wait until the game is feature complete, and kinda balanced out.<!--QuoteEnd--></div><!--QuoteEEnd-->Fair point, but when the game is feature complete and kinda balanced out, there may not be a competitive community. Demo recording from the outset helps it flourish from the outset.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->And performance tests would be better with an ingame benchmark...<!--QuoteEnd--></div><!--QuoteEEnd-->An in-game benchmark is more complex than you imagine. A demo can cover much of the same ground and work out better bang for buck in terms of development/maintenance time when you consider its added benefits (see my previous list).
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->but when the game is feature complete and kinda balanced out, there may not be a competitive community. Demo recording from the outset helps it flourish from the outset.<!--QuoteEnd--></div><!--QuoteEEnd-->
When the game is unbalanced, got lots of missing features and performance is still an issue in another few months there will be a competitive community because we can put a demo file under our QQ/rage threads?
<u><b>Demo recording is important - and uwe shouldnt wait too long</b></u>, but currently at least i dont see why they should waste time and resources on it. (most things you want are possible with fraps)
When the game is unbalanced, got lots of missing features and performance is still an issue in another few months there will be a competitive community because we can put a demo file under our QQ/rage threads?
<u><b>Demo recording is important - and uwe shouldnt wait too long</b></u>, but currently at least i dont see why they should waste time and resources on it. (most things you want are possible with fraps)<!--QuoteEnd--></div><!--QuoteEEnd-->
This is something I see being delayed till after NS2 v1.0, unless someone creates a mod of it first. While fraps does generally fit our needs right now, its like using an analog camera, it does the job but most of use would still prefer a digital one.