FPS Benchmarking; An Update

ScardyBobScardyBob ScardyBob Join Date: 2009-11-25 Member: 69528Forum Admins, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow
in NS2 General Discussion
Way back yonder, I did some benchmarking to gauge how different options affected my framerate. Both the game and my system have changed since then so I thought it useful to redo the analysis. The results aren't necessarily surprising, but its good to get numbers regarding it. Results below:

SHORT ANSWER
cdIr0jZ.png

Basically, I was able to get large increases in FPS via some basic changes. The biggest benefit was from disabling or minimizing all of the options, followed by reducing the resolution, turning off ambient occlusion, and shadows. The only surprising result was that turning the particle quality down seemed to reduce my FPS.

LONG ANSWER
I wanted to be a bit more rigorous with my updated benchmarking so I combined consistent, real world gaming experience (e.g. using a demo of an actual match), hypothesis testing (e.g. t-test), and a measure of the effect size (e.g. Hedges' g). Relevant background info is provided below.

System Specs:
CPU: Intel i7 2600k (OCd to 4.7GHz)
GPU: AMD Radeon HD6950
RAM: 16GB

Base Options:
- Resolution = 1920x1080
- Display Mode = Fullscreen Windowed
- Wait for Vertical Sync = Triple Buffered
- Texture Quality = High
- Particle Quality = High
- Infestation = Rich
- Anti-Aliasing = On
- Bloom = On
- Atmospherics = On
- Anisotropic Filtering = On
- Ambient Occlusion = High
- Shadows = On
- Texture Streaming = Off
- Multicore Rendering = On

Benchmark Setup:
Roughly 5 min demo benchmarked using FRAPs with each option changed one at a time.

t-test results:
Null Hypothesis = No difference between the base and changed benchmark's fps's
Alternative Hypothesis = A difference between the base and changed benchmark's fps's
Significance Level = 95% (alpha = 0.05)
Two sample, two-tailed test
nnquJgd.png

Tornado plot of the Hedges' g values:
kHgKzDu.png

The demo and file used for this benchmark are provided below.
Demo: http://dl.dropbox.com/u/5490563/B239_bench.zip
File: http://dl.dropbox.com/u/5490563/B239_Comparisons_Summary.xlsx
Off Topic Disagree Agree 10Awesome

Comments

  • SquishpokePOOPFACESquishpokePOOPFACE -21,248 posts (ignore below) Join Date: 2012-10-31 Member: 165262Members, Reinforced - Shadow
    I understand some of those numbers.
    Off Topic Disagree Agree 2Awesome
  • YMICrazyYMICrazy Join Date: 2012-11-02 Member: 165986Members
    edited February 2013
    ScardyBob wrote: »
    The only surprising result was that turning the particle quality down seemed to reduce my FPS.

    Yup I also noticed low particles does decrease frames. Also for me, infestation on rich also increase framerates. This happened since 236. Kind of odd but eh.

    EDIT: Ran the test and infestation on minimum does increase performance. There was a spot in explore mode where turning infestation to rich gave me more frames but that is useless compared to an actual benchmark.
    Off Topic Disagree Agree Awesome
  • Ghosthree3Ghosthree3 Join Date: 2010-02-13 Member: 70557Members, Reinforced - Supporter
    They messed up minimal infest in b236, I hate it now, it's got spots missing on infested areas, looks buggy, it's just bad, not surprised it messed up fps. (I hate 3D infest, obstructs vision). Interesting that low particles lowers fps though, hilarious actually, since it was added to get people more fps.
    Off Topic Disagree Agree Awesome
  • NeokenNeoken Bruges, Belgium Join Date: 2004-03-20 Member: 27447Members, NS2 Playtester, Reinforced - Shadow, WC 2013 - Silver, Subnautica Playtester
    Well, the results mostly confirm what we knew already. The only weird thing is particle effects. And yeah, minimal infestation is messed up since b236.

    Anyway, nice work ScardyBob.

    Off Topic Disagree Agree Awesome
  • XaoXao Join Date: 2012-12-12 Member: 174840Members
    Is this a thinly veiled graph pornography thread? What the fuck do half of these words mean...

    Changing res actually increases FPS now, what is this wizardry. Be interesting to see if particles and infestation high make that much difference, rich infest is gonna kill me tho.
    Off Topic Disagree Agree Awesome
  • alzarocalzaroc Join Date: 2003-07-26 Member: 18451Members, Constellation
    edited February 2013
    Interesting, thanks for sharing!
    Off Topic Disagree Agree Awesome
  • SquishpokePOOPFACESquishpokePOOPFACE -21,248 posts (ignore below) Join Date: 2012-10-31 Member: 165262Members, Reinforced - Shadow
    edited February 2013
    The guy above me has the best avatar.
    * Please refrain from off-topic posts like these, thanks. - Angelusz*
    Off Topic 2Disagree 2Agree Awesome
  • ScardyBobScardyBob ScardyBob Join Date: 2009-11-25 Member: 69528Forum Admins, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow
    Xao wrote: »
    Is this a thinly veiled graph pornography thread? What the fuck do half of these words mean...
    Yeah, its hugely more technical than most FPS benchmarking. All the info is there for people who are interested, but the bottom line is pretty unsurprising. I figure this will be most useful for people who want higher performance, but don't want to turn off everything. You can reduce your resolution or turn off ambient occlusion to get a big boost in FPS, but still keep the game looking pretty.
    Off Topic Disagree Agree Awesome
  • xtalxtal aka X-rayCat Join Date: 2009-06-28 Member: 67961Members, Constellation, Reinforced - Supporter
    It would be nice to see how gfx options affect minimum fps (Iower quartile?) not average, cause improvement of 40-100 to 40-120 fps isn't really an improvement, at least for me :) Nice scientific job ScardyBob.
    Off Topic Disagree Agree Awesome
  • Iron PriestIron Priest Join Date: 2012-04-27 Member: 151193Members, Reinforced - Supporter
    I knew 3 years of stats courses would come in handy...
    Off Topic Disagree Agree Awesome
  • Soylent_greenSoylent_green Join Date: 2002-12-20 Member: 11220Members, Reinforced - Shadow
    I think your system is an outlier, with a higher CPU/GPU performance ratio than the mean.
    Off Topic Disagree Agree Awesome
  • SteveRockSteveRock Join Date: 2012-10-01 Member: 161215Members, NS2 Developer, Subnautica Developer
    I'd be interested in knowing how many particles effects were active in your benchmark? For example, was there a lot of combat going on?

    What happens in LQ particles is, every particle (a puff of smoke, a spore mist, etc.) gets rendered in half-res, then composited back into full res. So there's is some additional - but fixed - cost in the composite step. So I would think that with very few particles, it may well be a little slower. But with many particles, like in heavy combat, I would expect the composite cost to be overwhelmed by the rendering cost (in which case, half-res is basically 4x faster than full-res, per-particle).
    Off Topic Disagree Agree Awesome
  • SteveRockSteveRock Join Date: 2012-10-01 Member: 161215Members, NS2 Developer, Subnautica Developer
    And now that I think about it, maybe we should just not use LQ particles unless there are more than ~10 particles on screen..hmm...
    Off Topic Disagree 1Agree Awesome
  • CrushaKCrushaK Join Date: 2012-11-05 Member: 167195Members, NS2 Playtester
    How does your particle aystem work? Is it something a designer sets up in it's entirety without much room for modifications?
    Or is it like in UE3, where you have one ParticleSystem asset that consists of multiple different Emitters with each of them being responsible for a different part of the effect and with it's own properties? Optimization is here usually done by just disabling those Emitters on lower quality settings that are meant to be just additional eye candy and don't make much of a difference for gameplay if they are lacking. You could even split the same effect in two different Emitters and then only spawn half as many particles on low quality as on high quality, which could work out for stuff like the Flamethrower.
    Off Topic Disagree 1Agree Awesome
  • matsomatso Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
    The rendering thread is run in parallel with the logic thread (if you have multi-core rendering on), so in order for render options to affect your fps, the renderer thread has to be the bottleneck.

    This is usually not the case when it comes to worst-case fps, unless you have a pretty awesome CPU and a pretty awful GPU, a fairly uncommon combo.

    You can run r_stats and see if the time waiting for renderer thread gets much above 1ms - its kinda hard to see, as the variable isn't
    strictly how much you wait for the renderer thread, it includes one garbage collection run on the main world as well, so it keeps flickering
    up and down. I'll see if I can get that changed.

    Off Topic Disagree Agree Awesome
  • soccerguy243soccerguy243 Join Date: 2012-12-22 Member: 175920Members, WC 2013 - Supporter
    That's the testing I needed! I will see if i can make some changes to get my FPS up a bit. Its as low as 30fps when the game gets busy.

    Off Topic Disagree Agree Awesome
  • CLARK_KENTCLARK_KENT Vancouver, Canada Join Date: 2002-11-21 Member: 9508Members, Reinforced - Silver
    edited February 2013
    ScardyBob, I was curious as to your thoughts on PhysX testing? I've tried searching, but I have found nothing empirically conclusive yet.

    Is it better on or off? If on, is it better to have your CPU or GPU process it?

    I can guess what your answers might be, but I was wondering if you've ever benchmarked it? I know you run AMD, but hey, maybe you had once tested on nVidia. :)
    Off Topic Disagree Agree Awesome
  • tarquinbbtarquinbb Join Date: 2012-11-03 Member: 166314Members
    this is actually really handy to know.

    i like to optimize graphical quality and performance, but never bothered to test everything myself :p
    Off Topic Disagree Agree Awesome
  • YashokiYashoki Join Date: 2006-12-26 Member: 59256Members
    One thing I have noticed is that turning down your game resolution does wonders for the FPS, but if it's not native to your monitor (I have a 1920x1080 monitor) the game looks absolutely disgusting, is there anything we can do about that?
    Off Topic Disagree Agree Awesome
  • tarquinbbtarquinbb Join Date: 2012-11-03 Member: 166314Members
    Yashoki wrote: »
    One thing I have noticed is that turning down your game resolution does wonders for the FPS, but if it's not native to your monitor (I have a 1920x1080 monitor) the game looks absolutely disgusting, is there anything we can do about that?

    i don't think so... unless you have a CRT monitor which can actually do multiple resolutions :/
    Off Topic Disagree Agree Awesome
  • OgraitOgrait Join Date: 2012-10-29 Member: 164306Members
    Yashoki wrote: »
    One thing I have noticed is that turning down your game resolution does wonders for the FPS, but if it's not native to your monitor (I have a 1920x1080 monitor) the game looks absolutely disgusting, is there anything we can do about that?

    Not true. Just need time to eye adapt low res. I have 1024x768, and its good. Full hd monitor. Fps around 15-80 with my crappy machine. Around 35-45 most of times.


    Interesting results scardybob.

    Off Topic Disagree Agree Awesome
  • DestherDesther Join Date: 2012-10-31 Member: 165195Members
    edited February 2013
    One thing that they do in Planetside 2 is to have separate resolutions for your UI and world. I can have nice and high resolution HUD and have the game at much lower resolution for performance. In NS2, if you drop your resolution down then the UI becomes ugly and obstructive, things like chat and comm UI taking up half the screen.
    Off Topic Disagree Agree Awesome
  • RainyCaturdayRainyCaturday Join Date: 2013-02-19 Member: 183202Members
    edited February 2013
    Furious_Gorge wrote: »
    ScardyBob, I was curious as to your thoughts on PhysX testing? I've tried searching, but I have found nothing empirically conclusive yet.

    Is it better on or off? If on, is it better to have your CPU or GPU process it?

    I can guess what your answers might be, but I was wondering if you've ever benchmarked it? I know you run AMD, but hey, maybe you had once tested on nVidia. :)

    I am actually curious about this and did some rough benchmarking just looking at the numbers with PhysX set to CPU and GPU. No conclusive results, as I havent benched this.

    Im currently running PhysX on my cpu.

    Some simple benching shows that I only gained a few fps from switching PhysX from CPU to GPU. This was by no means an intense benchmark designed to really test PhysX but there were quite a few ragdoll deaths in the 2 minutes I benched for.
    
PhysX on CPU
Frames, Time (ms), Min, Max, Avg
  8216,    120000,  43, 181, 68.467

PhysX on GPU
Frames, Time (ms), Min, Max, Avg
  8302,    120000,  44, 185, 69.183
    Off Topic Disagree Agree 1Awesome
  • XariusXarius Join Date: 2003-12-21 Member: 24630Members, Reinforced - Supporter
    Did any playtesters do benchmarking for the next update?
    Off Topic Disagree Agree Awesome
  • RainyCaturdayRainyCaturday Join Date: 2013-02-19 Member: 183202Members
    Xarius wrote: »
    Did any playtesters do benchmarking for the next update?

    I'll do another mini bench to compare as I literally did those benches 1 hour ago.
    Off Topic Disagree Agree Awesome
  • YashokiYashoki Join Date: 2006-12-26 Member: 59256Members
    Desther wrote: »
    One thing that they do in Planetside 2 is to have separate resolutions for your UI and world. I can have nice and high resolution HUD and have the game at much lower resolution for performance. In NS2, if you drop your resolution down then the UI becomes ugly and obstructive, things like chat and comm UI taking up half the screen.

    This is what I was actually refering to. Maybe I'm just used to the crispyness of the higher resolutions, but turning down this setting makes the game look muddy to me, and the UI definitely leaves something to be desired.

    This is kind of a no brainer, but turning down bloom, antialiasing, and shadows definintely gave me a 20 frame boost.
    Off Topic Disagree Agree Awesome
Sign In or Register to comment.