Performance benchmarks for NS2 alpha build 152

AvalonAvalon Join Date: 2007-03-04 Member: 60224Members Posts: 239
edited November 2010 in NS2 General Discussion
I decided that "laggy" wasn't a good enough metric for me, so I wanted to benchmark exactly how fast or slow NS2 was running on my machine. On top of that, I wanted to see what various things affected fps, such as resolution, in game quality settings, gpu overclocks, and cpu overclocks. Here's what I found....

*System specs*
Intel E8400 Core2Duo
4GB PC2-6400 DDR2 RAM
Gigabyte EP-35 DS3L
Visiontek Radeon HD4850 512MB
ATI Catalyst 9.11 driver set (fall 2009 :D)
Western Digital 1TB Black Edition 7200RPM HDD
Windows 7 Ultimate 64bit

The benchmark...
Spawn as a marine, run to the tram doorway, and start recording fps with fraps. Walk out into the tram long hall, and take a left towards alien hive. Walk into alien hive room and start shooting hive with rifle. Unload 2 clips. Pull out handgun, unload 2 clips. Reload rifle, unload two clips. Benchmark ends at 60 seconds at the end of the last pair of rifle clips.

1) My first run, at overclocked CPU settings, default GPU settings
1680x1050, high settings, e8400 @ 3.78ghz
min/avg/max
13/25.2/31 fps

2) My second run, I wanted to see what CPU speeds would do here, so I ran my CPU at stock
1680x1050, high settings, e8400 @ 3ghz stock
min/avg/max
9/22/27 fps

3) My third run, to see what effect overclocking my GPU would have
1680x1050, high settings, e8400 @ 3ghz stock, 4850 @ 700mhz (11% overclock)
min/avg/max
11/22.5/29 fps

4) My fourth run, to see what having everything overclocked would do
1680x1050, high settings, e8400 @ 3.78ghz, 4850 @ 700mhz (11% overclock)
13/25.3/31fps

5) My fifth run, to see what the best fps I could get at my native resolution was by reducing in game quality
1680x1050, awful settings, e8400 @ 3.78ghz, 4850 @ 700mhz
min/avg/max
22/27.4/33 fps

6) My sixth run, to see what the best fps I could possibly get at all while sticking with my monitor's native aspect ratio
1280x800, awful settings, e8400 @ 3.78ghz, 4850 @ 700mhz
min/avg/max
21/27/32 fps

From this, I can gather that the GPU is not being worked enough. Increasing the resolution is supposed to reduce the load on the CPU and increase it on the GPU. Vice versa, reducing resolution is supposed to increase the load on the CPU versus the GPU. The fact that I did not get a perceptible fps increase while overclocking my GPU or reducing my resolution proves this. I know a few of you have documented that your GPUs weren't even being used anywhere near 100%.

I did note that reducing the quality while leaving all else the same did result in a noticeable increase in fps. The visible changes to the game seemed to lie with shadows. Since we've determined the GPU to not be having much of an impact on fps, could we thus conclude that the engine is relying too heavily on the CPU for rendering shadows?
«1

Comments

  • HughHugh Cameraman San Francisco, CA Join Date: 2010-04-18 Member: 71444Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Reinforced - Silver, Reinforced - Onos, WC 2013 - Shadow, Subnautica Developer, Pistachionauts Posts: 1,779 admin
    This is very useful information. While there's no point benchmarking an Alpha for general performance, identifying particular CPU/GPU usage characteristics is worthwhile indeed. Nice work Avalon! I wasn't sure if you meant 'sprint' or just 'jog' to Alien Hive, I sprinted.

    With an X6 1090T, 5870, and 8Gb 1333 (All stock), my results were

    1) 1920x1080 Maxed
    Min/Avg/Max
    13/23.25/26


    2) 1280x720 Maxed
    Min/Avg/Max
    19/23.65/26

    CPU bound indeed.
    The fastest way to contact me is via @hugh_jeremy on Twitter! Add me on Steam: Strayan. I do not check forum PMs.
    Keep up to date on what is going on with: Unknown Worlds | Natural Selection 2 | Subnautica
    Chat with the UWE team on Twitter | The best way to be notified about Subnautica updates is to sign up to our development newsletters
  • juicejuice Join Date: 2003-01-28 Member: 12886Members, Constellation Posts: 1,044
    edited November 2010
    Yes but what happens when you go extreme low resolution, 640x480? Fps is still bad. So there's something wrong there with the engine... like it's CPU bound and it's already snagged up even at super low resolution by something that doesn't scale with the number of pixels...
  • PlasmaPlasma Join Date: 2003-04-26 Member: 15855Members, Constellation, Squad Five Blue Posts: 605
    I agree juice.

    It's interesting to me that I get a solid 60+fps at the main menu; but when I am in-game my fps drops to what people have posted here - ~30fps.

    Both have similar #s of faces/triangles etc being rendered according to r_stats.

    I wonder if its something like the 'game logic loop' for example (which does not exist on the main menu) that is responsible for the poor FPS.
  • devicenulldevicenull Join Date: 2003-04-30 Member: 15967Members, NS2 Playtester, Squad Five Blue Posts: 1,540
    edited November 2010
    I suspect they haven't correctly implemented the luajit code. I did the following:
    CODE
    function OnCommandVersion()
        Shared.Message("Jit is " .. tostring(jit.status()))
        jit.off()
    end


    When executing this command twice, I see the following:
    CODE
    Jit is true
    Jit is false


    which is expected. What I do not see is a noticeable FPS drop. Based on the graphs at http://luajit.org/performance.html , we should see a significant performance drop when Lua is disabled.

    Really, I would have expected that Lua would have been the bottleneck here, but apparently it's not even close. I wonder wtf else it's doing that could be sucking up a ton of cpu.

    Edit: Ran NS2 through a profiler, seems the time is mostly divded between:
    ntoskrnl (windows kernel)
    Engine.dll
    PhysXCooking.dll (physx cooking me dinner!)

    For the kernel, seems to be calling thread synchronization functions a lot, which isn't something we can do much about. The engine is spending a lot of it's time in M4::Model::GetBoneCoords. I wonder if we can find a way to play with significantly reduced models, to see if that helps any. PhysX is spending a lot of it's time in NxCookConvexMesh. I don't really know enough about physx (and the nvidia forums are down) to say what that is doing.
    Post edited by Unknown User on
    Rent NS2 Dedicated Servers at www.gameservers.com
  • AvalonAvalon Join Date: 2007-03-04 Member: 60224Members Posts: 239
    edited November 2010
    QUOTE (juice @ Nov 2 2010, 05:19 AM) »
    Yes but what happens when you go extreme low resolution, 640x480? Fps is still bad. So there's something wrong there with the engine... like it's CPU bound and it's already snagged up even at super low resolution by something that doesn't scale with the number of pixels...


    Why would I run it at 640x480 when dropping from 1680x1050 to 1280x800 did nothing? I'm already CPU bound, no use in running an ugly resolution that I don't even think my monitor can display :D

    Anyway, it's not like it will be this way forever. I was just trying to explore what different things did to the engine in its current state. I also found it very strange when I was sitting in the ready room, getting 45fps on awful settings, then turned straight into a wall and continued to get 45fps. Staring at a wall should make fps skyrocket, since there is less to draw.

    I am going to benchmark every build from here on out to see what improvements are realized, if any. Despite how it looks, I am actually not most concerned about fps at this point, but it is the only metric I have control over measuring.
  • Mkk_BitestuffMkk_Bitestuff Join Date: 2003-01-17 Member: 12407Members Posts: 210
    Game is CPU bound, running game at 1680x1050, gpu saw about 30-40% load (This is on a GTX 460). I get the same FPS i used to get with my 4850. Also, game really only seems to peg out one core, with the other mostly doing nothing. Seems that tasks need to be further split among available cores
  • xposed-xposed- Join Date: 2007-09-23 Member: 62412Members, Constellation Posts: 236
    "Benchmark" and "alpha" should not be in the same sentence.
    image xposed
  • nUfl0wnUfl0w Join Date: 2005-02-25 Member: 42412Members Posts: 235
    edited November 2010
    Hi

    i've experienced something similiar related to the models. I didnt do such a structured benchmark you did. I've started a lan server, joined aliens and enabled cheats. then went to marine base. staying there alone with noone else on the server i had around 26-30 fps. then i spawned eight extra skulks by typing "give skulk" in the console. they appeared next to my position in base. by doing that my fps dropped to 8-9 fps.

    this experience is the same people talked about.
    server with low playercount=> more or less playable
    server with higher playercount=>unplayable

    I dont know if this information helps anyone, but maybe you can setup a structured benchmark with that information to proof it, if the information is relevant to track down the problems


    cheers
    Post edited by Unknown User on
    cheers - Dolce & Lebowski
  • PlasmaPlasma Join Date: 2003-04-26 Member: 15855Members, Constellation, Squad Five Blue Posts: 605
    That's interesting. Do you have a list of console commands somewhere?

    I'd be interested in removing all models etc from the game and other things like that to see if FPS is boosted or not.
  • HughHugh Cameraman San Francisco, CA Join Date: 2010-04-18 Member: 71444Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Reinforced - Silver, Reinforced - Onos, WC 2013 - Shadow, Subnautica Developer, Pistachionauts Posts: 1,779 admin
    QUOTE (xposed- @ Nov 2 2010, 06:29 PM) »
    "Benchmark" and "alpha" should not be in the same sentence.


    This is not helpful. In this alpha, they should be in the same sentence. Benchmarking is providing data that will help solve an engine problem. No one is trying to max FPS numbers here.

    Your test got me thinking Avalon. I jumped onto a sever online (NOT a LAN) with one person sitting in the RR, and assets spawned all over the map. My FPS was a solid 50, looking in any direction. Using r_stats, it's obvious that FPS will not increase or decrease no matter the draw calls, primitives, lights, models or anything.

    I think that once this issue is sorted out, and I am guessing (as a very uninformed punter) that it is a combination of game logic and occlusion problems, FPS in NS2 is going to be very, very good.
    The fastest way to contact me is via @hugh_jeremy on Twitter! Add me on Steam: Strayan. I do not check forum PMs.
    Keep up to date on what is going on with: Unknown Worlds | Natural Selection 2 | Subnautica
    Chat with the UWE team on Twitter | The best way to be notified about Subnautica updates is to sign up to our development newsletters
  • kababkabab Join Date: 2003-12-15 Member: 24384Members, Constellation Posts: 560
    QUOTE (devicenull @ Nov 2 2010, 04:28 PM) »
    For the kernel, seems to be calling thread synchronization functions a lot, which isn't something we can do much about. The engine is spending a lot of it's time in M4::Model::GetBoneCoords. I wonder if we can find a way to play with significantly reduced models, to see if that helps any. PhysX is spending a lot of it's time in NxCookConvexMesh. I don't really know enough about physx (and the nvidia forums are down) to say what that is doing.

    Well i suspect given the item in the progress tracker that they are using Physx for all their collisions which is probably quiet expensive hence separating them out and having a more simplified collision detection system for general player movement?

    Perhaps also they are not doing the character skinning on the GPU doing the skinning on the CPU is very slowed compared to GPU...

    Max is a smart dude though so i'm sure he has thought of these things :)
  • PlasmaPlasma Join Date: 2003-04-26 Member: 15855Members, Constellation, Squad Five Blue Posts: 605
    Hm - I'm not a mapper; but could we have a simple test map created that was just a simple box (for example), and then introduce a model and see if FPS takes a dive?
  • PlasmaPlasma Join Date: 2003-04-26 Member: 15855Members, Constellation, Squad Five Blue Posts: 605
    I've made a simple map that only contains a flat floor; no lights, and a spawn position.

    image

    FPS are still 30fps as you can see, exactly like the ns_tram etc maps that are included with the game that obviously contain a lot more detail.

    This to me makes me think the rendering engine is fine. As mentioned earlier, the main spawn menu renders for me at 60fps too.

    I then actually spawned in the menu.level map (I added a ready room spawn marker using the editor) and in-game I got ~30fps too (instead of 60fps like at the game menu).

    As mentioned earlier, I agree with others that there must be some game logic code slowing down performance.

    For example, perhaps its LUA, or the networking code, or the 'Is the player pressing a keyboard button yet?' code, etc that is responsible, given a basic map renders as fast as a complex map.
  • w0dk4w0dk4 Join Date: 2008-04-22 Member: 64129Members, Constellation, Reinforced - Shadow Posts: 351 Fully active user
    edited November 2010
    It's been stated by Max that a massive bottleneck is actually the Flash GUI, which is ridiculously slow (10+ ms per frame).

    Once that bottleneck is sorted out, I guess it'll be easier to identify further bottlenecks.
    ArmoryChamber.net [ConstellationSlots/AdvancedBalance]
  • WheeeeWheeee Join Date: 2003-02-18 Member: 13713Members, Reinforced - Shadow Posts: 4,262 Fully active user
    edited November 2010
    according to the documentation, the PhysX call converts mesh data into a bitstream that's optimized for collision detection.

    http://www.assembla.com/code/pxcode/subver...ude/NxCooking.h

    QUOTE
    To create a triangle mesh object(unlike previous versions) it is necessary to first 'cook' the mesh data into
    a form which allows the SDK to perform efficient collision detection.

    NxCookTriangleMesh() and NxCookConvexMesh() allow a mesh description to be cooked into a binary stream
    suitable for loading and performing collision detection at runtime.

    NxCookConvex requires the input mesh to form a closed convex volume. This allows more efficient and robust
    collision detection. The input mesh is not validated to make sure that the mesh is convex.


    CODE
            virtual bool  NxCookConvexMesh(const NxConvexMeshDesc& desc, NxStream& stream)= 0;


            /**
            \brief Cooks a triangle mesh to a ClothMesh.

            \param desc The cloth mesh descriptor on which the generation of the cooked mesh depends.
            \param stream The stream the cooked mesh is written to.
            \return True if cooking was successful
            */


    i'm not really experienced in programming so i'm not qualified to speak on this.

    on another note, i decided to run out and do a little testing;

    noticed a little something interesting in the ready room of tram...when i look at the floor, i get ~22 fps. when i look at the ceiling, even though r_stats says i see more of everything (more lights, more geometry, more meshes, more primitives and draw calls, more face sets and occlusion queries), i get 38-40 fps. only thing that seemed different was the ceiling didn't seem to reflect much light, while there was a lot of lighting being reflected on the floor.

    ran out into the skybox in spectator mode. max fps achieved was ~65, with 0 of everything except draw calls, primitives (~3500), and occlusion queries(1). tried to vary the amount of stuff seen by changing perspective. looked at the map, minimum framerate was about 6-10fps until i ran far enough away that the engine distance clipped portions of the map (it was pretty far before this happened). i'm guessing scaling isn't optimized yet.

    also the HUD does make a huge difference. as does the muzzle flash - looking at a wall, 25 fps. looking at a wall while firing, 20-22 fps. *edit* maybe this drop in fps was due to flash updating the ammo count. not really sure.

    system specs:windows xp sp 3
    amd phenom ii x4 @ ~3.4 ghz (stock)
    HIS ATi radeon 4850 HD 1gig (stock)
    3g (4g, but windows xp can only use 3) ddr3 1333
    ati catalyst driver 8.780.0.0 (sept '10)
    1680x1050 res, medium qualit
    QUOTE (EEK)
    Don't assume that because I said something it means I actually was thinking that the core of the sun was going to be replaced with hot dogs.
  • kababkabab Join Date: 2003-12-15 Member: 24384Members, Constellation Posts: 560
    Another thing to they could also write a dx10/10.1/11 render paths... DX11 would give significant performance increases and we could have MSAA!!

    Having a Dx9 and Dx11 render path wouldn't be unreasonable.
  • NurEinMenschNurEinMensch Join Date: 2003-02-26 Member: 14056Members, Constellation Posts: 1,352
    This is a great thread! Two thumbs up!
    This is the kind of thing that goes on behind close doors on normal game releases, where they are optimizing it right up until the end. You guys are just getting to play with a version of the game that normally only the developers see.

    --Cory
  • AvalonAvalon Join Date: 2007-03-04 Member: 60224Members Posts: 239
    edited November 2010
    QUOTE (xposed- @ Nov 2 2010, 08:29 AM) »
    "Benchmark" and "alpha" should not be in the same sentence.


    There's no reason for that statement.

    Keep the discussing going guys, there's some good stuff in here. I personally enjoy seeing where the engine is at right now, and honestly am impressed at how well it's handling at such an early stage. It seems like there is a lot that can be done to improve performance, so I'm pretty excited to see how further refinements pan out, and like I said, will document any changes between builds.
    Post edited by Unknown User on
  • FuzionMonkeyFuzionMonkey Join Date: 2005-05-04 Member: 50889Members Posts: 127
    QUOTE (devicenull @ Nov 2 2010, 01:28 AM) »
    I suspect they haven't correctly implemented the luajit code. I did the following:
    CODE
    function OnCommandVersion()
        Shared.Message("Jit is " .. tostring(jit.status()))
        jit.off()
    end


    When executing this command twice, I see the following:
    CODE
    Jit is true
    Jit is false


    which is expected. What I do not see is a noticeable FPS drop. Based on the graphs at http://luajit.org/performance.html , we should see a significant performance drop when Lua is disabled.


    That really is bizarre.

    Lots of interesting stuff in this thread. I'd like to see what framerates someone with serious beefy hardware would get. I mean Crysis isn't the real test. The real test is if you can play NS2.
  • kababkabab Join Date: 2003-12-15 Member: 24384Members, Constellation Posts: 560
    QUOTE (FuzionMonkey @ Nov 3 2010, 02:13 AM) »
    That really is bizarre.

    Lots of interesting stuff in this thread. I'd like to see what framerates someone with serious beefy hardware would get. I mean Crysis isn't the real test. The real test is if you can play NS2.
    I got a i7 950 and a 5850 not the best of the best but pretty dam fast...

    Menu screen i get 100fps

    Connected to a server i get 30fps

    Playing lan (by myself) i get 40-50 fps
  • DY357LXDY357LX Playing since day 1. Still can't Comm. England Join Date: 2002-10-27 Member: 1651Members, Constellation Posts: 2,309 Fully active user
    edited November 2010
    Okay, great thread, some interesting stuff so I thought I'd just throw 2 quick things in:

    1: Is there a console command or a .LUA file you can edit to remove the flash-based HUD for FPS texting? (I think it's r_drawhud 0 in Source games etc.)
    2: If enough of us use @NS2 on twitter to point Flayra in the direction of this thread we might be: A: sharing useful info + B: get some info from UWE's perspective.
    "And so is the Golden City blackened With each step you take in my Hall. Marvel at perfection, for it is fleeting. You have brought Sin to Heaven And doom upon all the world."
  • juicejuice Join Date: 2003-01-28 Member: 12886Members, Constellation Posts: 1,044
    A totally empty map gives half the fps of the menu screen? Now that is messed up.
  • AvalonAvalon Join Date: 2007-03-04 Member: 60224Members Posts: 239
    Menu screen looks so good, too :O
  • Killpo1Killpo1 Join Date: 2009-11-13 Member: 69373Members Posts: 66
    I'm on a intel i7 1.6 ghz. Sure it's not a fast clock speed but it makes up by being a quad core with 3 fake cores. I think the game is just really heavy on one processor because generally during the game one of my processors is being overworked while the rest are generally under worked. Anyone else notice this? My graphics card is ati mobility Radeon HD 5400 Series. 6 gigs of RAM.

    It's not the graphics card because I've run awful/med/high and that does not seem to affect performance at all.
    RAM isn't an issue because loading times are fine.
    It almost like internet lag during gameplay, for instance, when it very laggy and connection is bad you start jumping around and almost lose control of your character, this is what happens to me generally.
  • DarkhandDarkhand Join Date: 2002-11-01 Member: 3012Members Posts: 102
    Someone mentioned bones or a skeleton?

    I know from my experience with 3d animation that skeletons are usually only retained during the preliminary phases. IE Once you have completed your model and rendered it you're left with just several different animation sequences.

    For Example

    You shoot your gun - Game plays the fire animation
    You walk - Game plays the walking animation
    etc.

    Unless the bones/skeleton is somehow associated with any ragdoll physics for things that happen on death I really don't think they should be present still and that would account for a huge drop in FPS because the model is still completely malleable in the final stage.

    The only thing I could think of is maybe they are having unique damage responses? Ala' Killzone 2? So the model needs to know how to coordinate the movements and the skeletons are still associated.

    Anyway I may just be rambling, wanted to throw my two cents in.
  • ZehtukaZehtuka Join Date: 2010-07-26 Member: 72853Members Posts: 11
    edited November 2010
    System:
    AMD Phenom II x3 720BE @ 3.6Ghz
    4GB DDR3
    GTX 275
    Windows 7 64-bit

    Ingame the FPS seems to be around 25-40, the graphical settings do have a massive effect on the amount of vRAM that's being used though.
  • juicejuice Join Date: 2003-01-28 Member: 12886Members, Constellation Posts: 1,044
    edited November 2010
    QUOTE (Darkhand @ Nov 2 2010, 03:08 PM) »
    Someone mentioned bones or a skeleton?

    I know from my experience with 3d animation that skeletons are usually only retained during the preliminary phases. IE Once you have completed your model and rendered it you're left with just several different animation sequences.

    Unless the bones/skeleton is somehow associated with any ragdoll physics for things that happen on death I really don't think they should be present still and that would account for a huge drop in FPS because the model is still completely malleable in the final stage.


    That's interesting. On the main menu there is a skulk running around so there would have to be a difference between how things are handled there versus in game. So maybe there is no ragdoll physics overhead on the main menu? Is there a way to set a console variable to not do ragdolls in game?
  • FocusedWolfFocusedWolf Join Date: 2005-01-09 Member: 34258Members Posts: 1,285 Advanced user
    QUOTE (Plasma @ Nov 2 2010, 12:50 AM) »
    I agree juice.

    It's interesting to me that I get a solid 60+fps at the main menu; but when I am in-game my fps drops to what people have posted here - ~30fps.

    Both have similar #s of faces/triangles etc being rendered according to r_stats.

    I wonder if its something like the 'game logic loop' for example (which does not exist on the main menu) that is responsible for the poor FPS.


    As i expected, if you open the file \natural selection 2\ns2\maps\menu.level with the map editor then you will see that the main-menu map is no bigger then 1 room. Obviously the problem is maps designed with more then 1 room :D
    The enemy cannot eat an IP, if you disable its head!
  • juicejuice Join Date: 2003-01-28 Member: 12886Members, Constellation Posts: 1,044
    QUOTE (FocusedWolf @ Nov 2 2010, 06:56 PM) »
    As i expected...


    Read Plasma's post on page 1 and you see that it is not the number of rooms.

    However, maybe
    QUOTE
    Separate collision detection from physics (server CPU usage)

    this has something to do with it? It's keeping the bones around for physics but they're slowing down regular collision detection. Or something. Even though I thought they just use rectangular bounding boxes for that. In any case the fact that you're running a server is the difference between the main menu and single room situations, is that right?
  • SgtBarlowSgtBarlow Level Designer Join Date: 2003-11-13 Member: 22749Members, NS2 Developer Posts: 1,122 Fully active user
    Max & Charlie says that the Main Menu excludes running of PhysX, All the core game mechanics, the multiple flash huds and many many more things.
    Its a minimum basic render.


    They do indeed use luaJIT and its functioning normally. You wont get a slow down if its not being hammered to death anyway?


    Like what was said, the Flash huds do take up a lot of render time and create an excessive number of draw calls and primatives. The Send Feedback text is flash.
    There is no need to spend any time trying to disable flash, I already ran some tests with it and presented the results using the engines core render that runs @ 1400 fps.
    I tried many things to speed it but as only able to squeeze another 10 fps out of it. besides that, the other reason was because its such a ballache to make the interfaces in flash.

    As for CPU core spread problems, I found this to actually be caused by the flash, when I replaced the GUI the CPU usage on core 0 dropped and raised on cores 12&3 giving an overall lower usage. Gameplay felt smoother too.
    Other than that without the flash there are still other issues they are aware of that is being delt with, going to take some time to optimize all the features that have been piled on.

    I am reading all your posts, questioning things i dont already know about and passing on anything new that is found. So keep it coming & good job all.
Sign In or Register to comment.