Map Change Times

ZEROibisZEROibis Join Date: 2009-10-30 Member: 69176Members, Constellation Posts: 1,017 Advanced user
Can someone please look into why the requirements for changing the map are so crazy. The issue is not really with how long it takes by itself but how the server process handles I/O priority. For example:
Server Process A <1min to change maps<br> Server Process B <1min to change maps<br> CS:S server A <15sec to change maps<br> CS:S server B <15sec to change maps<br>
When both CS:S server A and B change maps at the same time it takes <30 sec as one would logically expect. However, if you do a NS2 Server Process A and B at the same time you would expect it to take maybe 2min 3 at most but in reality it is like a black hole opens up on your server and it suddenly takes like 5+min to load.<br>
It is like for some reason the NS2 server process is just like a kid trying to push all the other kids cups away from the water fountain and as a result everyone gets water a lot slower b/c they are all fighting for it.

Is there something we server admins need to do to the server process so that it will get in line and order its I/O requests properly?

I can understand that there is a lot of stuff for NS2 to load but I do not understand why it appears to just tank when placed in a situation where it needs to queue for I/O resources. Perhaps there is some setting I am missing or the game just needs some programing optimizations in future builds to improve I/O management.
image
Server IP: NS2.IBISGaming.com
Server IP: NS2CO.IBISGaming.com

Comments

  • ZaggyZaggy NullPointerException The Netherlands Join Date: 2003-12-10 Member: 24214Forum Moderators, NS2 Playtester, Reinforced - Onos, Subnautica Playtester Posts: 3,654 mod
    5m is rather long, what hardware are you running the NS2 server on?
    What's in the server log during map loading?
    imageProud Dutchman image
    Join the Natural Selection IRC channel: #naturalselection on irc.gamesurge.net!
  • ZEROibisZEROibis Join Date: 2009-10-30 Member: 69176Members, Constellation Posts: 1,017 Advanced user
    edited September 2013
    Just the normal stuff it is only really long if two servers change at the same time. Instead of it taking twice as long as one would expect it takes about 5 to 10 times as long instead. To me this suggests some sort of deeper problem with the way the program is requesting the data.

    In the mean time I am going to take everything down to make sure the defrag is not ignoring any files in use ect and run it a few times to help things as much as possible.

    As I said before the normal load times are not long but when you get 2 servers changing at the same time is when things start to get out of hand.

    As for the HDD it is just a normal 7200rpm drive. During the test I just ran it took about 6 min for 2 servers to change at the same time, virtually 0% cpu usage ram at 40% usage HDD has 40% free space and was fully defragmented.
    Post edited by ZEROibis on
    image
    Server IP: NS2.IBISGaming.com
    Server IP: NS2CO.IBISGaming.com
  • ScardyBobScardyBob ScardyBob Join Date: 2009-11-25 Member: 69528Forum Admins, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow Posts: 4,983 mod
    I'm curious, do you have the servers setup in separate folders or are you running two instances from the same folder?
  • kais_4kais_4 Join Date: 2012-08-31 Member: 157127Members, Reinforced - Shadow Posts: 65
    I'm running my server off a high end SSD to counter the load times. Gotta do what ya gotta do :P
  • ZaggyZaggy NullPointerException The Netherlands Join Date: 2003-12-10 Member: 24214Forum Moderators, NS2 Playtester, Reinforced - Onos, Subnautica Playtester Posts: 3,654 mod
    ZEROibis wrote: »
    Just the normal stuff it is only really long if two servers change at the same time. Instead of it taking twice as long as one would expect it takes about 5 to 10 times as long instead. To me this suggests some sort of deeper problem with the way the program is requesting the data.

    In the mean time I am going to take everything down to make sure the defrag is not ignoring any files in use ect and run it a few times to help things as much as possible.

    As I said before the normal load times are not long but when you get 2 servers changing at the same time is when things start to get out of hand.

    As for the HDD it is just a normal 7200rpm drive. During the test I just ran it took about 6 min for 2 servers to change at the same time, virtually 0% cpu usage ram at 40% usage HDD has 40% free space and was fully defragmented.

    Wait, you're running two NS2 server applications on one physical server?
    I guess the combined load might be causing the long loading times then.
    What's the disk load during loading?
    imageProud Dutchman image
    Join the Natural Selection IRC channel: #naturalselection on irc.gamesurge.net!
  • ZEROibisZEROibis Join Date: 2009-10-30 Member: 69176Members, Constellation Posts: 1,017 Advanced user
    No they are in two different folders. What takes up 99% of the loading time is after the server says it has finished loading...
    image
    Server IP: NS2.IBISGaming.com
    Server IP: NS2CO.IBISGaming.com
  • ZEROibisZEROibis Join Date: 2009-10-30 Member: 69176Members, Constellation Posts: 1,017 Advanced user
    With the recent build the load times appear to have gotten even worse when multiple servers change on the same disk at the same time. The longest part of the loading process also continues to be when the server says it has finished loading.

    When the server changes by itself it only takes a few seconds but if anything else is using disk to change maps at the same time it takes like 5min. There has got to be something wacky with the way that the server process is accessing the disk that causes these massive load times.

    Can this issue please be investigated?
    image
    Server IP: NS2.IBISGaming.com
    Server IP: NS2CO.IBISGaming.com
  • dePARAdePARA Join Date: 2011-04-29 Member: 96321Members, Squad Five Blue Posts: 1,351 Advanced user
    What hardware are you running the NS2 server on?
    Why are you never answering this question?

    And i repeat it again. You server has other problems than loadingtimes:
    http://ns2servers.devicenull.org/servers/273675/174.37.194.20:27015

    Your server cant handle the game, thats it.
    image
    CD121cArNaGej2
  • ZEROibisZEROibis Join Date: 2009-10-30 Member: 69176Members, Constellation Posts: 1,017 Advanced user
    The server is not using any CPU during map changes. This is an issue related to disk management. The hardware is a 7200rpm HDD.

    Should the devs possibly state that the game is intended and requires individual disk drives for each server instance or can only be run on SSD? Given that it takes 60 times longer for 2 servers to change maps at the same time vs twice as long clearly indicates a problem.
    image
    Server IP: NS2.IBISGaming.com
    Server IP: NS2CO.IBISGaming.com
  • Omega_K2Omega_K2 Join Date: 2011-12-25 Member: 139013Members, Reinforced - Shadow Posts: 558
    ZEROibis wrote: »
    The server is not using any CPU during map changes. This is an issue related to disk management. The hardware is a 7200rpm HDD.

    Should the devs possibly state that the game is intended and requires individual disk drives for each server instance or can only be run on SSD? Given that it takes 60 times longer for 2 servers to change maps at the same time vs twice as long clearly indicates a problem.

    I had 4 ns2 servers running on a SAS drive previously and was not suffering at all from this problem. Must be on your end.

    There are programs that allow you to defragement in a way so that a specific folder has all its items in a space on the HDD, so it doesn't have as much seek time. Other solution would be a new partition (which simlar) or try to put the files on a ram.

    Other issue might be NTFS being shitty.

    If you are running a lot of servers you should consider a HDD upgrade.
    German Slaughterhouse | GS Servers | Server Hardware
    IGN: [#OMEGA] - K2 (Hive)

    NS2 long standing issues (Fix ETA: Never):
    Bad Netcode (no adjustable tickrate, extremly delayed hit feedback, bad hitreg),
    Bad Performance (Poor Multi-threading, Mouse Lag, bad server performance, almost no multi-threading in server application)
    Bad Balance (GL Spam, semi-useless exos, poor chance of comebacks, no upkeep/or RT dimishing returns...)
    Technical Issues (Mouse Lag, Memory Leaks, Memory Corruption (and memory corruption server crashes))
    Bad Workshop implementation (down = kicks people/connection prboelms, out of date = nobody can join server),
  • ZEROibisZEROibis Join Date: 2009-10-30 Member: 69176Members, Constellation Posts: 1,017 Advanced user
    On my new server the ns2 server just has its own dedicated HDD and it loads in about 1 second now.
    image
    Server IP: NS2.IBISGaming.com
    Server IP: NS2CO.IBISGaming.com
  • WyzcrakWyzcrak Pot Pie Aficionado Join Date: 2002-12-04 Member: 10447Forum Moderators, Constellation, NS2 Playtester, Squad Five Blue Posts: 920 mod
    I'm having this problem (in Build 261, but we've been having it for months) running two instances of the server software on the same Windows box and disk (different install folders). I'll likely give up soon and host our two servers on different boxes, but I wanted to document my experience, in case it helps anyone.

    Those of you who regularly play on the Tactical Gamer servers know them to be far and away capable server instances, despite our many mods (see below; repro is stock), and you've watched countless mapchanges happen without issue or delay.

    I've repro'd this problem many times running both instances completely stock, w/ no mods at all, but I've yet to find any input/timing/etc which will /reliably/ repro the problem (very frustrating!).

    My best luck reproducing the behavior is when both servers (again: different install folders) change maps at the same time (within 1sec), and it doesn't happen every time. In fact, it can take tens of attempts to intentionally repro it in testing. And so, of course, It can take tens and tens of games to repro in normal usage. Or just a few games. It's terribly intermittent.

    But, when it happens, one or both instances will take 5+ minutes to change maps. I've clocked times as long as 15 minutes, and 10 minutes is common if you leave it alone. The server has not crashed. Given enough time, it /will/ release and continue its operation.

    During this extended hang, I see in the server console the routine messages (config files loaded, consistency hashed, meshes cooked/built, "Finished loading 'maps/ns2_soandso.level', etc, etc).

    But then the server console stops all output, allowing no clients past "WAITING FOR SERVER". Clients see "WAITING FOR SERVER" whether they were present since the mapchange started, or if they try to connect several minutes into the hang. When the server is hung like this, they're not getting past "WAITING FOR SERVER" no matter what.

    If you already had webadmin loaded, it continues to show NaN for tickrate (as it briefly would during any mapchange). If you didn't already have webadmin loaded, and you browse to it, the only response you see is "Loading server..." while the server is hung.

    I can reliably and immediately resume the operation of one of the hung server instances by completely shutting down the other instance.
    TGNS Portal | NS2 side-by-side game viewer
    IronHorse wrote: »
    TG's win or lose mod that replaces the concede function is the best of both worlds and should be official.
  • ZEROibisZEROibis Join Date: 2009-10-30 Member: 69176Members, Constellation Posts: 1,017 Advanced user
    It does not matter if they are in different folders. the issue will persist as long as they are on the same HDD. You will find that to prevent this issue the game requires you to buy an individual HDD for each instance of the game you wish to run.
    image
    Server IP: NS2.IBISGaming.com
    Server IP: NS2CO.IBISGaming.com
  • WyzcrakWyzcrak Pot Pie Aficionado Join Date: 2002-12-04 Member: 10447Forum Moderators, Constellation, NS2 Playtester, Squad Five Blue Posts: 920 mod
    Ahh... so you're saying you've solved it by running two instances on the same box but with different physical disks? Any idea if it's sufficient to use different volumes (on the same disk)?
    TGNS Portal | NS2 side-by-side game viewer
    IronHorse wrote: »
    TG's win or lose mod that replaces the concede function is the best of both worlds and should be official.
  • ZEROibisZEROibis Join Date: 2009-10-30 Member: 69176Members, Constellation Posts: 1,017 Advanced user
    It will likely only be resolved by using physical disks as the issue appears to be caused by a problem with the way that the game handles the disk queue. If you monitor the disk activity you will find that the activity is not even being maxed out. Basically the two instances fight each other to get the disk activity.

    Think of it like 2 kids fighting to use a water fountain rather than take turns. Creating a virtual second fountain might help but as long as the kids keep fighting or they each get their own private fountain you will have the same issue.
    image
    Server IP: NS2.IBISGaming.com
    Server IP: NS2CO.IBISGaming.com
  • WyzcrakWyzcrak Pot Pie Aficionado Join Date: 2002-12-04 Member: 10447Forum Moderators, Constellation, NS2 Playtester, Squad Five Blue Posts: 920 mod
    I understand what you're saying. I'm not prepared at the moment to test the performance of my two servers on different disks.

    Isn't it true that plenty of server owners/hosters are running multiple servers per box? If so, and insofar as your disk-per-instance requirement is accurate, shouldn't 5+ minute mapchanges be a much more common (even if only intermittently) experience for the average NS2 player than it is?

    @ZEROibis, I know I'm just asking you to guess, but... how might you resolve three seemingly conflicting observations? 1) must have a disk per instance to avoid 5+ minute mapchanges 2) most players in the community at large don't experience 5+ minute mapchanges 3) plenty of server owners/hosters are running multiple instances per disk

    Is one of those observations broken? Am I missing something?

    My fear is that my problem (which, granted, might not be identical to yours) is more complicated (and not yet fully understood, despite my best efforts and your knowledge sharing).
    TGNS Portal | NS2 side-by-side game viewer
    IronHorse wrote: »
    TG's win or lose mod that replaces the concede function is the best of both worlds and should be official.
  • ZEROibisZEROibis Join Date: 2009-10-30 Member: 69176Members, Constellation Posts: 1,017 Advanced user
    2 is wrong b/c in those cases those servers are running on SSDs will not be effected. Second the issue will only occur if both servers change maps at the same time which is only going to occur every once in a while.
    image
    Server IP: NS2.IBISGaming.com
    Server IP: NS2CO.IBISGaming.com
  • Omega_K2Omega_K2 Join Date: 2011-12-25 Member: 139013Members, Reinforced - Shadow Posts: 558
    ZEROibis wrote: »
    2 is wrong b/c in those cases those servers are running on SSDs will not be effected. Second the issue will only occur if both servers change maps at the same time which is only going to occur every once in a while.

    I was running 4 servers off normal HDD before and it never happend. Not even when I restarted all servers at the same time.

    Still if you guys have issues try following:
    1) Use a single shared directory for the server files (symlink, or directly run off that folder)
    1.a) Use a program to put those files most closely together on the HDD to reduce seek time
    1.b) or, if you have plently of ram, try to load them into memory
    2) Try linux, it handles files differently then windows and has plently of file systems
    3) Upgrade your HDD
    German Slaughterhouse | GS Servers | Server Hardware
    IGN: [#OMEGA] - K2 (Hive)

    NS2 long standing issues (Fix ETA: Never):
    Bad Netcode (no adjustable tickrate, extremly delayed hit feedback, bad hitreg),
    Bad Performance (Poor Multi-threading, Mouse Lag, bad server performance, almost no multi-threading in server application)
    Bad Balance (GL Spam, semi-useless exos, poor chance of comebacks, no upkeep/or RT dimishing returns...)
    Technical Issues (Mouse Lag, Memory Leaks, Memory Corruption (and memory corruption server crashes))
    Bad Workshop implementation (down = kicks people/connection prboelms, out of date = nobody can join server),
Sign In or Register to comment.