Speeding up consistency checks.
endar
Join Date: 2010-07-27 Member: 73256Members, Squad Five Blue
Join Date: 2010-07-27 Member: 73256Members, Squad Five Blue
Hey guys,
I'm currently testing a method to speed up consistency checking on the server side. I know some people may be interested in this, as they don't run consistency checks because of the additional overhead they add to map loads, or they are currently running them, and would like to speed them up.
Requirements for this to work.
You basically need 2GB spare ram. You will need to do your calculations based on how many servers you run per box.
My situation: 8GB total, 4 total servers. They can use about 5GB, leaving me with 3GB of ram.
Quick overview:
Create a 2GB ram drive, and run NS2 server from there.
How to do this:
Download and install this program:
http://www.ltr-data.se/opencode.html/#ImDisk
It allows you to create ram drives from cmd prompt.
Create a batch file, schedule it to start with the system.
imdisk -a -s 2048M -m n: -p "/fs:ntfs /q /y"
ping -n 15 127.0.0.1
steamcmd.exe +runscript yoursteamupdatescript.txt
Server Consistency checks have gone from about 62 seconds, down to 3 seconds.
Map loading now takes about 12 seconds total, down from something horrid that I never actually sat and timed.
Individual results will vary, this was such a valuable thing for me to do, because my server hard drives are very slow. Sequential read speeds have gone from 55MB/s to 6000MB/s.
I'm not covering how to launch ns2 or update it, as these are covered by other people in other places.
I'm currently testing a method to speed up consistency checking on the server side. I know some people may be interested in this, as they don't run consistency checks because of the additional overhead they add to map loads, or they are currently running them, and would like to speed them up.
Requirements for this to work.
You basically need 2GB spare ram. You will need to do your calculations based on how many servers you run per box.
My situation: 8GB total, 4 total servers. They can use about 5GB, leaving me with 3GB of ram.
Quick overview:
Create a 2GB ram drive, and run NS2 server from there.
How to do this:
Download and install this program:
http://www.ltr-data.se/opencode.html/#ImDisk
It allows you to create ram drives from cmd prompt.
Create a batch file, schedule it to start with the system.
imdisk -a -s 2048M -m n: -p "/fs:ntfs /q /y"
ping -n 15 127.0.0.1
steamcmd.exe +runscript yoursteamupdatescript.txt
Server Consistency checks have gone from about 62 seconds, down to 3 seconds.
Map loading now takes about 12 seconds total, down from something horrid that I never actually sat and timed.
Individual results will vary, this was such a valuable thing for me to do, because my server hard drives are very slow. Sequential read speeds have gone from 55MB/s to 6000MB/s.
I'm not covering how to launch ns2 or update it, as these are covered by other people in other places.
Comments
Also, something worth mentioning. Since the throughput is so high, the next bottleneck on checking is your CPU, which means that it will hit 100% (of a single core) when it does it. Not a big deal i think, since any server should be expected to hit 100% or more anyway.
Loads pretty fast already
My chi server though I can't run this since I don't have access to the box
Likewise a similar method should be used client side except with a method to verify that the file has not changed from the hash to prevent exploits and hacking. This would dramatically improve load times for players.
And yes, I agree. It will most likely be solved in future by changing the frequency of hashing the files. But there are bigger issues plaguing consistency checks right now (of which I'm about to make a new thread).
same on my servers, but it could just be the first time run when the server is compiling the files for the map.
Via programming eh? How do you check that a file hasn't changed without hashing it? The *only* way I can think of involves an exclusive lock on the file, which means that you can no longer make changes to the server while it's running. I'm pretty sure that no one wants to have to stop their server to upload new content.
You can ask the filesystem when the files were last modified. But know that feature may not always be available.
How about hash the files that exist when the server starts. Save these hashes until the server closes. On each map load, check if theres any new files (from mods), and hash those, replacing old ones if they exist, while keeping the existing hashes in memory.
If I had to stop my server to upload new content so that every time a player connects they can join faster I would stop my server right now to upload that content!
It's not a bad idea to generate table of hashes once and save it to disk. And instead of reading the stuff again - just use already generated table of hashes. There is no reason to generate hashes again, again and once again if you don't make any changes. If you put a new stuff - just use a special console command in order to generate table of hashes again.
But how about a standby memory? When OS read files, it put files to free RAM in order to achive an instant access. So if program read this file again, it will be read from RAM, but not from a slow disk.
I don't understand what you mean. Can you reword it, or reference is against what I said?
And ibis, i'm not sure what you mean either. The file containing the hashes would be kilobytes in size, wouldnt matter if its ram or disk.
In task manager you can't notice that your free memory is used for a cache. Use a resource monitor instead.
That is what my point was, that the file would not be large enough for it to need to be stored in RAM constantly.
I did previously rely on it, and sometimes you would get nice load times (around 20s), down from the 60+ if you did two map loads in quick succession. But it was very random. If a map went for a long time, the chances of those files still remaining in cache are very low. Using a ramdisk will guarantee, with 100% certainty, that you will always have the fastest consistency checks possible.