Server upload choke
matso
Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
I had a discussion with ATF about clients getting yellow plugs even though the server was seemingly doing fine. He believed that the yellow plugs mostly happened to clients connecting last to the server. And it had gotten worse, not better when the bwlimit was increased.
It turned out that the server had a 10Mbit upload limit, which on the face of it sounds fine - 20 players at 25kb/sec gives a peak bandwidth use of 500kb/sec which is less than half the max.
However ... it turns out that you need to analyze this on the ms level, not the second level.
At the ms level, a 10Mbit connection is (rougly) 1Mbyte/sec = 1kb/ms.
This means that if the server for some reason needs to send a large update to all players at the same time you might be looking at each player getting a 2-3kb packet each.
And the server will loop through players from top to bottom... with 24 players and each player adding 2-3ms delay to everyone coming later, you can easily look at > 50ms upload choke latency added to the last few players.
Which means that they will receive the packet after > 100ms, ie an interp buffer overrun - teleporting and rubberbanding galore.
Even worse, trying to push 50-100kb of data in 1ms may well cause packet loss in the upload link, so late players may not be getting their packets AT ALL.
Now, a non-loaded server tries to service the clients every 5ms or so which spreads out the load a bit.
Unfortunately, if a server starts getting loaded, it will prioritize processing player moves... which means that if a server has < max ticks, it will only service the network once every tick... and if the ticks drops below the sendrate, it means that every network service will have ALL players updated at the same time.
And this usually happens just when there is a lot of things going on.. so you get smacked with large network updates to ALL players at once.
This should not be a problem for 100Mbit up connections, but if you are running a server on a 10Mbit upload, you might want to keep that in mind when you scale and configure the server.
Peak bandwidth use scales roughly with the square of the players, so a 24 player server requires 4 times the bandwidth of a 12 player server. This also means that dropping just a few players may well fix any bandwidth limit problems.
Apart from dropping playercount, decreasing sendcount and increasing interp is also possible - increasing interp to 150ms allows you to decrease sendrate to 13, which should result in about 30% less bandwidth use.
This came up after the 268 codefreeze, but for 269 I'm hoping to add server peak bandwidth use to the ServerPerformanceData. Might also be possible to make the server aware of the upload bandwidth limitation and spread out the pain among the players a bit.
It turned out that the server had a 10Mbit upload limit, which on the face of it sounds fine - 20 players at 25kb/sec gives a peak bandwidth use of 500kb/sec which is less than half the max.
However ... it turns out that you need to analyze this on the ms level, not the second level.
At the ms level, a 10Mbit connection is (rougly) 1Mbyte/sec = 1kb/ms.
This means that if the server for some reason needs to send a large update to all players at the same time you might be looking at each player getting a 2-3kb packet each.
And the server will loop through players from top to bottom... with 24 players and each player adding 2-3ms delay to everyone coming later, you can easily look at > 50ms upload choke latency added to the last few players.
Which means that they will receive the packet after > 100ms, ie an interp buffer overrun - teleporting and rubberbanding galore.
Even worse, trying to push 50-100kb of data in 1ms may well cause packet loss in the upload link, so late players may not be getting their packets AT ALL.
Now, a non-loaded server tries to service the clients every 5ms or so which spreads out the load a bit.
Unfortunately, if a server starts getting loaded, it will prioritize processing player moves... which means that if a server has < max ticks, it will only service the network once every tick... and if the ticks drops below the sendrate, it means that every network service will have ALL players updated at the same time.
And this usually happens just when there is a lot of things going on.. so you get smacked with large network updates to ALL players at once.
This should not be a problem for 100Mbit up connections, but if you are running a server on a 10Mbit upload, you might want to keep that in mind when you scale and configure the server.
Peak bandwidth use scales roughly with the square of the players, so a 24 player server requires 4 times the bandwidth of a 12 player server. This also means that dropping just a few players may well fix any bandwidth limit problems.
Apart from dropping playercount, decreasing sendcount and increasing interp is also possible - increasing interp to 150ms allows you to decrease sendrate to 13, which should result in about 30% less bandwidth use.
This came up after the 268 codefreeze, but for 269 I'm hoping to add server peak bandwidth use to the ServerPerformanceData. Might also be possible to make the server aware of the upload bandwidth limitation and spread out the pain among the players a bit.
Comments
They do consume more bandwidth in general because they jump around the map a lot - when they do, the new entities need a full state update rather than the much smaller delta update (same thing happens when you go through a PG).
But it should obviously not be that bad.
If a server wanted to increase the bandwidth limitation without increasing interpolation (which trades rubber banding for delays) could they just modify their bwlimit variable to a higher value?
Interpolation has nothing really to do with the bwlimit beside that by increasing the sendrate your server will send more packets per second and therefore uses more bandwidth and the other way around
Overall increasing the bwlimit is kind of a nobrainer to do as long as the server has enought total bandwidth.
Imho the default value of bwlimit should be increased to 50 kb/s (or even better 100 kb/s) just to avoid any kind of choke but that would mean 1,2 MB/s max upload is needed for 24 slots, so for hosting a server it would be requiered to have at least a connection of 10 Mbit/s.
So the real issue at woozas server is not having enought bandwidth avaible overall but that's not a real suprise for me.
The bwlimit has nothing to do with the client it's a server only config param.
But yeah could explain your issue as voice chat packets are handled abit different afaik. Beside that maybe the server packets just get lost due to the latency you have.
Seems like the server you are playing on runs into chokes. Just use net_stats and watch the choke field (should stay at 0%)
Why doesn't it run as before with bwlimit 25?
There's disappointment in the ranks of our valuable remaining players. Not an ideal situation if you want to avoid people stop playing NS2.
( Pregameplus 10+ a game ) vs. ( 1-2 without pregameplus )
Of course what was interesting is that I noticed no tickrate drop, no choke, no errors, and not every player would red plug sometimes. Even after 10 seconds plus the connection would come back.
Another thing that helped was increasing interpolation buffer to 150 however I am still having problems. I run a 18 slot server and I was hoping to have increased rates or non standard like 20 sr 20 mr or 25/25 or my goal was 30/30. sigh... well I hope to see some new tools
I made a query to my provider to see my networking limits. I think this is the cause due to reading this. I find it odd that mods I have used for a long time seem to be causing problems now.