ServerPerformanceData and (proposed) 269 ServerBrowser
matso
Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
During 268, the new ServerPerformanceData kit will be phased in.
This kit is going to be used for 3 things
1. Let server admins accurately scale their server.
2. Rating servers on the Server Browser
3. Warning players on a server that the server is overloaded
There are two console commands available straight out of the box, "perfmon" and "perfbrowser". Also, the "net_stats" console command has a new line using the ServerPerformanceData to display server info.
"perfmon" controls dumping out data to the log file. It cycles through various detail levels, with the most detailed dumping out the per-second snapshot.
"perfbrowser" toggles the server browser to show the proposed 269 version of the ServerBrowser.
The intent is that server admin use these tools during 268 to ensure that theirs servers are properly scaled and scores well on the 269 ServerBrowser. Do checkout the json docs for the ServerPerformanceData, and the ServerPerformanceData.lua file
ServerBrowser and scoring
The 269 ServerBrowser will group servers into The Good and the Bad.
The browser will ALWAYS list the Good servers on top and the Bad at the bottom, below all Good servers.
The score is calculated as "%idle - BadThingsHappeningToYourPlayers".
%idle is simple the percent of time the server is fiddling its thumbs and not doing anything. To avoid running into BadThings territory, a good rule of thumb is to have the server at about 20% idle, even in the endgame. Ideally. But as long as the server don't BadThings to its players, it will still be counted among the Good.
The only way to get listed as a Bad server is if the server consistently does BadThings to its players. And the only bad thing that counts is causing players to get an interpretation buffer overrun. Players experience an interpretation buffer overrun as teleporting, rubberbanding and loss of avatar control - basically, the game becomes impossible to enjoy.
Not a desirable situation.
In normal settings, the interp is set to 0.1 or 100ms, and normally the server sends out a network update every 50ms. So it is only if the server ever gets so busy that it is unable to send an update inside 100ms that it gets a really bad score (we give a small negative score for exceeding 75% of the interp buffer as well).
The server inner loop tries hard to maintain the sendrate, and it takes a concerted effort (or rather, very optimistic settings) to make it break it.
And with averaging and worst-case discarding, a server will only be in the Bad group if it consistently gives its players a bad game.
The server will maintain a performance history - basically, the worst 30-second periods in recent history - and use this to update the master server performance information.
Note: THIS IS EXPERIMENTAL - DO provide feedback.
PS. And no, the ServerBrowser will not look like that in the final version; the perf-line is just there to provide feedback. DS
This kit is going to be used for 3 things
1. Let server admins accurately scale their server.
2. Rating servers on the Server Browser
3. Warning players on a server that the server is overloaded
There are two console commands available straight out of the box, "perfmon" and "perfbrowser". Also, the "net_stats" console command has a new line using the ServerPerformanceData to display server info.
"perfmon" controls dumping out data to the log file. It cycles through various detail levels, with the most detailed dumping out the per-second snapshot.
"perfbrowser" toggles the server browser to show the proposed 269 version of the ServerBrowser.
The intent is that server admin use these tools during 268 to ensure that theirs servers are properly scaled and scores well on the 269 ServerBrowser. Do checkout the json docs for the ServerPerformanceData, and the ServerPerformanceData.lua file
ServerBrowser and scoring
The 269 ServerBrowser will group servers into The Good and the Bad.
The browser will ALWAYS list the Good servers on top and the Bad at the bottom, below all Good servers.
The score is calculated as "%idle - BadThingsHappeningToYourPlayers".
%idle is simple the percent of time the server is fiddling its thumbs and not doing anything. To avoid running into BadThings territory, a good rule of thumb is to have the server at about 20% idle, even in the endgame. Ideally. But as long as the server don't BadThings to its players, it will still be counted among the Good.
The only way to get listed as a Bad server is if the server consistently does BadThings to its players. And the only bad thing that counts is causing players to get an interpretation buffer overrun. Players experience an interpretation buffer overrun as teleporting, rubberbanding and loss of avatar control - basically, the game becomes impossible to enjoy.
Not a desirable situation.
In normal settings, the interp is set to 0.1 or 100ms, and normally the server sends out a network update every 50ms. So it is only if the server ever gets so busy that it is unable to send an update inside 100ms that it gets a really bad score (we give a small negative score for exceeding 75% of the interp buffer as well).
The server inner loop tries hard to maintain the sendrate, and it takes a concerted effort (or rather, very optimistic settings) to make it break it.
And with averaging and worst-case discarding, a server will only be in the Bad group if it consistently gives its players a bad game.
The server will maintain a performance history - basically, the worst 30-second periods in recent history - and use this to update the master server performance information.
Note: THIS IS EXPERIMENTAL - DO provide feedback.
PS. And no, the ServerBrowser will not look like that in the final version; the perf-line is just there to provide feedback. DS
Comments
Is this going to come with some actual improvements to server performance? Currently I do not see how this is really even needed. Is there still such a major issue with shit performance servers? All this seems to be trying to do is to slit hairs and favor a select group of servers.
Also 20% my ass I checked and it gives me not good even when at 30tick and <50% on my highest core. (# in game was -3 to -7)
I could see this if maybe we were still having really bad performing servers running the game after actually doing some updates to focus on server performance for a chance.
Also right now a good number of servers do not even list as good even with 0 players on them. I guess since the match up to mine in this level when they are empty and mine is full they will all be listed as bad when full but still this can not be right.
Its more transparent for admins now how there servers perform and they can do what they want with this informations.
Most players didnt care anyway or didnt notice the constant rubberbanding.
Your 22 slot server @ZEROibis cant handle this slotammount btw. I dont need these new tools to see that.
Wasnt that a problem in the past? no
Is this a problem in the furure? no, cause people dont care
For me as an admin who want the best possible perfomance, these are great changes.
Also your 18 slot server when full just now was pulling in with the same rating as mine (the one at 89.163.177.130 even though it was full 30 tick as well. So unless your trying to say that your own server has just as bad performance as you alleged mine does you might want to reconsider if you really like the koolaid or not.
Your server has problems in endgame wich result in rubberbanding.
Its still on an relative payable level during endgame but the issues are there.
You can use the new tools optimize things (lower playercount to 20 maybe) or let the server as it is.
Your choice
Dont know what your problem is with the new performancerating.
And btw. the tickrate in Ns2 is used for AI. There more important rates wich affect the players.
These more important rates are part of the new performancerating now, wich is great.
Ways more important is that the time between ticks stays constant that their is enought cpu load buffer for lategame, that there is enough bandwidth avaible so you don't run into choke issues and so on.
The tickrate is therefore only one of the many values that determ if your server performance is okay. If you look into how the spark engine and its networking works you will understand that this new algorithm is ways better to say anything about the server performance.
You can compare reading the server performance from the tickrate to guessing tomorrows weather from looking at the clouds at the evening.
Yes sometimes it works like if there is a horrible storm coming but in most cases its not really possible to do
And for the end have in mind that this is not anywhere near being done. That's why matso shows it here.
Yes that is kinda planned.
Or are we now punished for high ping players?
There is still a bwlimit of 25 kb/s by default so even if you have 100mbit/s ns2 won't use them unless you increase the bwlimit ... .
So your server could still have choke issues due to that and that could influence the rating .
The most important value of a server is the -level of fun- users perceive.
Whether a community server is popular or not is currently self-regulated by the level of fun people have playing there. The result is: un-fun servers are empty.
The level of fun factors in a lot of different things including for example, how nice the regular players are.
You are now making plans to regulate something that so far has been just fine by itself. Could you elaborate on the reasoning behind this decision?
Forcing small empty servers with annoying/restrictive settings and adolescent admins on everyone is not how you save NS2.
Your server performance measurement and information endeavour will work without this. It will actually have more impact without it.
You might try to argue that server performance affects fun. It does no doubt. But it's way down the bottom of priorities.
If performance were such an issue, Wooza's would be empty.
Some peopple didnt see the difference between analog->digital, some dont care about a blurry pixelated interpolated picture and others want a perfect super sharp television picture. Thats why they invest extra money in calibration hardware for example.
The server rating is a rating about the serverperformance, nothing more.
And as i wrote above: some people dont care.
So whats the problem?
Its not a rating of the regular player base of the server.
If i am new to a game and see an red warning that the server has problems i would try another one if im rubberbanding.
This "warning" tells me: this is not the normal case.
Like in 1000 endless why people prefer large servers threads (only the different way in this case):
If people want to play on optimal running servers, its there choice.
And now they can see the real perfomance wich is great.
Btw.
Here is a small example of how "funny" rubberbanding shit servers are:
http://www.twitch.tv/schustor/c/4527004
Go focus your efforts somewhere else, server crashes increased 500% since 267, try to get those memory leaks out.
Or come up with a solution to the workshop problem, which currently makes custom maps an impossibility
Or improve client performance, it has gotten a lot worse with 267,
Let us focus on the player count, we don't need developers to police around.
Coincidence? Dont think so.
Maybe cause there servers are not on top of the serverlist anymore?
Maybe cause players finally see the issues with these kind of servers?
Hmm, who knows.
New players are, since b267, aware thanks to a short message when they attempt to connect to a large server NS2 hasn't been designed for a such amount of players. When they read this, they should understand performances aren't guaranteed either.
So, bad or good for this kind of servers seems a bit out of subject here. In my opinion, and personally, I think this system shall be used for a smaller number of slots & regarding of their performances, etc. Like a 18/20 slots with rubberbanding or whatever and seriously, there are a bunch of bad performances servers...
If virtually all the servers are going to be listed as sub par performance anyways with this what is the point other than to communicate to the players how bad the games server code really is? That is what I mean when I call this entire thing a joke.
...........
[s
What I know is that larger servers are more popular, because of the following reasons.
A, On a map change, there is less chance of a server dying due to more players connected to it.
B, There is more action on the map, and more to do.
C, There is less of a feeling that an individual can fuck up the game for the entire team, means there is less pressure to those players. (for example dying as an onos has less impact to the entire team)
D, There is more of a feeling of teamspirit (which results in more fun)
E, Attacks can be organized on different fronts, instead of just one or maybe 2
This has nothing to do with the quality of the server, if this system would that matso is developing is taking the above in account and would calculate that into the equation aswell only then this system would be transparent as the developer says what the purpose should be of this system.
And the purpose of this system is to define the GOOD servers from the BAD servers. saying the server is BAD by reading performance statistics is completely ludicrous. That is the same like saying Well this pizza is bad because of the thickness ofthe bottom of the pizza, But totally disregards the toppings that are on it.
This is why this system is retarded, and completely irrelevant. Like I posted before, go focus on the real problems this community is suffering from. Work on a solution for the workshop. 75% of the time not all mods are loaded in on my server. They are important, they define administration, balance, configuration and custom maps.
The playercount doesn't matter for the server performance rating. All it takes into account is your cpu load, tickrate, sendrate, your network bandwidth and if that fits to the needs of your server.
So you can still get a good rating for a 24+ slot server if you have a powerfull enought cpu + enought bandweight.
Server with cpu load issues and too less bandwidth will be rated by this system as bad no matter what playercount they have.
So if your server is rated as bad try to find out why it's so and fix the issues your server seems to have.
There are advantages & disadvantages to join a large server or skip it. Personally, I don't like them but I can understand why some people, contrary to me, enjoy to play on these servers. But, you have to admit, this kind of servers doesn't reflect the initial gameplay made by UWE for some reasons i'm not going to explain now & here.
Your comparison, isn't really appropriate I think. If the performances are terrible, experienced and old players are aware & they know that when they attempt to join these servers, but new players aren't. You can enjoy some games despite the performances, that's right, but, just imagine reactions of new players if they join these servers for their first try, seems logical if they filter out their servers depending of the amount of players.
GOOD or BAD should show up regarding of the performances. Why? Because as new player, the community & developers want to show them a great image of the game & their servers (that is not currently true on some servers, even some <24 slots). And what the best indicator except performances? You can have different systems, if the server is often full, performances, skill levels, whatever. These indications should appear on the loading time, i don't know, something like: "This server cannot guarantee optimal performances/the amount of slots can affect performances/this server is popular concerning the last visits".
It really depends on what criterias this system is based. Blame a server for its performances seems legit (personally, as a new player, the unique condition i'm going to rely is performances) but of course, and that can be easily understandable, large servers, which ones cannot ensure acceptable or optimal performances will be shown as BAD.
269 marks the end for my server, I will take it offline and host for other games where there is better support and stability.
Hope other large server operators will do the same.
Bye
This is not a discussion about large servers.
Its a discussion about bad performing servers.
There 16 slot servers out there wich are bad.
The chance that a large server is marked as "bad" is just higher cause the nature of NS2.
Despite everything said, there is something that seems like it actually does penalize large servers. Excerpt from the comments on the performance scoring code:
So if a server with 32 players gets a "lag spike", noticeable by every player (as it probably is if it's down to server performance which we're interested in now, though players will get data sent to them in sequence so not necessarily...), the score penalty will be twice that of a server with 16 players. I don't think that is neither fair for ranking in a browser nor useful for an operator doing performance analysis. In my opinion, if a server hosting 32 players and a server hosting 16 lags equally (every player lags), the score should be equal since the experience of the individual player is equal.
The obvious but wrong thing to do would be to penalize based on the average players interp issues. But then if not everyone lags but only a smaller portion of players, the smaller server will be more heavily penalized instead...
This is feedback is only based on what I feel is right (but I'm not sure what's correct), maybe it's accurate enough, I don't know.
By the way, I really appreciate this was implemented in Lua even though it will be done in the engine, so that it can be read, cheers for that! Now for some questions...
Where does the GetNumInterpFails/Warns come from? If a client is experiencing interp failures due to something not directly related to the server, for example a bad network at the clients end, does that increase the counters, and thus the performance score penalty?
When viewing the raw tickrate (as shown as the last value in perf row with perfbrowser enabled), it appears that servers with currently bad performance shows their ticks (lower than 30), but every other server show some value often higher than 100. What's that?
How are custom rates handled? If a server is performing badly, is would increasing interp/decreasing ticks (of various kinds) be an acceptable adjustment? Perhaps this needs be indicated somehow, as that could cause a different kind of lag, although maybe it's a valid option. On a side note I would really want to see the rate settings on the detailed popup, the custom rates warning was just annoying so I had to disable it.
---
@the_tick, I played a game at your server and kept track of how the performance score was changing to see how things were doing. The server was full, and ran fine at the start of the game.
For reference, this is how the browser interprets the performance score (a value ranging from -99 to 100):
Higher than 20: Good
Lower than 20: OK
Lower than 0: Loaded
Lower than -20: Bad
About 8 minutes in I started getting brief rubberbanding in the combat. At this point the score was lowered from around 20-30 to down to 15 but then up again to near 20.
Next few minutes the rubberbanding got significantly worse and the score was lowered to less than 10, often less than 5 and as low as 0.
A few minutes before the finale the pauses were long enough that it was no longer just some rubberbanding but instead getting the red plug for a second or so, oddly the score increased to above 10.
After a few red plugs one other player asked if it lagged and another confirmed, so this was not just me.
Still, during all of this the game remained playable in a casual way and apparently enough players to fill a large server will accept that tradeoff, so I'm sure they still will even if the performance stats are adjusted to be more accurate! Note that the server never once got close to getting a Bad performance rating, I never even saw it reach Loaded. The server browser remained at OK. "The only way to get listed as a Bad server is if the server consistently does BadThings to its players." holds true and I don't see what's so horrible about it. I think you're grossly overreacting.
I've checked the perfbrowser a few times and it's only the same few servers that are marked as bad, and it's consistently them, so my assumption is they're consistently lagging, and their tickrate was low - your server is not even one of those. Unfortunately I couldn't join any of them because they were all full and I'm not really dying to get in either :P (and by the way, one of the three servers hosted only 16/16 players).
Edit: Almost forgot, here's the log from the game: http://pastebin.com/N2bBKs74
What is wrong with ensuring players have a smooth, lag free experience like they assume they would get for their money?
Kids running servers on their laptops occurred in the first few weeks of launch and this prompted us to include the performance tab right away because of the HORRID first time experience customers were receiving. It's always been an insufficient measurement... but you cannot deny such a gauge has been very much needed.
Lastly, like others have told you : Large servers have nothing to do with a performance gauge!! Some servers can't handle 16 players... and yet its possible to run 40 just fine if you have the hardware.
This is about providing the admin an accurate measurement of their server load for once, so that they can adjust as necessary to best provide the expected lag free experience to customers.
Else, you are potentially just another kid with a laptop who does not care about such a thing...
The only way for me to resolve this is to cold restart, (shutdown and restart the server completely) I wait till the round ends and then perform a restart. Then the server has te repopulate again, and with my luck in this case, """"""""0""""""" mods loaded in, zero, nill nothing, nada, which means there is absoltely no server configuration loaded in. Which is pissing me off so much.
So to sum this up, due to crashes, (twice a day average) my server performance gets fucked, which means it is a BAD server, which means it gets blacklisted by the newly implemented server browser,
For me this is the last straw, and I say to you, goodbye.
and lwf, accoording to your time index you joined after the server crash.
We are monitoring the performance of the server closely while we keep pushing it to the limits. The new tools will make that a lot more precise. Awesome!
If such a thing occurs that it goes below a playability / can have fun level, which, regrettably, it did with the 267 patch, settings will be adapted. In the worst case, it will be forced to drop slots.
Since then a number of regulars asked when it's going to be back to it's original size. It makes Wooza's babbler sad to disappoint them with "We hope to work something out for 269". Fortunately, matso has been utterly helpful in that regard ( Thanks! ).
I'm all for the RATING and pointing fingers at terribly performing servers. The suggested system labels a server's performance good, ok or bad. Done.
Yet, I am asking intently, who thought it would be a good idea to additionally RANK servers by the performance rating in the server browser?
"Bad things" is a term that is used a lot by matso in his post. He's undoubtedly right. But network issues are not the only "bad things" that players experience.
I've seen rookies ask questions like "Hey, is this also a server that kicks you because you're below a certain hour count?"
Most servers do not even allow you to pick your team anymore. What label (or should I go so far as to say -ranking-) do those get?
Showing an accurate performance rating = yay | Ranking by only performance = nay
Allow me to rephrase ZEROibis' post into a question, since he appears to be getting ignored:
Does the majority of the servers not receiving a good rating right now signal that all plans to ever increase dedicated server performance have been scrapped and instead the server operators are forced to adjust their hardware or settings? The timing with the official server shutdown comes to mind.
Server operators that believe the server performance rating (sans the ranking) specifically targets large servers are mistaken. Large servers are accepted to not always have a perfect performance. The small and medium servers are getting the short end of the stick because they are all suddenly not good, only "ok". Those aren't all laptops now, are they?
Well, the biggest negative score a single one-second performance data will give you is -99, so it won't matter in practice - if the server starts hickuping enough to trigger it, the number of players won't matter; you are pretty much assured a -99 score for that second no matter how many players you have.
It comes from how long the interval was between two network updates, compared to the interp variable. Warns is for > 75% of interp, Fails is for > 100% of interp buffer.
Network latency or client problem does not figure into it, so internet problems will not affect the server performance score.
It it because what is show as tickrate is actually number of times the server goes through the main loop - a lightly loaded server waits 5ms on the network socket before doing a loop, so it tops out at 200 "ticks" a second. It will only do a "proper" tick, ie update the AI enties every 33 ms.
When a server becomes loaded, it will spend more and more time processing player moves and loop around fewer times.
The server performance is not affected by ratings - it just looks at the net result. Lots of idle is good, gives you margins. Only when the server is not able to fulfill its obligations will it get bad results.
That being said, the best way to save yourself from going bad is to increase the interp variable - it allows the server more margin to deliver updates to the players.
Yea, I have also noticed that Woozas and Ticks servers are actually not having server performance issues (well, they are running really close to full, but from what I have seen, they do manage to stay on the right side of the line).
Considering the amount of red-plugging you actually experience, compared to how well his servers actually manage to perform, I think that Woozas is probably correct in that his server can drive the amount of players he has... or at least close to it.
The real problem is probably insufficient upload bandwidth (see my other post about that), and that is much more difficult to catch, as the server actually writes the data for the clients in a timely fashion - the problem is that the packets then get queued up/lost at the upload bottleneck.
The bandwidth unlocking in 267 probably made things WORSE for Woozas... I think he may have tried to increase it and then got even worse problems.
Required bandwidth goes up by about the square of the number of players ... at a VERY rough guess, you can probably fit 16 players fits inside a 10Mbit line... 32 players would be about 40Mbit or so, but I would definately recommend a 100Mbit line for anyone running > 24 players.
All in all, I'm kinda pleasantly surprised by how few servers are actually bad.
Also yes onto my note before if we can all stop for a min crying about our individual servers take a look at the list and see how many servers in your area have good performance the majority of them have ok performance around 0.
I think the reason so many are upset is that it should be noted that the server operators here are trying to do the best we can to balance between offering players a fun server with the number of slots they want and performance impact. There is only so much we can do we can not force intel to magically make 20Ghz single core xeons to run slow server code. It takes a lot of money and time to run and manage servers and when we see stuff like this it makes us feel betrayed.
In line with what was said before about servers on laptops, this is the sort of thing that really upsets server operators. We feel that basically the view is oh the game is fine nothing wrong with the code or performance at all. Just these evil server operators ruining the game we need to put a stop to these bad guys. Now I am sure there might have been people running severs on laptops or who knows what when the game launched but how long ago was that? Is there really such a major issue with laptop servers we even need to bring that up b/c it just reinforces to server operators that they are not liked and makes them upset.
The reality is that most server operators already have gotten themselves the best equipment they can for their servers and this is why build after build after build server operators have been looking for more performance from the game and getting less. Operators have been waiting to hear about simple long standing issues with the dedicated servers like the poor workshop implementation being fixed.
Instead of being told how server performance is going to get better or that server issues are going to be fixed, operators get told how their server is now going to be marginalized and segregated. Rather than actual improvements to the servers, this is what time is being spent on.
I understand from the devs perspective they want to be able to inform clients about how server performance impacts the game but doing this before even trying to improve anything with regard to the servers is what has everyone up in arms.
The real problem is that this being such a big issue more so highlights how badly the actual server software in its current form runs on most of the servers. You never hear of this being such a big deal in other games which have properly optimized server code.
I guess at the end of the day you could call the entire issue on both sides here the pot trying to call the kettle black.
(on another note what us server operators are calling bad is anything not rated good, it is those of us in the middle or on the edge that are most impacted by this (which is the vast majority of the servers))
Also do I understand that this system should punish servers that use trubo mode on a cpu to increase permanence when needed or other sorts of cpu based throttling? If so is there anything that can be recommended for us to run to force the core the process is running in to always be in turbo mode to prevent us from being scored negatively?