Skill rating system and server side filters - aka poor rookies, aka I saw an aim botter
joederp
Join Date: 2012-11-02 Member: 165992Members
Join Date: 2012-11-02 Member: 165992Members
There are recurring topics about hacker accusations, and / or people complaining about highly skilled players ruining servers by dominating one team or the other until the server clears out.
On the flip side, I often play in non-rookie friendly servers and yet will often see complete noobs trying to comm, or just being complete wastes of space which is a huge disadvantage when everyone else in the server is decently skilled and well organized.
I think something needs to be done about this. I think there should be a built in skill rating system where you earn a skill rating of something like 1-100. This rating could be a compilation of data, some automatically collected during the game such as kills to deaths, kills to deaths in comparison to the other players, overall accuracy while in combat, rounds won vs rounds lost, and even a peer based rating. All of these factors would be combined, along with the skill rating of the server you are on (rookie, intermediate, or advanced) and influence your skill rating.
The servers would then be able to define skill filters, to only allow players within a certain range in - such as rookie 1-30, intermediate 20-75, and advanced 50-100. So the range would still be pretty broad and you would still have plenty of options for servers to join, but there would be less frustration for new players and highly skilled players alike. Rookies would no longer get stomped by someone going 45-2, organized teams would no longer have to wish the noobs computers would just explode so they leave, and pros would no longer get accused of hacking and banned from servers every day.
Also to combat team stacking my system would give servers the option to enforce the teams to be 'even' based on adding up the skill points for a given team. This would make it so if Aliens already have 500 skill points, and marines have 120, and Bob Stacker with a rating of 90 tries to join aliens? Sorry Bob, you get auto-joined to Marines.
I really think this can work, as long as the rating system was designed properly. And UWE, there is a monetary gain for you here. I have definitely heard several rookies complain about how they wasted their money, and I think its pretty clear they would never recommend their friends to buy the game. This could put an end to that as well. I think you would also see much less discussion about the game being unbalanced, since most of the time an unbalanced game is due to unbalanced teams. When someone gets dominated the first response is often to go start a thread complaining about it and why the game isn't fair.
My background - 15 years gaming experience, some competitive but very few victories. I would rate around a 65 on my 1-100 rating so I do decent but I'm no stomper. I don't mind playing with highly skilled players at all. I'm just bringing all this up because it just feels right.
Thoughts?
On the flip side, I often play in non-rookie friendly servers and yet will often see complete noobs trying to comm, or just being complete wastes of space which is a huge disadvantage when everyone else in the server is decently skilled and well organized.
I think something needs to be done about this. I think there should be a built in skill rating system where you earn a skill rating of something like 1-100. This rating could be a compilation of data, some automatically collected during the game such as kills to deaths, kills to deaths in comparison to the other players, overall accuracy while in combat, rounds won vs rounds lost, and even a peer based rating. All of these factors would be combined, along with the skill rating of the server you are on (rookie, intermediate, or advanced) and influence your skill rating.
The servers would then be able to define skill filters, to only allow players within a certain range in - such as rookie 1-30, intermediate 20-75, and advanced 50-100. So the range would still be pretty broad and you would still have plenty of options for servers to join, but there would be less frustration for new players and highly skilled players alike. Rookies would no longer get stomped by someone going 45-2, organized teams would no longer have to wish the noobs computers would just explode so they leave, and pros would no longer get accused of hacking and banned from servers every day.
Also to combat team stacking my system would give servers the option to enforce the teams to be 'even' based on adding up the skill points for a given team. This would make it so if Aliens already have 500 skill points, and marines have 120, and Bob Stacker with a rating of 90 tries to join aliens? Sorry Bob, you get auto-joined to Marines.
I really think this can work, as long as the rating system was designed properly. And UWE, there is a monetary gain for you here. I have definitely heard several rookies complain about how they wasted their money, and I think its pretty clear they would never recommend their friends to buy the game. This could put an end to that as well. I think you would also see much less discussion about the game being unbalanced, since most of the time an unbalanced game is due to unbalanced teams. When someone gets dominated the first response is often to go start a thread complaining about it and why the game isn't fair.
My background - 15 years gaming experience, some competitive but very few victories. I would rate around a 65 on my 1-100 rating so I do decent but I'm no stomper. I don't mind playing with highly skilled players at all. I'm just bringing all this up because it just feels right.
Thoughts?
Comments
So if Bob is really good, but has bad luck and always ends up on the losing team, he gets penalised?
I often wondered if anyone in my team is just done with the rookie hours. It is hard to estimate other player's skill and if the teams are fair.
I think this is a crucial lack of information that costed NS2 many players since the beta. I mean, a game with a degree of authority and skill variation that big, like the commander, should have an indication if it's worth my time playing under his command, or if I just should swap the teams in the beginning to balance the game.
Actually im pretty mad that there is no implentation of such a thing.
If they implement a 'rank' of somesort, everyone just starts stacking, because people don't like to lose. I've seen a few 'pro' players dominate on servers, and sometimes they don't even give a shit about teamplay, they just want to get 50-0 stats and then brag about it. It's really sad, then new players just stack on the same team next round.
I say a better way is to implement a force random option on round start. Could get lucky, could get even teams.. who knows.
And getting stomped is not always a bad thing. One just needs to figure out what went wrong and approach it differently next time. That's how I learned to play ns1. If one is unwilling to learn and adapt, maybe this is the wrong game..
No. Round won / lost would be a very small factor. A good player will still have good accuracy and a decent k:d ratio even if his entire team is bad.
I know what you mean about stat mongers - but people are doing it now anyway, as you said. I often see good players drag the game out just seeing how many kills they can get instead of playing the game properly. They don't even attack the base structures even if the entire enemy team is dead.
Also to combat team stacking my system would give servers the option to enforce the teams to be 'even' based on adding up the skill points for a given team. This would make it so if Aliens already have 500 skill points, and marines have 120, and Bob Stacker with a rating of 90 tries to join aliens? Sorry Bob, you get auto-joined to Marines.
There are of course holes you can punch in this plan, it is not *perfect* but it could be very close to perfect with some effort. And it would certainly be much better than the free-for-all we have now.
Honestly how many of you can say you don't have more fun during a close match? The evenly matched games are what make this game fantastic. Inversely, stacked teams make this game boring or frustrating. I really think UWE needs to put something like this in place to draw in and KEEP new players. As a side from that, everyone would generally have more fun with the game because they would be much more likely to join evenly matched games!
If you disagree, try to list the reasons in bullet points. This may not be something UWE ever implements but I really think it deserves a fair consideration.
The rank/level/skill/points/whatever should be based on an Elo-style system, not any other stats such as kdr or accuracy or time played or win/loss record.
Elo measures the only thing you actually care about: how much more likely is a team to win if the player in question joins the team? It isn't open to stat-mongering; the only way to increase your Elo is to more consistently beat better opponents, i.e. get better at the game.
Maybe you should add this little example to op
I know, I'm all in even teams. I play almost exclusively as random. On the verge of a loss, your 'uncooperate team' suddenly pulls together, the tides miraculously change, that feeling.. that got me hooked with ns1
EDIT: I take my words back, I was hasty to disagree.
But what metrics does Elo actually measure? I'm confused about this. It must be based on some kind of performance measure such as win loss, accuracy, kill rate etc. If your rating increases every time you get a kill, then that is solely based off number of kills which really is not a very accurate way to measure 'skill' since a bad player could just put in thousands of hours and have a higher skill rating than a great player who only gets in an hour or 2 a week.
I do like the idea of not showing detailed stats to the user - this would keep people from focusing on them so much. Maybe it would just give them their overall rating to let them know what "level" servers they can join.
ELO uses win/loss to rate players.
There are other systems, like Trueskill - http://en.wikipedia.org/wiki/Trueskill which would work out better for a game like NS2, imo. Instead of calculating purely from win/loss, it also matches your personal performance against that of the rest of your team. So if, in a losing game, you did pretty well, while the rest of your team did terribly, the hit to your Trueskill number would not be as significant. Or if in a winning game you did way better than the rest of your team, the bonus to your Trueskill number will be greater than what the rest of the team will get. In other words, if you do better than Trueskill thought you were going to do, you get a bigger increase. Or if you do worse than expected, you will get a bigger decrease. Doing as expected will not yeild big changes to your score (if you lose/win games that you were expected to lose/win, it's not going to noticeably impact your score much).
Sounds like a good system, but it only would be effective once most users have a skill history since initially everyone would be even. So initially the results would be skewed but after everyone has a few hours playing it would probably get more accurate.
Tribes Ascend as well as Call of Duty use such a system. And I have seen many players in Tribes who have had hundreds of hours of playtime under their belt according to their gold or platin rank but still sucked at combat and team strategies.
Likewise have I seen silver ranks wreck the faces of guys with twice as much playtime.
You can't tell anyone's skill apart in CoD either due to the prestige system. At best it tells you if someone is a rookie, but we got that same information here already. Anything beyond that point isn't covered by that system.
K/D is a bad indicator for personal skill as well. For one it makes players care too much about their death, even with classes that are supposed to be used as cannon fodder. Gorges might do a great service to their team by blocking routes and healing their team and structures up but might not get a lot of kills in the game.
A Lerk can help his team a lot with area denial by simply covering an area of the map enough to keep marines from going there. He is not awarded any points for his presence.
If my team has a hard time holding their ground, I may throw myself as solo Skulk into groups of marines where I am likely going to die a lot but will try to damage them enough or even kill one or two of them to make them reconsider their immediate attack until they got reinforcements. There are even times where I just leap around marines without caring about my own life in order to draw their attention in another direction so that the rest of my team can follow - no system can distinguish such a tactic from being a bad player.
W/L as indicator makes people blame their team for a bad personal stat and will cause more rage within some parts of the community and promotes winner-team-joining over actually helping out the team that needs it more to create an even match for everyone.
It would also clash with the surrender ability, I think. Does a surrender count the same as a loss? Will it cause more people to try hard rather than conceding when a game is actually clearly over?
And how to rate good and bad commanding? Success for alien commanders depend a lot more on his alien soldiers, success for a marine soldier depends a lot on the marine commander. One is arguably a lot easier than the other, so should it be weighted equally?
And if a rating system were to be implemented the score system would need to be much more extensive, giving more recognition to support classes and having things like assists, defends, etc taken into account.
I see, especially with your comments on k/d or point-based systems. The only system I would imagine as having any merit is a zero-sum system based on team win/loss. I maintain team win/loss is really the only important stat as *this is a team-based game*. And I do acknowledge that you have some good insight that people would be lining up to team stack to get ahead in this system.
Still I maintain team W/L is the purest, most direct metric for skill in a team-based game. So how to address your concerns?
Either :
-Always force random teams (obviously won't happen)
-Only record ranking changes on random team servers (again pretty silly)
-Normalize the rewards for winning based predicted advantages. Starcraft 2 does this. Say the teams 1 and 2's sum of player skills are 1500 to 3000, respectively. If team 1 won despite all odds, team two would stand to make some greater multiple of the standard win reward, and team 2 would be deducted that much. If team 2 won as expected, they stand to only steal a fraction of the regular win reward from the other team. *So when choosing a team, the expected ranking change should be equal for either team, at any time, ideally*
Still people would brake any such or similar system, creating fixed games against trash accounts or bots, or cheaty custom server mods. That is why I don't think any ranking system that attempts to accurately show player skill is realistic in a game with private and custom servers. At least things like the aforementioned ranks/titles for sheer number of games played gives players somewhat of a feel for roughly how decent a player is.
Elo is based on win/loss, but not simply win/loss. When you win, your score goes up. When you lose, your score goes down. But the key point is that how much it goes up or down is based on the strength of your opponent:
An Elo rating can be used to estimate the probability of a match going one way or the other. For example, if your rating is about 200 points higher than your opponent's, then you've got about a 75% chance of beating him.
Elo doesn't use any other statistics other than whether you win or lose and the Elo ratings of the people you have beaten and lost to. KDR, accuracy, time played, etc are not used. Thus, tracking people's Elo will not incentivize maximizing those ancillary stats. It will only incentivize winning games against good players.
---
That is not correct. See the Team Games section of this TrueSkill FAQ:
Continuing on:
That is sort of correct. But "doing better than expected" isn't based on your in-game score, or any other part of your in-game performance other than whether your team won or lost. The expectation part comes in when you compare your TrueSkill rating to your opponents. If theirs was higher than yours, and you won, then that's "better than expected" (because you were expected to lose) so your rating goes up a lot. But if you lose, then that's "as expected" so your rating only goes down a little bit.
There's a further complication to TrueSkill in that it also maintains an "uncertainty" level about a player's rating. The more rated games you play, the smaller the uncertainty about your rating becomes, and thus the less your rating will change for any win or loss regardless of the difference between your rating and your opponent's rating.
TrueSkill, or something like it, would be a better ranking system for NS2 then Elo, since Elo was designed for single-player games and TrueSkill was designed to handle both single-player and team games (as well as multi-sided games, although that wouldn't apply for NS2). Elo, Glick, TrueSkill, and many other similar ranking systems are all built around the same basic concept: your ranking is based on whether you win or lose games against players that are ranked higher or lower than you, such that your score reflects your statistical probability of beating another player of a different particular score.
The nice thing about Elo, TrueSkill, etc is that you don't need to take those things into account. Your skill as a support player is already recognized due to its contribution to your teams' win/loss records. If you're an awesome gorge and a lousy skulk, but you mostly play gorge and your teams mostly win, then your individual ranking will go up accordingly.
---
Elo-style systems aren't affected by team stacking, because your ranking isn't determined simply by your win/loss record - it also takes into consideration how strong your opponents are relative to you. A stacked team is expected to win, so when they win they only go up a very small amount, and when they lose they go down a huge amount.
---
I think an Elo-style (or TrueSkill-style, etc) rating system for NS2 should provide separate ratings for each player as a Commander and as a field player.
I might even suggest maintaining FIVE different ratings: marine, marine com, alien, alien com, and a composite number representing overall skill. I'd use the overall number to matchmake players between games, and the specific ratings to balance individual games.
Making all of this work correctly would be an interesting exercise in statistics. Creating and refining the Elo and TrueSkill systems has been non-trivial; extending them to work well for NS2 would also be non-trivial. But it would be fun, and a very valuable addition not only to the game but to the field of game science in general.
Then the "points" you earned, plus the "skill" of the opposing team in comparison to your teams skill, which team won, combined with some other potential factors, would invoke a calculation that I'm not smart enough to figure out that would determine how many skill points you should win or lose for that game.
Also, I do not think players should lose skill points very easily. If you have a few bad games or play a few pro teams you should not be sent back into noob land. The skill system would mainly be governed by gaining points, but it would not be easy to do so - you would need to play against other teams who have nearly equal or higher skill to your own to win any significant points. If you lose, no big deal you don't lose many points. Kind of like you just 'failed to advance' . But if you lose over and over against teams worse than you then yes you should be knocked down a notch.
As for commanding, you bring up an excellent point. There is no real way the game could automatically determine commanding skill. So for this it would have to be a peer based rating system. At the end of the round, you would be prompted with a small menu asking to rate your commander on a 1-10 scale, 1 being terrible 10 being the best comm you've ever seen. This rating would carry more weight for players with a higher 'skill' rating since they typically know the game better.
It would obviously take a lot of fine tuning and there would be many many variables, but I think UWE could do it, and I think the community would appreciate it!
Think of how it would apply in a multi-team game, if players changed teams normally, when players have more than one generic playstyle (e.g. classes). These are the tips of very large icebergs which ELO and True Skill ignore. Notwithstanding all distinct play attributes/tactics, and more (There's a lot more).
Anyways, in lack of anything else, yeah, sure, lets do the true system. Going to suck for non-team stackers though, and forced match making isn't going to happen globally. Anyways, my whole point is, there's often a lot more to something than what you're seeing, so be careful.
What I think could be interesting to look into is start collecting all kinds of statistics that might have something to do with individual skill. Also, add in a peer based rating system, averaging the player's score out of 100. Then use the statistics you've collected in the first part, and use an AI algorithm to get it to learn off the peer based score - thereby developing a 'best fit' for matching the statistics to the peer scores (e.g. numbered weightings on statistics like kills and deaths, rounds played etc where the weighted statistics correspond to the peer scores as closely as possible). Granted, the peer scores might not be perfect (e.g. a new player might rank somebody who's just okay as a '100/100' cause he/she's never encountered a player who would deserve such a ranking), and also AI learning algorithms don't guarantee accuracy even with lots of data (sometimes too much data can even be worse than not enough in AI, as it overcompensates by creating too much of a fit for the sample, and begins to misrepresent the whole picture)... however I think it may be a good starting off point.
The idea is that most people WILL hit full rank if they play enough. Rank would be based on the number of games played as a race or as a comm.
*** Win/loss does NOT matter. The goal is to show experience, not victories. ***
50 games as alien/marine = full rank alien/marine respectively
10 games as commander/khaamander = full rank as comm/khaam respectively
After 50 games, most people know the ins and outs of the game and maps.
How would this be displayed? Maybe a 4 digit number, each digit representing marine/alien/marinecomm/aliencomm. 9 would be highest rank so 9999 would be full rank in everythign.
This would ONLY be visible in the ready room and before the game starts. This discourages people who join later from camping the readyroom for an open spot on a "strong looking" team.
I disagree wholeheartedly. The system you describe would still lead to uneven teams constantly, it would just protect rookies a little better.
The system I have in mind would actually rank your raw skill as a player, which is really the only important thing to consider when matching teams. How exactly we determine raw skill is still up for discussion - I still think it should at least use the number of kills compared to the rest of your team / other team at the end of the match as a factor among other factors. Maybe also consider the lifeform you were for the majority of those kills.
Consider this example: some highly skilled player joins an evenly matched game- but everyone else is way worse than him. He racks up 50 kills and dies only once because he is just unstoppable, the other players can't even kill him when they gang up on him. This guy single handed won the game while the rest of his team just followed his trail of destruction. So does it really make sense for the entire team to get the same mediocre skill boost? No it doesn't. That one super player should get a huge boost in his skill rating, so that after a few dozen games like this he is classed up enough to where he is forced to join higher skilled servers and play with players on his own level.
It may work in a game like Quake... or chess... but NS is more complex! What about gorges and commanders? Skulks or marines killing all the RTs at all the right places, avoiding opposition, or damaging it down to a quarter HP and someone else taking care of the rest? Building the RTs or defending it, which may not mean getting kills, just keeping enemies at bay? Taking out objectives without regard for your own life, like a power node on which half the aliens will die, but it will go down so it's not a problem? Being the first or second to run through a PG under attack by multiple aliens? With ELO you're encouraging players to be scared of dying and discourage them from taking up roles that doesn't include getting a lot of kills.
However, if UWE did some sort of ranking in the background, not visible to players or servers only to be used for matchmaking that could be useful as it's only quietly observing. Once you start assigning ranks or scores to players it will change how they play to fit that system. That's a problem unless it's a 1:1 mapping of actual usefulness of a player in the game which ELO isn't, and it would be very difficult to make such a ranking system.
I actually already touched on all your concerns in previous posts. I agree with you and that stuff could all be handled with the built in "Points" system NS2 has. It just needs to be adjusted to award points to people who do things to help the team as a whole but may sacrifice their K:D ratio as you said. Award people points for every hit on an important structure, or for every second of building / repairing stuff etc.
The system may never be perfect, and you are correct in that some people will find ways to artificially ramp up their skill rating - but what will be the result of this? Mediocre players will end up 'ranked' very highly, and any time they play, they will start to get destroyed by the superior players in the game. If they can't ever kill anyone their rank will start to come down slowly because the system I have in mind would take into consideration all things to award skill points.
that's enough of a starting point to avoid situations where all the experienced players are on one team
I think if win/loss or rating scores are publicized it can hurt in several ways. anytime you give players a rating, they will do lame things to try to boost it - leaving matches early, trying to stack teams, waiting to steal kills, etc.
gain levels fast in the beginning but you really have to work for it to gain your rank once you get to higher levels. Maybe it could be attached to how many hours you've spent playing?
I was thinking steam's 'xx hours played' at first, but you could just as well just start the game, tab out and gain levels.
Some sort of "master server" might be too much of a hassle.. Hm, wonder if it's possible.
With that said, the standard public games don't need a ranking system, you as players should be able to gauge one another after 1 game and proceed to stack, or balance, the teams accordingly.
If there were matchmaking however I would expect a ranking system (I like the sound of trueskill!) to be in place.
Unfortunately this is not really true, at least not for most of the games I have played in my 300-ish hours of NS2. Of all the highly skilled players I have encountered, I would estimate less than 10% of them ever make any effort to balance the teams. They all love showing off their huge kill count and many of them are actually terrible team mates, they steal kills to inflate their ratio even further. If you truly want to discourage this type of behavior we would need to remove kills / deaths from the scoreboard entirely. Not that I am suggesting that. I think we need ranking based largely on our best measure of 'skill' we can manage to come up with, combined with either a matchmaking system or an auto-team balance system.
K/d is not. Time is not. Points are not.
The most effective way of balancing teams I know of is electing captains and having them choose people in turns.
Also, a global ranking system would require a server of some kind to process this. It seems much more uwe style to have ranking be on per individual server and processed by that server. Most servers are community hosted. Servers I guess could subscribe to a global data sheet too if one were to be made.
I just had this idea...
What if everyone self ranked themselves out of 1-100? If you rate yourself a 65 and play with a bunch of 60-70 people and notice consistently do a lot worse than them, you would lower your rating, or vice versa if you were doing better consistently. It would only work if one was honest. I think there is a strong incentive to be honest though. If you rate yourself highly and often do poorly you will want to lower your rank because it will balance the game funny. It probably would take a game or two to figure out your rank in comparison to the players you are currently playing.
The numbers are self rated and don't actually mean much so bragging "I am xx score I am so awesome" would just be plain idiotic.
Someone could be dishonest. Intentionally rating yourself higher would not help you win more in theory because you would be balanced with lower rated people on your team. Intentionally rating yourself lower to be balanced with higher rated people as less preventing it though. This would create a race to the bottom if everyone did it and would make the whole rating pointless so the majority would not do it. If it was only a minority then it would just be an annoyance because 14/16 players are properly balanced.
It would be an interesting social experiment no matter what happened.
Perhaps a peer based rating system? Rating would be optional, or random so that each player does not have to rank everyone on the server every round. The rank you give would carry more weight depending on the rank of the rater. There will of course be idiots & trolls throwing bad ratings out there but as long as the majority of the people ranked eachother fairly it could work to provide a 'rough estimate' of your skill level, just enough to come up with somewhat balanced teams.