It still seems broken. Look at No.1 on the hive stats. He's getting consistently 30 points per win. Surely he should be getting less points per win the higher skill he is because he's playing people with less skill.
It's looking atm a lot like the "Total score" leaderboard in that the more time you spend, the more "skill" you have.
You know, i checked him out yesterday and the sight of nothing but wooza on match history was pretty amusing to begin with, but i decided to go and test the server for the first time, i left at 11min mark when the warping began.
I thought servers like that wouldn't be on the whitelist tho, but ye system is bit flawed still.
i've been tracking my skill for 90 games now (since the fix), so here is some data and plots (but keep in mind that this is just 1 personal example so not everything can be generalized from this).
been playing on various servers inbetween 16 and 24 slots and usually vanilla (although there might have been compmod sometimes, i usually try to avoid it though), both races, almost always doing my best to win a match. roughly like 3-10 games per server in a row before switching the server. i vote force even teams basically whenever possible (a minor amount of servers still has it disabled) but it will not always succceed and some people switch sides after the vote. i don't really care wether a server is rookie friendly or not (emphasis on performance, decent ping and being almust full) and i guess more than half were rookie friendly. i avoid servers with red double-down skill icon but play on all others. i have 1369 hours of steam-gametime, 1.15 winrate (807 recorded games on hive) and sometimes play commander. time spent on either race is 107-108 hours in hive.
raw data values (if you have matlab, easily plot it with bar(y) ):
absolute difference between one game and the next game:
average difference between one game and the previous 9 games (the first <10 games are averaged with less games which results in more extreme values)
absolute average difference between one game and the previous 9 games (the first <10 games are averaged with less games which results in more extreme values)
some might now think "this doesn't converge nicely even after 90 games!" but keep in mind that most other players will probably have played less games so far, which in turn influences my own value to jump around since their values might not be representative enough yet.
still, i don't really see a pattern of the changes becoming less per game, so it might indeed be worth looking into that.
one thing i regret: should have kept track of server switches. the only thing i have now are the last ones, so the amount of games per server in reverse order is: 10, 1, 4, 4, 1+ (dunno how often for the last one).
also yes: there was one game where my skill value didn't change at all. the game lasted 1h 3min but i actually played there for quite a while (hard to say how long, i guess 20 mins) and eventually won the match.
Out of curiosity I made a similar graph for my Hive stats as @Laosh'Ra did. My statistics is only over 20 rounds as seen in the Hive history.
The red line is the short trend (3 round moving average) and the black line is a long term trend (17 rounds).
The dip at the end seems to be due to a loss-streak as shown in the below graph where I overlay the change in skill with win/loss markers.
As there are many factors influencing the skill-value now it is evident from the last graph that it is possible to loss a game and get a positive gain in skill. AND it is possible to win a game but get a loss in skill-value.
@Laosh'Ra mentions that the skill-value doesnt appear to converge, I want to add to this comment.
The convergence rate will depend on what the intended convergence rate of the skill-based system is. I only had a brief read through the calculations of the update formula and can acknowledge that the model is not unreasonable. However, in order to say if the expected fluctuations will be within 1 or 2 (now normal distributed) standard deviations we need more data and a better analysis of the model. (My analysis is quite simplified since the model does not assume normal distributed player skills).
I think there are still some issues with it. Not enough to require a reset of the skills, but it's not quite working properly yet. I'm talking to Acedude, and we'll hopefully have it sorted out soon.
Was there any issues with the skill system? Have you had time to look at it yet?
As far as I can tell, quite a few servers in NS2 involve communities of regulars that tend to only or primarily play on that specific server, with largely the same group of people.
Is it fair to say that the global accuracy of the skill values is dependant on players intermixing?
And has this been taken into account when developing the system or is the mix rate such that the risk of 'skill pools' (groups of players for whom the skill is accurate between, but lacks predictive power outside of that group) is negligable?
Is there any idea of when the skill system might be fixed? I really thought the new v v, v, =, ^, ^ ^, tags would help a lot, but ever since the last reset nearly ever server shows as v v. Very rarely do I see a v, and never ^ or ^ ^, and the only time I ever see = is when I'm the only one in the server.
Is there any idea of when the skill system might be fixed? I really thought the new v v, v, =, ^, ^ ^, tags would help a lot, but ever since the last reset nearly ever server shows as v v. Very rarely do I see a v, and never ^ or ^ ^, and the only time I ever see = is when I'm the only one in the server.
that mostly means that you are above the average skill value of most servers, that doesn't mean the skill system is broken.
I'm pretty sure what we currently have implemented isn't working as @moultano originally presented.
What we have now is too limited in the amount that can be gained or lost - this means that essentially the more you play, the higher your skill will be (you can only lose a small amount even if on the obviously stacked team that loses to eg a rush): provided that you maintain a win/lose ratio of greater than 1, your skill will be ever-increasing and simply becomes a function of how many hours you put in. This is clearly not the intended behaviour!
As the skill goes up, so should the amount you can lose when on the losing team (as you have a high skill, the team should have a high skill and thereby lose more if they lose the round).
The only exception to this would be that you join a weaker team overall, but honestly, I cannot believe that this is working as intended - just look at all of the players on the first page of hive: it's statistically extraordinarily improbable that EVERY single round won by those players has been against a higher-skill stacked team.
Is there any idea of when the skill system might be fixed? I really thought the new v v, v, =, ^, ^ ^, tags would help a lot, but ever since the last reset nearly ever server shows as v v. Very rarely do I see a v, and never ^ or ^ ^, and the only time I ever see = is when I'm the only one in the server.
that mostly means that you are above the average skill value of most servers, that doesn't mean the skill system is broken.
Well the thing is it's not most servers like 3/4ths of them.. it's most servers like 99.9% of them... and when I join a server with v v and it's full of players like loMe, Emo, BillyTalent, etc... who are so far above me in terms of skill it's not even funny... it must be broken somehow.
For a few days right after the system was introduced it seemed to be giving somewhat accurate readings for the most part... then there was that reset and ever since it's been nothing but a sea of v v.
people on hive have only small steps inbetween their skill value and the next better/worse player (the list below 6 goes on like this). the two exceptions on top both have a very high winrate and tons of wooza-games. so maybe there are some issues with higher player counts?
I stand by my comments entirely! nothing in your post disagrees with what I said: if you have a win to loss ratio of more than 1, you will only go up in skill.
The modifier to your skill should probably be multiplicative rather than additive, perhaps with logistic weighting to ensure that there is a maximum and minimum possible value for skill and also that these values are essentially asymptotic...
As far as say, premier level players, does the skill system track their stats on CompMod servers, or just vanilla?
If the skill system doesn't track stats for CompMod, Comp players who rarely play anything tracked (i.e. public servers) will take much longer to become accurate in the system....right?
Why it ALWAYS has to shuffle me to the other side i'm currently in? I mean like ALWAYS. Few days ago when ppl were voting to enforce the balance i was typing chat: "NS2, don't even try to shuffle me to alien side (i was marine)" and just when i hit ENTER, it shuffled me to alien side. And i was like FUUUUUUUUUUUUUUUUUUUUUUUUUUUUUK. Some people though that was funny. I didn't feel that way...
moultanoCreator of ns_shiva.Join Date: 2002-12-14Member: 10806Members, NS1 Playtester, Contributor, Constellation, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Gold, NS2 Community Developer, Pistachionauts
I finally got a chance to sit down and look at the code properly and make some changes. We'll have to wait til Acedude has some free time to update it and check over it.
I finally got a chance to sit down and look at the code properly and make some changes. We'll have to wait til Acedude has some free time to update it and check over it.
For those of us who like technicality, may we hear what has changed?
AsranielJoin Date: 2002-06-03Member: 724Members, Playtest Lead, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow, Subnautica Playtester, Retired Community Developer
And no need for skill resets, the major problem was that the values did not converge correctly, your skill would jump up and down too much. But overall the skill value would still be correct (good players are rated better than worse), with the bugfix there will just be less noise in data
Has the live system been changed already? Since today (Aug 31), my skill value seems to fluctuate a lot less. I was winning/losing 30-40 points per game before, today the biggest change was 12 points, with most around 4-6 points.
Why it ALWAYS has to shuffle me to the other side i'm currently in? I mean like ALWAYS. Few days ago when ppl were voting to enforce the balance i was typing chat: "NS2, don't even try to shuffle me to alien side (i was marine)" and just when i hit ENTER, it shuffled me to alien side. And i was like FUUUUUUUUUUUUUUUUUUUUUUUUUUUUUK. Some people though that was funny. I didn't feel that way...
I tend to get shuffle to marines (or stay marine). I wonder how the shuffle system works? I would like it to rank the players in the server and do a captain's pick (team 1 gets top player, then team 2 gets next 2 etc...). As it is, I feel it is trying to match the skill levels of the 2 teams to be roughly equal, and I see some horrible stacks when we shuffle (like the top 3/4 of the top scorers last game is in the same team).
Why it ALWAYS has to shuffle me to the other side i'm currently in? I mean like ALWAYS. Few days ago when ppl were voting to enforce the balance i was typing chat: "NS2, don't even try to shuffle me to alien side (i was marine)" and just when i hit ENTER, it shuffled me to alien side. And i was like FUUUUUUUUUUUUUUUUUUUUUUUUUUUUUK. Some people though that was funny. I didn't feel that way...
I tend to get shuffle to marines (or stay marine). I wonder how the shuffle system works? I would like it to rank the players in the server and do a captain's pick (team 1 gets top player, then team 2 gets next 2 etc...). As it is, I feel it is trying to match the skill levels of the 2 teams to be roughly equal, and I see some horrible stacks when we shuffle (like the top 3/4 of the top scorers last game is in the same team).
I'm reading through that code to try to figure out how it works. I think it uses a blend of your score from last round with your skill value, sorts the players by skills and assigns them to alternating teams going down the list. Still getting a handle on it though.
skill system seems to be a measurement of win vs loss ratio as opposed to how good a player is, i have more loss games than wins in general but that doesnt mean im a bad player. it never seems to progress much, goes up goes down but stays low even though i have the highest score in the games i play usually
Comments
It's looking atm a lot like the "Total score" leaderboard in that the more time you spend, the more "skill" you have.
I thought servers like that wouldn't be on the whitelist tho, but ye system is bit flawed still.
been playing on various servers inbetween 16 and 24 slots and usually vanilla (although there might have been compmod sometimes, i usually try to avoid it though), both races, almost always doing my best to win a match. roughly like 3-10 games per server in a row before switching the server. i vote force even teams basically whenever possible (a minor amount of servers still has it disabled) but it will not always succceed and some people switch sides after the vote. i don't really care wether a server is rookie friendly or not (emphasis on performance, decent ping and being almust full) and i guess more than half were rookie friendly. i avoid servers with red double-down skill icon but play on all others. i have 1369 hours of steam-gametime, 1.15 winrate (807 recorded games on hive) and sometimes play commander. time spent on either race is 107-108 hours in hive.
raw data values (if you have matlab, easily plot it with bar(y) ):
some might now think "this doesn't converge nicely even after 90 games!" but keep in mind that most other players will probably have played less games so far, which in turn influences my own value to jump around since their values might not be representative enough yet.
still, i don't really see a pattern of the changes becoming less per game, so it might indeed be worth looking into that.
one thing i regret: should have kept track of server switches. the only thing i have now are the last ones, so the amount of games per server in reverse order is: 10, 1, 4, 4, 1+ (dunno how often for the last one).
also yes: there was one game where my skill value didn't change at all. the game lasted 1h 3min but i actually played there for quite a while (hard to say how long, i guess 20 mins) and eventually won the match.
The red line is the short trend (3 round moving average) and the black line is a long term trend (17 rounds).
The dip at the end seems to be due to a loss-streak as shown in the below graph where I overlay the change in skill with win/loss markers.
As there are many factors influencing the skill-value now it is evident from the last graph that it is possible to loss a game and get a positive gain in skill. AND it is possible to win a game but get a loss in skill-value.
@Laosh'Ra mentions that the skill-value doesnt appear to converge, I want to add to this comment.
The convergence rate will depend on what the intended convergence rate of the skill-based system is. I only had a brief read through the calculations of the update formula and can acknowledge that the model is not unreasonable. However, in order to say if the expected fluctuations will be within 1 or 2 (now normal distributed) standard deviations we need more data and a better analysis of the model. (My analysis is quite simplified since the model does not assume normal distributed player skills).
Was there any issues with the skill system? Have you had time to look at it yet?
As far as I can tell, quite a few servers in NS2 involve communities of regulars that tend to only or primarily play on that specific server, with largely the same group of people.
Is it fair to say that the global accuracy of the skill values is dependant on players intermixing?
And has this been taken into account when developing the system or is the mix rate such that the risk of 'skill pools' (groups of players for whom the skill is accurate between, but lacks predictive power outside of that group) is negligable?
that mostly means that you are above the average skill value of most servers, that doesn't mean the skill system is broken.
What we have now is too limited in the amount that can be gained or lost - this means that essentially the more you play, the higher your skill will be (you can only lose a small amount even if on the obviously stacked team that loses to eg a rush): provided that you maintain a win/lose ratio of greater than 1, your skill will be ever-increasing and simply becomes a function of how many hours you put in. This is clearly not the intended behaviour!
As the skill goes up, so should the amount you can lose when on the losing team (as you have a high skill, the team should have a high skill and thereby lose more if they lose the round).
The only exception to this would be that you join a weaker team overall, but honestly, I cannot believe that this is working as intended - just look at all of the players on the first page of hive: it's statistically extraordinarily improbable that EVERY single round won by those players has been against a higher-skill stacked team.
Well the thing is it's not most servers like 3/4ths of them.. it's most servers like 99.9% of them... and when I join a server with v v and it's full of players like loMe, Emo, BillyTalent, etc... who are so far above me in terms of skill it's not even funny... it must be broken somehow.
For a few days right after the system was introduced it seemed to be giving somewhat accurate readings for the most part... then there was that reset and ever since it's been nothing but a sea of v v.
I just don't get it...
people on hive have only small steps inbetween their skill value and the next better/worse player (the list below 6 goes on like this). the two exceptions on top both have a very high winrate and tons of wooza-games. so maybe there are some issues with higher player counts?
edit: just noticed, there are people with negative skill values again... http://hive.naturalselection2.com/rank/skill/page-555
The modifier to your skill should probably be multiplicative rather than additive, perhaps with logistic weighting to ensure that there is a maximum and minimum possible value for skill and also that these values are essentially asymptotic...
If the skill system doesn't track stats for CompMod, Comp players who rarely play anything tracked (i.e. public servers) will take much longer to become accurate in the system....right?
For those of us who like technicality, may we hear what has changed?
I tend to get shuffle to marines (or stay marine). I wonder how the shuffle system works? I would like it to rank the players in the server and do a captain's pick (team 1 gets top player, then team 2 gets next 2 etc...). As it is, I feel it is trying to match the skill levels of the 2 teams to be roughly equal, and I see some horrible stacks when we shuffle (like the top 3/4 of the top scorers last game is in the same team).
I'm reading through that code to try to figure out how it works. I think it uses a blend of your score from last round with your skill value, sorts the players by skills and assigns them to alternating teams going down the list. Still getting a handle on it though.