Skill System is Good

13468916

Comments

  • UncleCrunchUncleCrunch Mayonnaise land Join Date: 2005-02-16 Member: 41365Members, Reinforced - Onos
    moultano wrote: »
    I'm reading through that code to try to figure out how it works. I think it uses a blend of your score from last round with your skill value, sorts the players by skills and assigns them to alternating teams going down the list. Still getting a handle on it though.

    Why reading this ? Better make a new one that is properly connected with the skill system.
  • RoobubbaRoobubba Who you gonna call? Join Date: 2003-01-06 Member: 11930Members, Reinforced - Shadow, WC 2013 - Shadow
    MrFangs wrote: »
    Has the live system been changed already? Since today (Aug 31), my skill value seems to fluctuate a lot less. I was winning/losing 30-40 points per game before, today the biggest change was 12 points, with most around 4-6 points.

    @AceDude, @moultano‌ this doesn't sound right to me. The amount a player should be able to lose or gain in points should not be so limited - if a team of complete greens wins against a team composed of hive-front-page players, then the losing team players should lose a LOT of points, and the greens should gain a LOT of points. It's going to be an extreme edge-case scenario, but reflects the fact that the probability of the scenario is very small. Likewise, in the same setup, if the greens lose, they should lose no points and the winning 'pro' team should gain no points, as the outcome is essentially inevitable.
    In the previous incarnation, there seemed to be little variance in the amount gained and lost, but players on the front page still seemed to be gaining lots of points (ie up to the apparent cap of +40) for winning games, which doesn't seem right to me if they're apparently so good (high 'skill' score) that they can reasonably be expected to carry most teams to victory.

    Are you going to publish the algorithm(s) currently implemented for this?

    Cheers

    Roo
  • MrFangsMrFangs Join Date: 2013-03-27 Member: 184474Members, Reinforced - Shadow
    Roobubba wrote: »
    MrFangs wrote: »
    Has the live system been changed already? Since today (Aug 31), my skill value seems to fluctuate a lot less. I was winning/losing 30-40 points per game before, today the biggest change was 12 points, with most around 4-6 points.

    @AceDude, @moultano‌ this doesn't sound right to me. The amount a player should be able to lose or gain in points should not be so limited - if a team of complete greens wins against a team composed of hive-front-page players, then the losing team players should lose a LOT of points, and the greens should gain a LOT of points. [...]

    Actually, the new score changes look more sensible than the old ones (if it wasn't pure coincidence). Before, I tended to lose 30-40 points even when losing against a stacked team, and likewise, when winning *with* a stacked team. Intuitively, there should have been very little change, if at all, because that would be the expected outcome of those stacked matches. (Of course, that assumes that the current skill values loosely match actual skill already.)

    In yesterday's games, there were some stacked games (with the expected results) again, which led to the 4-6 point changes, and only one balanced game, which resulted in the 12 point change. So this is looking better than before - again, if it wasn't just coincidence.
  • KungFuJVKungFuJV Join Date: 2003-04-03 Member: 15167Members
    I wonder how the shuffle system works? I would like it to rank the players in the server and do a captain's pick (team 1 gets top player, then team 2 gets next 2 etc...). As it is, I feel it is trying to match the skill levels of the 2 teams to be roughly equal, and I see some horrible stacks when we shuffle (like the top 3/4 of the top scorers last game is in the same team).

    The server I mainly play on does captains only in order to pick teams. The thing is, since there is no mod it relies on the player themselves to assign captains and pick teams( which can sometimes take a while).

    On the plus side the games are considerably more even and if it isn't they will usually repick after the round ends. I have had some of the most even, communicative, back and forth games on those servers. Things I rarely find on non captain servers lately
  • moultanomoultano Creator of ns_shiva. Join Date: 2002-12-14 Member: 10806Members, NS1 Playtester, Contributor, Constellation, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Gold, NS2 Community Developer, Pistachionauts
    @Roobubba‌ We might need to increase the update rate at bit at least intially, but I think it's taking the probabilities into account correctly now. http://hive.naturalselection2.com/profile/53772110 Look at the games since september 1. On every win he's gained <10 points, sometimes 1 or 2, and on losses he's lost 20-30 points.

    The updates are always going to be bounded unless we do something substantially more complicated like optimizing the scores continuously (I would love to do this, but not sure if any of us will have time.) You are never going to see huge swings on one game. The magnitude of the updates corresponds to how much we think recent games matter more than older games. If we update quickly, then it's equivalent to effectively throwing out more of that history.
  • lwflwf Join Date: 2006-11-03 Member: 58311Members, Constellation
    Can't you recalculate all the skills from the raw data from previous matches now that the algorithm has changed? Surely you have taken this into account and saved all the raw data so it can be replayed for debugging, experimentation and adjustments.
  • moultanomoultano Creator of ns_shiva. Join Date: 2002-12-14 Member: 10806Members, NS1 Playtester, Contributor, Constellation, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Gold, NS2 Community Developer, Pistachionauts
    Btw, @Roobubba‌ I think you might like this article: http://moultano.wordpress.com/2013/08/09/logs-tails-long-tails/ It talks a little bit about the choice of model and why it matters.
  • moultanomoultano Creator of ns_shiva. Join Date: 2002-12-14 Member: 10806Members, NS1 Playtester, Contributor, Constellation, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Gold, NS2 Community Developer, Pistachionauts
    lwf wrote: »
    Can't you recalculate all the skills from the raw data from previous matches now that the algorithm has changed? Surely you have taken this into account and saved all the raw data so it can be replayed for debugging, experimentation and adjustments.

    Ideally, but it's a bit of work. What I'd really like to do is in the faster convergence section here: http://moultano.wordpress.com/2014/08/04/a-skill-ranking-system-for-natural-selection-2/

    But we'd need a machine to burn through some cpu on and access to the database, so probably not going to happen soon.
  • NordicNordic Long term camping in Kodiak Join Date: 2012-05-13 Member: 151995Members, NS2 Playtester, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow
    edited September 2014
    moultano wrote: »
    lwf wrote: »
    Can't you recalculate all the skills from the raw data from previous matches now that the algorithm has changed? Surely you have taken this into account and saved all the raw data so it can be replayed for debugging, experimentation and adjustments.

    Ideally, but it's a bit of work. What I'd really like to do is in the faster convergence section here: http://moultano.wordpress.com/2014/08/04/a-skill-ranking-system-for-natural-selection-2/

    But we'd need a machine to burn through some cpu on and access to the database, so probably not going to happen soon.

    Having a cpu to do the work wouldn't be that hard I don't think. Access to the data is another question. I donate cpu resources for medical science all the time.
  • amoralamoral Join Date: 2013-01-03 Member: 177250Members
    MoFo1 wrote: »
    Is there any idea of when the skill system might be fixed? I really thought the new v v, v, =, ^, ^ ^, tags would help a lot, but ever since the last reset nearly ever server shows as v v. Very rarely do I see a v, and never ^ or ^ ^, and the only time I ever see = is when I'm the only one in the server.

    you gotta really appreciate where "average" actually falls. when i play i'm out of my league when i'm getting 1:2 kills on one other guy on the team... but this means that i'm still tearing through the rest of the team pretty consistently. it sometimes feels like even if 5 other guys in the game are 1:1 k/d with you, you've still got filler of 10 other guys on the team that make the server a double down arrow.

    also remember, some of the monsters don't play many pubs.
  • MrFangsMrFangs Join Date: 2013-03-27 Member: 184474Members, Reinforced - Shadow
    Ok, something weird happened. We started a match with low player count, another fairly experienced player + me + 2 rookies on Marines, one rookie and two experienced players (IIRC) on Alien.

    We did pretty ok for the first 10 minutes, given that we were mostly explaining stuff to the two rookies on our team. But over time, more players joined - and all the rookies joined Marines, while all the experienced players went Alien. With these teams, of course we lost.

    But then, after the match, I checked my Hive stats - and discovered that I lost a whopping *113* points :-O from that game:
    0ZbWvh9.png

    I'd say that's not what losing with a rookie team should result in?

    Also, isn't it weird that you can drop below 1000 fairly easily by a few unlucky losses (including troll/bad comm etc.), while rookies start with a score of 1000, so Hive considers you *worse* than a player with zero experience? I remember there were plans to initialize the score based on play time - this would give a much better starting point, I think.
  • meatmachinemeatmachine South England Join Date: 2013-01-06 Member: 177858Members, NS2 Playtester, NS2 Map Tester, Reinforced - Shadow, WC 2013 - Supporter
    Setting the default skill value to 1000?
    ...
    ROOKIE MISTAKE B-)
  • UncleCrunchUncleCrunch Mayonnaise land Join Date: 2005-02-16 Member: 41365Members, Reinforced - Onos
    Hi,

    The idea of a factor has been submitted and is frankly better as long as ppl are able to see their hive stats.

    There are 2 ways:
    A / ppl can see their stats.
    B / It is decided that it should be hidden.



    The factor would be a number that will multiply a basic game result number :
    +1pt for loosing
    +2pt for winning

    Then comes the result :
    -Team supposed to win, loose= * 0
    -Team supposed to win, win= * 2
    -Team supposed to loose, loose= * 3 (we want them to enjoy play more)
    -Team supposed to loose, win = * 5


    -Stackers won't get much points; they still enjoy game as they played with stackers.
    -Rookies progress faster. As they get points they don't give up on NS2. If scenario A (keep stats available) is kept, make a dev to show them in-game they earned something.
    -Nobody loose point.
    -It becomes a matter of speed of progression.


    It can be limited with another factor like "hours played". so nobody can go above a ceiling value. The more you get close to 2000h the more the time factor get close to 0 but always above 1. The experience accumulated is the right indicator for randomization tool. Getting to top won't be out of reach for rookies.

    Adjusting value so it won't go too fast or too slow is just a matter of what precision is wanted from 0 to 100 or from 0 to 100000.


    Simple, easy to implement even for a simple mind.

  • RoobubbaRoobubba Who you gonna call? Join Date: 2003-01-06 Member: 11930Members, Reinforced - Shadow, WC 2013 - Shadow
    Also, with the previous system giving +30-40 points for essentially all wins, there are some players who've essentially started out at a relatively high value since the recent changes (like 4000 or so). I don't have a feeling for how quickly those will converge to 'sensible' values under the new scheme, but in general it seems to be working approximately as expected.

    @moultano I think you're right about the rate needing to be increased a bit, or if there's a way to yoink all historical data from hive and run the calculations offline to give a set of more likely starting values, that'd help overall.

    As it is, someone with a very high skill score (who maybe got it under the previous system after the skill reset simply by playing a lot of matches), increases the expectation of a win by an otherwise average team - then when they lose, ALL the players lose a lot of points and are punished for the inaccuracy of the initial score for that one 'high skill score' player. Of course, this should result in a lower expectation of victory next time, which should result in a higher score awarded for win, but it will take a very long time for that one very high score to get back on scale with the others - I think we need some fixing of the initial scores somehow!

    The number of points you lose/gain is not a function of your own individual score but of the team's collective score. However, what's not clear to me because I haven't seen the algorithms in use, is whether the score on hive is actually a log probability (and therefore the addition/subtraction applied is actually a multiplication applied to the probability used in the calculations), or if it isn't a log. If it's not a log, then I think the current method of adding/subtracting scores may be flawed, again unless that addition/subtraction is just showing us the result of an underlying multiplication...

    It's interesting that the UCSF switched from a gaussian to a logistic model, and actually the logistic model could make a lot of sense for this system as well: the chances of a much worse team winning a round against a better team aren't actually very small: there are lots of rounds that are won through base-rushes, and maybe the commander of the stacked team isn't that great and misses the beacon/doesn't place another chair etc - I've seen this happen a lot in my own limited sample of games!


    Or am I missing the point? Would be great to see the equations currently in use :)
  • _INTER__INTER_ Join Date: 2009-08-08 Member: 68392Members, NS2 Playtester, Reinforced - Shadow
    edited September 2014
    The factor would be a number that will multiply a basic game result number :
    +1pt for loosing
    +2pt for winning

    Then comes the result :
    -Team supposed to win, loose= * 0
    -Team supposed to win, win= * 2
    -Team supposed to loose, loose= * 3 (we want them to enjoy play more)
    -Team supposed to loose, win = * 5
    Baad. Those playing more will rise on top no matter what. Even if they're complete shit and it doesn't prevent stacking / rookie farming too. Even worse than previous system.
  • UncleCrunchUncleCrunch Mayonnaise land Join Date: 2005-02-16 Member: 41365Members, Reinforced - Onos
    _INTER_ wrote: »
    Baad. Those playing more will rise on top no matter what. Even if they're complete shit and it doesn't prevent stacking / rookie farming too. Even worse than previous system.

    It's already the case. They already are on top as each game they play and win gives them the 30pt (or 40?, never mind). As some point, with experience an newbie or average player starts to win more. At least not being a dead weight for the team.

    Stack ? Who can seriously pretend to prevent stack today ? The Moultano system (and previous btw) doesn't.
    As i recall he stated himself : The system just try to balance upon whats left in RR (more or less). It means for the ones who wants to stack to join faster... ... great.

    If rookies get farmed it doesn't really matter simply because it doesn't change from now (on any type of game either). They already are, they will be. The thing is that they can see they earn something by playing. And if they don't get better at the game; they will finally just leave as they think it's boring (loosing is boring). So it won't be a disturbance for randomization as they just won't join a game.

    The Ultimate Dumb may exist of course, but you have to find the dumbest guy in the world not to see someone progress in this game by playing on a regular basis. At some point a young recruit (i hate that rookie word) gets that RT are money...

    I don't pretend that this system is accurate 10 decimals after the mark. It just don't produce examples like MrFang showed. The new system was supposed to change things for the better. We aren't there yet... Obviously not. Waiting for the system to converge isn't realistic as it doesn't prevent stacking. So doesn't promote player retention by extent. It's just a fancy way to make calculation on a predicted failure.

    Having all individual win something is better. If you can loose something, ppl start to do whatever they can to get more (incentive). Stacking goes along with this.

  • NordicNordic Long term camping in Kodiak Join Date: 2012-05-13 Member: 151995Members, NS2 Playtester, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow
    When the server thinks it is even for me, I am feeling it is even. What I think are even matches are making me gain or lose 1-3 points. It might be working.
  • HarmoniusZHarmoniusZ Poland Join Date: 2013-11-06 Member: 189038Members
    UncleCrunch, do you really think that for rookies (or just an average player) "getting better" means to get high rating, to move up some kind of ranking and that this kind of "certified" progress keeps them playing the game?
  • meatmachinemeatmachine South England Join Date: 2013-01-06 Member: 177858Members, NS2 Playtester, NS2 Map Tester, Reinforced - Shadow, WC 2013 - Supporter
    Preach it, Roobaby, preach!
  • _INTER__INTER_ Join Date: 2009-08-08 Member: 68392Members, NS2 Playtester, Reinforced - Shadow
    edited September 2014
    @UncleCrunch
    While the current hive doesn't prevent stacking, it atleast doesn't encourage it (actually force-even works pretty well now if you play on a server with same people on all the time). Unlike your idea. Another disadvantage of your system is that skill gets completely uncomparable. Meaningless... you wouldn't even know if a skill of 2000 or 20000 is good or bad and if you compare it with someone that got 900 you couldn't deduce anything useful from it.
    Only "advantage" I see is that you wouldn't feel bad loosing a game. Again encouraging bad behaviour...
  • Laosh'RaLaosh'Ra Join Date: 2011-12-09 Member: 137232Members
    hiding the hive skill stat also means hiding the fact that stacking does not pay off.
    it also prevents feedback from players, wether the parameters are right and if there are some weird bugs occuring such as the one posted by MrFangs.
  • UncleCrunchUncleCrunch Mayonnaise land Join Date: 2005-02-16 Member: 41365Members, Reinforced - Onos
    Frankly i never seen so many stacked game than the last 2 weeks. Never. You still win points by stacking. 30-40 points... done. Why do you think it's still continue?

    What happening ?

    Under the skin of a new comer :
    I bought the game as a sale. it was time get 3 or 4 cheap games and select what could interest me.
    Configuration enable this , disable that...
    I don't have the time nor am interested in tutorial (yes this one noticed it)

    I see a organized play but it's rather empty, never mind. So I will connect on server: Few are crowded but i noticed that I can join that +24 slots (up to insane 40 slot) server. Connecting... Connecting... Connecting... Connecting... Connecting... well I'm gonna get a tea while it's loading.

    Ah! I'm in the RR. OK let's do Marine first, i don't know the game but i assume that it's closer to what i know already in marine side. OK i spawned i see people running and i follow them. My FPS is holding a +70FPS, it's OK for now.

    My two teammates are doing something with a structure. Wow! what's that ? everybody shoot at it. I should too... ratatatatata. OK the 2 teammates died but I finished it. I decide to wander a little to explore the map. Somebody is yelling about extractors not finished... I continue my exploration.

    I get killed by the same creature as before. They walk on walls, it's fun.

    I decide to continue playing for a while. I get killed most of the time and wait to re-spawn. The more i play the more i get raped as the enemy apparently gets stronger. I don't know why. In the end they are at the base doors and i see message saying "mister voted... surrender". I barely move inside the base as the alien are killing us in the base.

    I repeated this on marine side and got the same result over a bunch of game. Some other game were won but i died the same way in this games. While i was voting for a balanced game, i didn't see much difference in the teams on each map.

    So i decide to get some help to understand the game. Forums it is. I see words like "stacked", "balance", "competitive", "hive stats". Hmmm stats... let's see. wow !!! some are really strong compare to me. And I see I barely won 2 games over 20. Basically it's loosing game and loosing XP with people yelling in there.

    I'm afraid I'm not gonna stay. It looks like a gay bar filled with people eager to get some fresh meat and I feel I'm gonna get raped every time I go in there.

    Better try the other games I bought. Probably more people, fast paced, and maybe my PC can handle it better.


    And I'm generous. I know ppl that just don't push after ONE game like that.

    tadaaaa!
  • RapGodRapGod Not entirely sure... Join Date: 2013-11-12 Member: 189322Members
    Possible suggestion - set a designated time like 30 seconds or less after a round that keeps people from joining a team... that way people can't immediately stack... and gives time for those loading a map and join the game only to see a stacked round from the get-go. Idk, an idea.

    That aside, I have a question. Does this balance system take into account assists? A majority of the games I play I end up with a k/a/d of something like 8/15/4. That's gotta count for something, right?
  • ZeframZefram Join Date: 2004-05-11 Member: 28611Members, Forum Moderators, Constellation, NS2 Playtester, Squad Five Gold, NS2 Map Tester, WC 2013 - Shadow, Subnautica Playtester, Pistachionauts, Retired Community Developer
    RapGod wrote: »
    Possible suggestion - set a designated time like 30 seconds or less after a round that keeps people from joining a team... that way people can't immediately stack... and gives time for those loading a map and join the game only to see a stacked round from the get-go. Idk, an idea.

    That aside, I have a question. Does this balance system take into account assists? A majority of the games I play I end up with a k/a/d of something like 8/15/4. That's gotta count for something, right?

    As of right now, whether you're 100 kills or with 100 assists, it doesn't matter. The only thing that matters is winning the round.

    Although I believe it's incomplete, I'm told it's the easiest way to figure out skill (put a number on effectiveness in winning).
  • RapGodRapGod Not entirely sure... Join Date: 2013-11-12 Member: 189322Members
    Zefram wrote: »
    RapGod wrote: »
    Possible suggestion - set a designated time like 30 seconds or less after a round that keeps people from joining a team... that way people can't immediately stack... and gives time for those loading a map and join the game only to see a stacked round from the get-go. Idk, an idea.

    That aside, I have a question. Does this balance system take into account assists? A majority of the games I play I end up with a k/a/d of something like 8/15/4. That's gotta count for something, right?

    As of right now, whether you're 100 kills or with 100 assists, it doesn't matter. The only thing that matters is winning the round.

    Although I believe it's incomplete, I'm told it's the easiest way to figure out skill (put a number on effectiveness in winning).

    Yea... there are waaaay too many variables in this game other than matches won. Bah, I guess I'll just continue to kamikaze and get assists.
  • RoobubbaRoobubba Who you gonna call? Join Date: 2003-01-06 Member: 11930Members, Reinforced - Shadow, WC 2013 - Shadow
    It uses only win/loss for several good reasons, probably the main one of which is that this is the goal of the game. The goal of the game is not to get a really high KDR (we've all been that guy on the losing team who couldn't carry the entire team to victory but still destroyed the opponents), and by specifically avoiding rewarding certain aspects of gameplay, it's fairer to reward people who achieve the goal of the game: to destroy the enemy's CCs/hives.
    In the long run, this *should* reduce the instances of people farming rookies - and at the very least it will stop people from getting credited for doing that as was happening before.
  • MoFo1MoFo1 United States Join Date: 2014-07-25 Member: 197612Members
    edited September 2014
    amoral wrote: »
    MoFo1 wrote: »
    Is there any idea of when the skill system might be fixed? I really thought the new v v, v, =, ^, ^ ^, tags would help a lot, but ever since the last reset nearly ever server shows as v v. Very rarely do I see a v, and never ^ or ^ ^, and the only time I ever see = is when I'm the only one in the server.

    you gotta really appreciate where "average" actually falls. when i play i'm out of my league when i'm getting 1:2 kills on one other guy on the team... but this means that i'm still tearing through the rest of the team pretty consistently. it sometimes feels like even if 5 other guys in the game are 1:1 k/d with you, you've still got filler of 10 other guys on the team that make the server a double down arrow.

    also remember, some of the monsters don't play many pubs.

    Even if I were an above average player it shouldn't make every server show as vv though. Hell I can join the KKG captain server when it's full of players like Emo, Billy Talent, Zachy, etc... I can literally be the worst player on the entire server and it will still show as vv.

    If we're going to have a skill indicator in the browser then it needs to be at least somewhat accurate.



  • RapGodRapGod Not entirely sure... Join Date: 2013-11-12 Member: 189322Members
    Roobubba wrote: »
    It uses only win/loss for several good reasons, probably the main one of which is that this is the goal of the game. The goal of the game is not to get a really high KDR (we've all been that guy on the losing team who couldn't carry the entire team to victory but still destroyed the opponents), and by specifically avoiding rewarding certain aspects of gameplay, it's fairer to reward people who achieve the goal of the game: to destroy the enemy's CCs/hives.
    In the long run, this *should* reduce the instances of people farming rookies - and at the very least it will stop people from getting credited for doing that as was happening before.

    Yea, I don't like the idea but I see no alternative. A player can do nothing while he has a person carry the team, he can suddenly get points at the end of the round. Me no likey. I think it can give rookies more points that will make the skill rating on the servers seem weak. But like I said, I have no idea what else to do.
  • MrFangsMrFangs Join Date: 2013-03-27 Member: 184474Members, Reinforced - Shadow
    Ok, I think something needs to be tweaked in how the skill system handles rookie players.

    I already posted one example above where many rookies were present, and the skill change was quite excessive.

    Now I've got another case where I went "teaching commander" for a team with 6 rookies out of the 8 players. As the other team had 4 experienced players, we lost - and then I saw that I lost 50 points from that round. :-S Now I don't care too much about my skill ranking in and by itself, but it shows that something is off. Also, some people *do* care about their skill, and this may discourage them from joining rookie teams, which reinforces stacked teams... one of the things the skill system is explicitly trying to prevent.

    When I play on servers with mostly veteran players, the skill changes seem reasonable though, which makes me think rookie level handling may be the issue here.
Sign In or Register to comment.