Are win/loss stats relevant?
RobustPenguin
Join Date: 2012-08-17 Member: 155719Members
Join Date: 2012-08-17 Member: 155719Members
Hi all,
Been wondering about a few things, recently alot of players have been saying in servers how marines are underpowered and need buffs etc because they lose more frequently than aliens, it is definitely true that the win/loss rate on ns2stats is alien favoured by a bit but I'm trying to figure out if that is really relevant? Looking at the games played (around 350) it seems that there are 25 games which aliens would have lost if we were looking for a perfect 50/50 split. Now, 25 out of 350 is quite small (1/14th). My proposition is that these 25 games are not 'legitimate' games, they are when you have 2 people on a server messing around waiting for it to be filled, the game will still start and it will still be recorded as a game but in these situations it is very easy for aliens to spawn rush the marine base and win, giving rise to these 25 games. Obviously it would be easy to tell if you had the data relating to game length and plotted it with a win or loss then removed any clear outliers (30 seconds victories for example) or could just exclude any games of less than 5v5. Whilst I'm sure that ns2stats probably has this information, I cant work my way around the site to find it.
In general Im also looking to pull a more rigorous statistical approach to various issues to try add in a more concrete statement on balance. If anyone who knows ns2stats could let me know if information like kills with specific weapons, what killed what (eg fade killed shotgunner), building kills, lifespans etc is held and where to get at it I'd be appreciative
Been wondering about a few things, recently alot of players have been saying in servers how marines are underpowered and need buffs etc because they lose more frequently than aliens, it is definitely true that the win/loss rate on ns2stats is alien favoured by a bit but I'm trying to figure out if that is really relevant? Looking at the games played (around 350) it seems that there are 25 games which aliens would have lost if we were looking for a perfect 50/50 split. Now, 25 out of 350 is quite small (1/14th). My proposition is that these 25 games are not 'legitimate' games, they are when you have 2 people on a server messing around waiting for it to be filled, the game will still start and it will still be recorded as a game but in these situations it is very easy for aliens to spawn rush the marine base and win, giving rise to these 25 games. Obviously it would be easy to tell if you had the data relating to game length and plotted it with a win or loss then removed any clear outliers (30 seconds victories for example) or could just exclude any games of less than 5v5. Whilst I'm sure that ns2stats probably has this information, I cant work my way around the site to find it.
In general Im also looking to pull a more rigorous statistical approach to various issues to try add in a more concrete statement on balance. If anyone who knows ns2stats could let me know if information like kills with specific weapons, what killed what (eg fade killed shotgunner), building kills, lifespans etc is held and where to get at it I'd be appreciative
Comments
i talked with zeikko about this because i smelled a bug when a game was not registered. he told me that he actually has exactly this limit for the average number of players if you sum up all played seconds, which is now slightly dropped to take account for the few missing people at the beginning or end of a match.
so yea, it is safe to say that games averagely smaller than 5v5 are not registered. i don't know if there is a minimum game length, though. that might be a bit tricky, because you don't want to encourage people instantly quitting (to force a new round) when they are not happy with their initial matchup for some reason.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->A game with a 50/50 win ratio is balanced. Whether that is something to strive for, on the other hand, is questionable.<!--QuoteEnd--></div><!--QuoteEEnd-->
i don't see your point, why would you not want a balanced game?
I disagree with the statement that just because it has a 50/50 winloss ratio its balanced, things are generally less clear cut than that. Ill use an example of a game I used to play called allegiance, one of the races in that was called the gaataaran, if the game was less than 5v5 they won every time. Any more and they lost, because the community at the time was quite small (the game was intented to be run in very large scale, 20v20 etc) it took alot of time to get servers up and running so looking at their stats the GT had pretty much 50/50 win loss inspite of the fact they were terrible in the actual game. This is what calls me to question the usefulness of looking at this number in exclusion of everything else.
<!--quoteo(post=1978670:date=Sep 16 2012, 10:38 PM:name=Laosh'Ra)--><div class='quotetop'>QUOTE (Laosh'Ra @ Sep 16 2012, 10:38 PM) <a href="index.php?act=findpost&pid=1978670"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->i talked with zeikko about this because i smelled a bug when a game was not registered. he told me that he actually has exactly this limit for the average number of players if you sum up all played seconds, which is now slightly dropped to take account for the few missing people at the beginning or end of a match.
so yea, it is safe to say that games averagely smaller than 5v5 are not registered. i don't know if there is a minimum game length, though. that might be a bit tricky, because you don't want to encourage people instantly quitting (to force a new round) when they are not happy with their initial matchup for some reason.
i don't see your point, why would you not want a balanced game?<!--QuoteEnd--></div><!--QuoteEEnd-->
Potentially, but aliens are far far more likely to f4 out (given they cant just recycle). Is that taken into account in the recording? If its not counted as an alien loss then again we could be seeing reasons why they seem to win so much.
Badly phrased. A game can be 50/50 and hence balanced, but utterly atrocious to actually play. For example, imagine a game of flipping a coin and whoever gets heads is the winner. 50/50 win but really godamn ######. So perfect 50/50 results aren't always something to run towards, at least not while disregarding everything else.
<!--quoteo(post=1978672:date=Sep 16 2012, 11:40 PM:name=RobustPenguin)--><div class='quotetop'>QUOTE (RobustPenguin @ Sep 16 2012, 11:40 PM) <a href="index.php?act=findpost&pid=1978672"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->I disagree with the statement that just because it has a 50/50 winloss ratio its balanced, things are generally less clear cut than that. Ill use an example of a game I used to play called allegiance, one of the races in that was called the gaataaran, if the game was less than 5v5 they won every time. Any more and they lost, because the community at the time was quite small (the game was intented to be run in very large scale, 20v20 etc) it took alot of time to get servers up and running so looking at their stats the GT had pretty much 50/50 win loss inspite of the fact they were terrible in the actual game. This is what calls me to question the usefulness of looking at this number in exclusion of everything else.<!--QuoteEnd--></div><!--QuoteEEnd-->
I'm not talking about anecdotal evidence showing that, after 13 games were played, 6 were won, 6 were lost, and one was called off due to rain. I mean concrete objective universally solid evidence. Like, 1 0000000000000000000000 games were played, with every assortment of every number of players and ping and frame rate and every other external factor. Not that that can actually be obtained, but that means that the game is balanced, irrespective of everything else.
I'm not talking about anecdotal evidence showing that, after 13 games were played, 6 were won, 6 were lost, and one was called off due to rain. I mean concrete objective universally solid evidence. Like, 1 0000000000000000000000 games were played, with every assortment of every number of players and ping and frame rate and every other external factor. Not that that can actually be obtained, but that means that the game is balanced, irrespective of everything else.<!--QuoteEnd--></div><!--QuoteEEnd-->
This is true, and the current amount of games played this patch is well beyond a reasonable threashold for the central limit theorem (the idea that 1000000000 datapoints is better than 10 effectively) but my point is still that, if 1/14 games played end with a minuet 1 skulk rush in 4v4 then you will still have that systemic error regardless of how many trials you do.
you have a point there, but it is rather minor: as soon as aliens start to f4, the game is usually over rather soon. nd the players who left will still have contributed to the overall player-seconds (e.g. 540 seconds in a 600 second match when leaving a minute before the end), these will not be discarded.
@Imbalanxd
so is this about certain techs not being viable (e.g. xenocide) or too important? i can relate to this regarding the competitive matches, but i don't think it is a bad thing to have a 50-50 balance in general. it means whenever you tweak something, the overall balance is more likely to turn out decent.
@Imbalanxd
so is this about certain techs not being viable (e.g. xenocide) or too important? i can relate to this regarding the competitive matches, but i don't think it is a bad thing to have a 50-50 balance in general. it means whenever you tweak something, the overall balance is more likely to turn out decent.<!--QuoteEnd--></div><!--QuoteEEnd-->
Its very difficult to balance the tech paths I feel. Currently I pretty much always go shift->shade because I feel cara is just such a bad upgrade I would rather have silence for the early game and actually be able to claim a 3rd rather than go cara and get mauled
If 1/14 games end that way then they need to be taken into account when determining balance. Its like balancing this game around 6v6 when >80% of the games are closer to 9v9, its pointless. The most prevalent games are the ones that need to be prioritised, and if you just take a bunch of games, then statistically the most common games will be the most represented, and you will get accurate and useful figures.
I remember allegiance, does anyone still play it? Cause it was awesome!
When I left 3 years ago it was still pretty big, MS actually released the code to the community a fair few years back now and they have a pretty large scene coding/supporting it with their own server system, a few new races and general code upgrades. They have a noobie training program and such, obviously during the day its not especially busy (only 9 in game right now) but in the evenings I assume it picks up. Freeallegiance.org, free download and all that stuff so its worth checking out again.
To me it just seems like they've lost track of what is good gameplay and went for brining this arbitrary 'balance figure' closer by throwing the nerfbat around in all directions.
Something I find frustrating is how naive people are with statistics. For example, a 50/50 win/loss ratio does not mean the teams are balanced and they win equally often. It simply tells you from this sample marines and aliens won 50% of their games. I make my living out of statistics and the most important thing to be aware of is that statistics tells you more about what you don't know than it does about what you do.
There are so many variables (many human behavior) that invalidate the statics too. For example I, and many of my friends, will frequently switch teams if one losses too frequently ensuring that the win/loss ratio is stable no matter what balance is like.
True story...
so yea, it is safe to say that games averagely smaller than 5v5 are not registered. i don't know if there is a minimum game length, though. that might be a bit tricky, because you don't want to encourage people instantly quitting (to force a new round) when they are not happy with their initial matchup for some reason.<!--QuoteEnd--></div><!--QuoteEEnd-->
This is right. The round needs to have an average of 10 players playing for it to be tracked into the stats. This was not the case in the first couple of weeks of the lifetime of ns2stats, so bear this in mind when comparing win/loss ratio stats between builds.
There is no requirement for length of the round though.
<!--quoteo(post=1979012:date=Sep 17 2012, 05:51 AM:name=Argathor)--><div class='quotetop'>QUOTE (Argathor @ Sep 17 2012, 05:51 AM) <a href="index.php?act=findpost&pid=1979012"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->The tiny sample pool and large array of variables make the statistics currently gathered largely worthless.
Something I find frustrating is how naive people are with statistics. For example, a 50/50 win/loss ratio does not mean the teams are balanced and they win equally often. It simply tells you from this sample marines and aliens won 50% of their games. I make my living out of statistics and the most important thing to be aware of is that statistics tells you more about what you don't know than it does about what you do.
There are so many variables (many human behavior) that invalidate the statics too. For example I, and many of my friends, will frequently switch teams if one losses too frequently ensuring that the win/loss ratio is stable no matter what balance is like.<!--QuoteEnd--></div><!--QuoteEEnd-->
This is largely why you need to control for those variables (I state four in <a href="http://www.unknownworlds.com/forums/index.php?showtopic=114175&hl=" target="_blank">my thread</a>; map, playercount, match length, and team skill). The collected stats (especially the ns2stats.org ones) have gotten pretty good imo, providing enough data for someone with the time and skills to analyze them properly. Also, it might just be the engineer in me, but I don't see the point of using statistics if it doesn't at least help you learn more about something. The whole statistician's precise but functionally useless language has long frustrated me.
I dont run mods on my servers after dealing with problems with other popular server mods for so many months. A new patch would roll out and then some oddities would occur.
Its vanilla for me for the time being..
There is a reason behind the language. As humans we are genetically created pattern finding machines, which makes us oddly incapable of impartially analysing statistics. Naturally we see patterns and see meaning in statistics, when sadly they actually tell us far more about what we do not (or cannot) know.
Unless NS2 becomes incredibly popular and we get a much larger sample size, it will be difficult to draw any conslusions at all. As it stands, I could write 10 pages on what the current statistics do not tell us and about half a page on what they do.
P.S. '<b>Fooled by Randomness</b> by Nassim Taleb' is a great read.
P.P.S. I make a living out of predicting outliers in the world's economies and financial markets, which is much easier than predicting common occurrences!
<i>edit: I really enjoyed reading your thread but Wilson put a very important point across, in essence no matter how you measure it you might be completely wasting your time as there may be factors effecting balance (the type of balance game developers strive for) that you can never quantify with statistics.</i>
Unless NS2 becomes incredibly popular and we get a much larger sample size, it will be difficult to draw any conslusions at all. As it stands, I could write 10 pages on what the current statistics do not tell us and about half a page on what they do.
P.S. '<b>Fooled by Randomness</b> by Nassim Taleb' is a great read.
P.P.S. I make a living out of predicting outliers in the world's economies and financial markets, which is much easier than predicting common occurrences!
<i>edit: I really enjoyed reading your thread but Wilson put a very important point across, in essence no matter how you measure it you might be completely wasting your time as there may be factors effecting balance (the type of balance game developers strive for) that you can never quantify with statistics.</i><!--QuoteEnd--></div><!--QuoteEEnd-->
I do understand the reasoning behind the language, but find it frustrating how it makes its functionally useless towards solving real problems. Determining how balanced NS2 is in any given build is a real problem and a basic statistical analysis of server-collected data, however flawed, seems a better approach than player anecdotes or even personal experience, as shown by the multitude of threads on this forum who disagree on which side is op.
I've also read a bit of Nassim Taleb (mostly Black Swan and a few articles he's written) and I do like his stuff. Its a good reminder of the limits of our predictive and analytical abilities, but I don't think that should stop us from applying statistics to non-ideal situations. We just have to remember to be careful and try to get further verification or refutation from other sources, if possible.
There is no requirement for length of the round though.<!--QuoteEnd--></div><!--QuoteEEnd-->
Interesting, is the round length variable collected at the same time as the win/loss ratio? I'd like to see how things are when we cut out the 2 minuet alien victories. The information does all seem to be on ns2stats if you look for it, I might do a few bits and bobs, try model a few things to remember my basic stats crap before uni starts again
<!--quoteo(post=1979281:date=Sep 18 2012, 12:49 AM:name=ScardyBob)--><div class='quotetop'>QUOTE (ScardyBob @ Sep 18 2012, 12:49 AM) <a href="index.php?act=findpost&pid=1979281"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->I do understand the reasoning behind the language, but find it frustrating how it makes its functionally useless towards solving real problems. Determining how balanced NS2 is in any given build is a real problem and a basic statistical analysis of server-collected data, however flawed, seems a better approach than player anecdotes or even personal experience, as shown by the multitude of threads on this forum who disagree on which side is op.
I've also read a bit of Nassim Taleb (mostly Black Swan and a few articles he's written) and I do like his stuff. Its a good reminder of the limits of our predictive and analytical abilities, but I don't think that should stop us from applying statistics to non-ideal situations. We just have to remember to be careful and try to get further verification or refutation from other sources, if possible.<!--QuoteEnd--></div><!--QuoteEEnd-->
Its also worth noting that what one counts as a 'small' sample size depends, the central limit theorem really starts applying beyond around 10 data points. Indeed, with nearly 400 games played and multiple data points within those games (20 players per game) we dont exactly lack data. We arent going to be 5 sigma levels of precision but do we need to be? Provided we take a small measure of care when looking at the data (eg spotting those games which end after 30 seconds (in my view a systemic problem with the powernode system being so easily overpowered early on)) then I believe we can draw credible conclusions from looking at the data in a more rigorous approach.
Statistics can be a tool to help you balance, but it cannot be a tool to tell you if the game is balanced.
Seeing 80:20 win loss on summit would be a clear indication of imbalance, the current statistics are good for showing obvious imbalances such as this
trying to get precision balance (ie. the all important 50:50 ) using statistics with such a poor level of detail and low sample size ... its like trying to do surgery using a shotgun.
So to answer the op : Win/loss stats are only very slightly relevant and should be taken with a grain of salt.
I've also read a bit of Nassim Taleb (mostly Black Swan and a few articles he's written) and I do like his stuff. Its a good reminder of the limits of our predictive and analytical abilities, but I don't think that should stop us from applying statistics to non-ideal situations. We just have to remember to be careful and try to get further verification or refutation from other sources, if possible.<!--QuoteEnd--></div><!--QuoteEEnd-->
There is a good reason for that, it is functionally useless for solving real problems. A large number of the world's problems come from the human inability to understand and interpret statistics correctly, yet most people still think they are special and can see patterns, gaining useful insight. Sadly they are not.
The problem is a 50/50 win/loss ratio is not the end goal and it is an incredible mistake to consider it so. It is, relatively speaking, reasonably easy to obtain a 50/50 ratio. It is incredibly difficult to get a 50/50 ratio with engaging, fun, dynamic, varied and complex gameplay. These is a challenge that statistics are not equiped to deal with <i>at all</i>.
I personally think you would get a lot more productive information regarding NS2 balance by using a combination of things like networking theory, game theory and general psychology. Statistics just aren't equipped to deal with the problem at hand.
Absolutely incorrect.
in a game like starcraft that is mostly correct
But this is an rts AND an fps, you have to take player skill into account, if you're saying that because the win/losses are equal you are extremely blinded. people who are extremely good can carry the extremely underpowered team simply with skill.
This is balance!
Win ratio balance is only one of several balance goals that NS2 should strive for. Another is ensuring that different tech trees or path are viable. Additionally, making sure all starting map tech nodes are viable is another worthy balance goal.
*
One of the problem is that fixing as a goal to get 50% winrate is a very ill defined problem. Let me explain.
The winrate depends on a large number of parameters : skulk health, res influx, lmg damage, tech cost, etc.
Taken alone almost every one of this parameters allows to get a 50% winrate. The reason why this is true is the following. Take the lmg damage as an example. Put it to zero, the marine winrate drop to zero. Put it to infinity (one shot everything), the marine winrate goes to 100%. By a continuity assumption there is a lmg damage value where the winrate is 50% (it cannot go from 0 to 100 without passing by 50).
The same is true for almost every other parameter. What this means is that saying "parameter X is responsible for the non-50% winrate" is meaningless since it is trivially true for any parameter.
I don't go into combination of parameters (e.g. for two parameters there is probably a curve of 50% winrate, for three a surface, etc.) but the problem become even more degenerated.
So, what we need right know is more, clearly stated, balance criteria, in addition to the 50% winrate.
*
This is what I wanted to say, the 50% percent win-rate goal is a narrow view of balance, and it's actually an easy problem, because there is lots of solutions to it. The hard balance problem is to get most of the tech tree to be useful, to maximize the number of valid build orders. Starcraft bw reached that by having almost all the units being used at the pro level (even the scout), and different viable tech paths (bio vs mech in TvZ).
And to be honest different tech tree viability is one of the biggest issues in NS 2 today. There simply isn't a whole lot of variety in (viable) builds for both the alien and the marine commander. For aliens it's usually a choice between shift or fast second hive, for marines it's almost always a choice for quick obs and phase tech. (Since TF isn't a viable choice and you need beacon and phase gates to maintain map control) Alien gameplay revolves entirely around acquiring the fade.
NS 1 gameplay by comparison was so much more diverse. (Think CC relocs, TF starts, etc)
in a game like starcraft that is mostly correct
But this is an rts AND an fps, you have to take player skill into account, if you're saying that because the win/losses are equal you are extremely blinded. people who are extremely good can carry the extremely underpowered team simply with skill.<!--QuoteEnd--></div><!--QuoteEEnd-->
Are you talking objective balance statistics, or subjective personal experience? Individual skill means nothing with regards to balance because when analyzing balance you typically use a very large data set. If the average skill level of a player is some arbitrary number 7, then over X number of games, X sufficiently large, the skill level of any team in the dataset is 7 multiplied by the number of players. This is of course unless better players gravitate towards one team over another for whatever reason, but even if this is the case, if it happens often enough then it must be included in any balance analysis.
<!--quoteo(post=1980702:date=Sep 20 2012, 02:01 PM:name=Yuuki)--><div class='quotetop'>QUOTE (Yuuki @ Sep 20 2012, 02:01 PM) <a href="index.php?act=findpost&pid=1980702"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->One of the problem is that fixing as a goal to get 50% winrate is a very ill defined problem. Let me explain.
The winrate depends on a large number of parameters : skulk health, res influx, lmg damage, tech cost, etc.
Taken alone almost every one of this parameters allows to get a 50% winrate. The reason why this is true is the following. Take the lmg damage as an example. Put it to zero, the marine winrate drop to zero. Put it to infinity (one shot everything), the marine winrate goes to 100%. By a continuity assumption there is a lmg damage value where the winrate is 50% (it cannot go from 0 to 100 without passing by 50).
The same is true for almost every other parameter. What this means is that saying "parameter X is responsible for the non-50% winrate" is meaningless since it is trivially true for any parameter.
I don't go into combination of parameters (e.g. for two parameters there is probably a curve of 50% winrate, for three a surface, etc.) but the problem become even more degenerated.
So, what we need right know is more, clearly stated, balance criteria, in addition to the 50% winrate.<!--QuoteEnd--></div><!--QuoteEEnd-->
This is kind of what I was getting at when I said 50/50 balance wasn't always something to be pursued at all costs. Typically the fewer elements around which you balance, the less fun a game becomes, whether or not it becomes more "balanced". If LMG damage was simply increased to the point where 50/50 balance was acquired, I think a lot of skulks would be having a pretty boring time.