How are you going to beta test
locallyunscene
Feeder of Trolls Join Date: 2002-12-25 Member: 11528Members, Constellation
Feeder of Trolls Join Date: 2002-12-25 Member: 11528Members, Constellation
<div class="IPBDescription">and what will you be looking for?</div>I read good article today <a href="http://www.tentonhammer.com/node/66885" target="_blank">about past and present beta testing</a>[tentonhammer.com]. It's pretty long, but asks some good questions I haven't seen articulated. Mainly:
<ul><li>Should a game be "Feature Complete" at beta?</li><li>Should a game be fun for the beta?</li><li>How many and how big can the bugs be before the beta is "broken"?</li><li>Or should the beta testers be expected to shape the game to be fun?</li><li>Should the beta be just a tool for generating hype ?</li></ul>
It's mostly about MMO betas, but an interesting read regardless.
<ul><li>Should a game be "Feature Complete" at beta?</li><li>Should a game be fun for the beta?</li><li>How many and how big can the bugs be before the beta is "broken"?</li><li>Or should the beta testers be expected to shape the game to be fun?</li><li>Should the beta be just a tool for generating hype ?</li></ul>
It's mostly about MMO betas, but an interesting read regardless.
This discussion has been closed.
Comments
I think it would make more sense to gather a wide range of testers.
<ul><li>Should a game be "Feature Complete" at beta?</li><li>Should a game be fun for the beta?</li><li>How many and how big can the bugs be before the beta is "broken"?</li><li>Or should the beta testers be expected to shape the game to be fun?</li><li>Should the beta be just a tool for generating hype ?</li></ul>
It's mostly about MMO betas, but an interesting read regardless.<!--QuoteEnd--></div><!--QuoteEEnd-->
<ul><li><b>Should a game be "Feature Complete" at beta?</b>
Generally, yes. That's supposedly the idea of the Beta stage. On the other hand, if there's something glaringly obvious at Beta that could be remedied by a new feature, it would be sensible to remedy it.</li><li><b>Should a game be fun for the beta?</b>
Generally, yes. But this highly depends on if it's an internal and early Beta or if it's a late and public beta. Internal testers should be of a the type of calibre that they can deal with a game not being fun before submitting bugs and feedback. For public testers, you have to expect they will speak their mind, submit erroneus bugs, submit poor bugs reports and generally not do any of this if the game isn't fun enough for them to bother complaining about it. If the game isn't fun, the public won't bother complaining. If you are getting a lot of complaints, at least you know people care enough about the game to complain.</li><li><b>How many and how big can the bugs be before the beta is "broken"?</b>
This is very dependant on the testers. In a way, if they complain the game is broken because of balancing issues it's a far better result than if they complan it's broken because of mechanics issues. In the former, at least you know it's working properly and you are onto the balancing phase of testing. Generally a high priority A class bug means the Beta is broken, but since the game is still in test, it's not a major problem, since that's what the test is for. I've found A class bugs in submission (a.k.a. 'release candidate') builds. As long as it's still in test and hasn't gone Gold, it's a good thing you found it at this stage of dev, even if it seems late. In those cases, the console manufacturers (Sony, Microsoft, Nintendo) were informed that these bugs had been found and fixes were being implemented - these were known issues that could be patched in post-retail or fixed before going retail.</li><li><b>Or should the beta testers be expected to shape the game to be fun?</b>
Of course. More specifically, as well as finding compatibility bugs, you should expect the Beta testers to shape the game to be <i>more</i> fun.</li><li><b>Should the beta be just a tool for generating hype ?</b>
No. It's primary function is functionality testing, compatibility testing, stress testing and balance testing... in that order. If it also generates hype that's a bonus, but you really need to have a good idea of what stage your Beta is before hoping it will generate hype; the opposite could be true: you could end up generating negative hype.</li></ul>
On the pro versus amateur tester debate: both are very important in testing. People who know the game intensively will give you better balance feedback because they won't overlook counterpoints as much. But on the other hand, those same testers will frequently overlook areas of the game that make it hard for beginners to learn the game, such as poor UI, poor tutorials, poor controls, beginner balance, etc.. The best practise is to bring in a bunch of 'virgin' testers prior to release to pick up on usability issues the testers who have been on a project for many months have gotten used to as the game has been forged over time.
LoTRO on the other had was... different, I absolutely loved playing the game in the beta to the point where I even pre-ordered a copy. But for some reason I never actually played in the retail release of LoTRO, I don't know whether or not I had gotten enough out of it playing up to level 14 in the beta or if I didn't want to P2P. Fortunately I didn't get the $200 limited life subscription.
I guess my point is that there are betas where you'll want to pay for the retail copy and keep playing it, and there are those where you wont. NS2 will definitely be the former for me though.
I think the beta should be more of a dry run for pre-retail more so than another round of play testing which should be done in the alpha stages.
</ramble>
edit: oh yeah, early beta testing for consties too.
The PT Crew should players that are pretty knowlegable and skilled to the point where any ignorance or stupid play can be ruled out of any balance decisions (Not to say that players don't make mistakes; but you certainly do not want the players who cannot kill a single skulk with a full LMG clip but happen to know a "ton" about NS).
Whether these players are competitive or not really is not all that important unless either style of player is not producing a quality tester.
There are a few things I believe a quality PT should possess.
1. Dedication to the playtesting schedule: A fairly obvious point, this is not a gamespy beta, so that means pubbing without feedback really is of much value to the devs. Playtesters should be expected to make most playtests.
2. Accepting that the game is not complete: I don't think that this will be challenge in this community, but again it is important when testing to realize that the game is not complete and there will probably be bugs (hence the testing).
3. Enough skill to be able to "compete" against those that maybe considered significantly better. An example of this would be to not allow Mustang to go 121 and 0 during playtests for balance purposes. Having a bunch of players that do not possess the FPS skills and tactics to at least put up a good fight really does not have much value.
4. Being able to think outside the box. What I mean with this is trying to think up of the 1.04 "sYn" rush before the game gets released to the public. PTs would be of much value if they can find those exploitative strategies (especially now on the alien side based on how they tech) early in the betas so the balance can be tweaked for later versions so that the game can be fair for both sides. This is probably the most crucial quality of all, especially since the game is geared for retail release and not being a free mod.
5. The ability to communicate well with team members and developers when relaying suggestions/input. It would be best if the playtester could have some sort evidence (i.e. Video/Demo; Screenshots) regarding a suggestion or some type of input. A bit of a pain in the rear for the PT, but providing a visual stimulus with feedback in my opinion is much better than providing just the feedback.
That is all I can think of now, if anyone has more qualities that should be added to the list, please feel free to.
I think you could release the game crippled (ie if some features were not ready) but nothng that would effect gameplay. For example you could release it without all modes of play, or with reduced amounts of visuals or maps, but you wouldn't want to release it with say a limited amount of guns available, as you aren't really testing the gameplay. I think for a lot of people the first impression will count a lot, and while it pains me to say it, I think a more polished beta would be more beneficial than a really buggy one. Due to deadlines a lot of software tends to get released before it is ready, that's understandable but bugs that cripple gameplay would be really bad.
I agree witht he guys who have already said in this thread that PTs with creativity are important to discover really exploitive strategies, however I think there is a place for a range of playtesters with lots of different skill/experience levels, as the game needs to be tested for noobs as well.
Your methodology is geared towards making a game that only performs well in a test-tube environment, where nobody is new to the game and the skill range is very narrow and very high. If people designed multiplayer games to this end only a small amount of people would enjoy them. The sales would be poor and the community would not expand as quickly because competitive communities are by nature insular and restrictive. The game would get a reputation for being 'too hard' because it hadn't been designed with beginners in mind, and because most players would be of a far higher skill level than them.
[Edit] Maybe your list of qualities is not supposed to be in order of priority, but it's curious you should think of 'communication' last.
With your set of preferences you'd get a bunch of good destructive testers who are able to accept the game is not in a stage of completion, who would be of a high skill level, but bottom on your list is communication. So, whatever valid input the tester may have to give, however hard they are able to push the game, the least important thing for you is that they are able to communicate issues succinctly, clearly and accurately. In other words, you risk getting a bunch of "shotgun is broken, imba pos" testers. From experience it's a lot harder to improve people's level of communication than to teach them how to be good testers. A good lead will focus testers on the right areas of the game and ensure good coverage, but it's very time consuming to be teaching testers how to communicate properly if they lack that ability. Communication skills are paramount in testing.
The other thing about your priorities is you are focusing on an area of testing that is usually very easy to fix post-release: balance. Since UWE is making a whole new engine, I wouldn't be surprised to see a lot of compatibility issues emanating from the Beta, such as crashes because of different types of sound and graphics cards, even conflictions with other software. You don't need to be a proskill gamer to find those sorts of issues, you just need to have a more marginal PC spec, and depending on how many people a compatibility issue affects, it could be far more devastating on reception of the game post-release than a unit being OP.
Pushing the game and its balance to their limits is important, but it's not the sole focus of a multiplayer Beta. The main goal of a Beta is to open it up to a wider audience and increase coverage to supplement what a small test team cannot achieve on its own in a short timeframe and with limited resources.
I would re-order your set of preferences to look more like this:
1. Good communication.
2. Previous experience.
3. Dedication and passion for the game.
4. A flair for destructive testing.
Here you know that whatever issue is found you have someone who can communicate it properly. Previous experience isn't a neccessity, but it would cover things like testing unready games, which can often be frustrating, it also means they have been introduced to the world of testing (or maybe they have dev experience), and will have seen the sort of issues that can be found. Dedication is very important, if a crash is found it has to be reported, but the temptation for some people is to get back into the game as soon as possible and keep having fun. Finally, destructive testing is good, but the majority of bugs are not found through destructive testing, I'd say destructive testing (i.e. finding exploits, going bughunting, or 'breaking' the game) will only account for roughly half the bugs in a game. It's a very valuable ability, but it shouldn't be the backbone of a test cycle. The other half (maybe more) are just stumbled across while playing the game normally. You don't have to be a pro gamer to find this 50%, but you do need to be able to report it informatively, without leaving ambiguity or being misleading. That's why communication is top for me.
I disagree. Firewater isn't asking for a team full of CAL champions. He was simply saying that the people play testing the game should be able to put up some kind of resistance towards the enemy and contribute something to their team once in game.
If I remember correctly, the sYn rush Firewater is referring to is as follows: Spawn in, Make IP's, Attack Hive with LMGs, resupply as necessary. It took 5 people a surprisingly low amount of clips to destroy a hive. How was this, probably the most basic strategy in the game, not discovered before release? I chalk it up to the playtester's either not having the proper mindset (ie: win as soon as possible) and/or not being in that average/above average skill level where most of these strategies are developed. For beta <i>testing</i> I agree with Firewater completely. In my opinion, beta is a time most of the bugs have been fixed and it's time to work on balancing the game. Obviously, more in depth exploits and random bugs will occur, but that isn't the focus. Likewise, unless the development team is very poor, the UI should be pretty well established aside from the occasional tweak here and there. If a team had to redo their entire UI due to tester feedback, I'd doubt their vision and ability to make games.
You make a great point about having virgin gamers, both to video games and the NS IP in general, coming in and giving their opinions. However, I would consider that an addendum to what beta testing actually is and I doubt you would need a significant amount of such virgin testers.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->The other thing about your priorities is you are focusing on an area of testing that is usually very easy to fix post-release: balance.<!--QuoteEnd--></div><!--QuoteEEnd-->
Balance is also one of the major reasons most people stop playing a game. I could list nearly endless examples of balance issues that killed off games. I hate when I'm playing a game and I think "man how did <i>this</i> make it in game?" Furthermore, while some crashes may make it past beta, if the game is fun to play then players generally suffer through them knowing a fix is soon on the way. If the game has a glarring imbalance that ruins game play, then players generally stop playing. The bottom line: if it's so easy to fix, it shouldn't have made it to release anyway.
The idea of a beta is that it should be fairly close to feature complete and has had several iterations concerning balance and checking for basic bugs. Betas are typically aimed at stress tests and large-scale balance. Some of the bigger companies (Blizzard) do all their balance in-house. Valve used the TF2 beta for some balance and server stress testing. Indie groups foolishly release 'betas' that are little more than 70% of the content with the promise that they'll fix it "eventually" and this generally kills a huge chunk of the player base.
A release should be as close to perfect as the team can get it. If done correctly, the game should survive for at least a month before major issues arise that demand a patch. Of course, nothing will be perfect. StarCraft took years to get the stable balance, WarCraft3 is on patch 1.2+ for Frozen Throne. And good golly the number of TF2 and WoW patches. However, the devs should NEVER publish a game, then the next week release a patch that fixes holes and exploits. This implies they knew there were still issues and were working on fixing them. Or that they left really obvious and simple stupid errors, i.e. they didn't stress test enough in the beta. This, once again, kills the release playerbase and causes you to lose the chance to take advantage of the novelty high (see the Battlefield series).
We expect a complete and polished package on a "release" of a game. That is what they mean by making it *perfect*. Of course, they can't account for everything, so the sign of a good dev team is eventual patches and added tweaks, but a release is something the devs should be able to proudly say we did our best to give you the best. That is the hallmark of an excellent dev team.
Granted, this runs against things like release/publicity timing and money.....
The idea of (intentionally?) not fixing balance issues, or saying 'we'll deal with that later,' because 'people love waiting for updates' somehow doesn't make any sense to me.
I disagree. In my experience people love small content updates, but usually hate major gameplay changes.
The fact that NS2 is a commercial game also means, that people won't tolerate a large number of bugs.
Therefore, the game should be tested and balanced at release.
I doubt that UWE has enough resources for intensive professional testing at a large scale.
This means that NS2 playtesting should be more than just a marketing gag.
But they are NOT making real game changing modifications.
NS has seen a long list of amazing version changes that upended the game and always for the better; but they simply cannot do this with a retail release. It has to be pretty darn close to perfect when released. They can modify and maybe do some smaller feature changes and additions, but they can't just suddenly change the Fade blink. Or replace the lerk spike. Or put PGs back in. They couldn't even add the 5 extra marine armor (if you don't understand the importance of that 5 armor, feel free to look it up; it's a brilliant design lesson).
This is why I feel NS should have simply been ported first, and THEN had NS2 work started on. But we're on this path and we gotta support the devs however we can.
There are a ridiculous amount of fundamental changes happening to NS, and frankly I think a good chunk of it will end up being axed. But for all the changes that will work, as well as those that will be attempted, we've got to figure some way out of getting the best, and the worst, of the NS community's faithful players in to test the crap out of it.
The community is here for the team... we just need to see if they'll use us.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->Your methodology is geared towards making a game that only performs well in a test-tube environment, where nobody is new to the game and the skill range is very narrow and very high. If people designed multiplayer games to this end only a small amount of people would enjoy them. The sales would be poor and the community would not expand as quickly because competitive communities are by nature insular and restrictive. The game would get a reputation for being 'too hard' because it hadn't been designed with beginners in mind, and because most players would be of a far higher skill level than them.<!--QuoteEnd--></div><!--QuoteEEnd-->
I disagree entirely, what the developers are trying to do is balance the game for those who can exhibit some sort of competence in it. Badies will stay badies unless they learn how to play better. You can't balance the game around the badies, but that does not mean they should be ignored. However, for the playtests, I would make sure that developers have a minimum level of competency for the players. If they wanted to open it up to the consties where the only qualification is to make a donation, that would be fine. However, I would fine tune the game balance first before doing that. People purchasing a retail game that is not balanced are not as forgiving as those who downloaded a free mod (see ET:QW and W:ET).
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->[Edit] Maybe your list of qualities is not supposed to be in order of priority, but it's curious you should think of 'communication' last.<!--QuoteEnd--></div><!--QuoteEEnd-->
My list is in no particular order however I believe that communication is very important. Without the rest though it does not matter how well one can communicate.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->The other thing about your priorities is you are focusing on an area of testing that is usually very easy to fix post-release: balance. Since UWE is making a whole new engine, I wouldn't be surprised to see a lot of compatibility issues emanating from the Beta, such as crashes because of different types of sound and graphics cards, even conflictions with other software. You don't need to be a proskill gamer to find those sorts of issues, you just need to have a more marginal PC spec, and depending on how many people a compatibility issue affects, it could be far more devastating on reception of the game post-release than a unit being OP.<!--QuoteEnd--></div><!--QuoteEEnd-->
Hence why I believe the game should be "feature complete" that should be playable with a few crash exceptions. Balance is not as easy as it looks to fix. In order to achieve balance, you need to take two relatively skilled teams and put them against each other lots of time to see how well the teams work. I would define balance on a macro scale that the marines and aliens win about %50 of the time with a +-5% margin of error (some may go up to 10%). On the player level I would define balance that the players have enough options to achieve victory provided they have the access to those options at a reasonable time under reasonable circumstances.
If I remember correctly, the sYn rush Firewater is referring to is as follows: Spawn in, Make IP's, Attack Hive with LMGs, resupply as necessary. <b>It took 5 people a surprisingly low amount of clips to destroy a hive [...] probably the most basic strategy in the game</b> [...] I chalk it up to the playtester's either not having the proper mindset (ie: win as soon as possible) and/or not being in that average/above average skill level where most of these strategies are developed.<!--QuoteEnd--></div><!--QuoteEEnd-->It's the mentality that was the problem. You do not need skill to find that bug, please see the bits of your own comments I have emboldened. If a Commander asked a bunch of people to do this in a game, your summary suggests it is so basic it could be achieved on a pub server by average players playing against average players. In this case it only takes a QA Lead to create a test case that specifically targets this area of the game. I am saying that if you have a good QA Lead, to remedy the biggest issues with the balance you do not require skilled testers, only ones who can listen and follow instructions for test. The right test cases will draw out bugs such as the one you described.
I agree some high-skilled players are definitely needed to test the balance to the limit, but I wouldn't say any more than 16 are required. Split them into two teams of 8 (playing 6v6 and some 7v7 or 8v8 when scheduling permits) and have them play PCWs exclusively and eventually the most dominant strategies and loopholes will out themselves.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->For beta <i>testing</i> I agree with Firewater completely. In my opinion, beta is a time most of the bugs have been fixed and it's time to work on balancing the game. Obviously, more in depth exploits and random bugs will occur, but that isn't the focus. [...]
You make a great point about having virgin gamers, both to video games and the NS IP in general, coming in and giving their opinions. However, I would consider that an addendum to what beta testing actually is and I doubt you would need a significant amount of such virgin testers.<!--QuoteEnd--></div><!--QuoteEEnd-->You are talking about public beta testing. At this stage the game should have had most of the bugs <i>found</i> (you will never have even the majority of them <i>fixed</i> by this stage, but at the same time all should have been <i>actioned</i>). I.e. all major bugs have been fixed or scheduled fixes and open minor bugs have been waived for release or deprioritised -but not fixed- as necessary.
I see two stages of Beta for a small studio like this. The first would be internal testing, which would get the game into a shippable state. You can still use volunteers for this, although they would need a fulltime QA Lead if it's to be tested properly. The second would the Public Beta. The public beta is the stage you are describing, where most (major) bugs have been identified and fixed or are awaiting fixes that will go into the release. Here you choose one or two of your stronger levels and open it up to a wider audience. I would suggest that for such a small team it's best to keep it as a closed public beta, or the sheer volume of bugs (especially duplicates and NMIs) would be too much to deal with and would waste a lot of time.
Your focus here is, as I said above, functionality testing > compatibility testing > stress testing > balance testing.
Functionality, because there could be some loopholes that your test team hasn't found, especially if you have no dedicated internal test team of professional calibre. There would be the sort of loopholes that stop the game working or create massive exploits. For example, the bug below is not a balance issue, it is a functionality issue. This is why functionality is still the main focus. (Also, going back to your point about your preference for 'skilled' players finding bugs that break the balance: any tester -not just skilled players- could stumble across this bug, but if they can't describe it clearly the first time they report it, it's going to take longer to fix)
<center><object width="450" height="356"><param name="movie" value="http://www.youtube.com/v/Zwe9oC66caQ"></param><embed src="http://www.youtube.com/v/Zwe9oC66caQ" type="application/x-shockwave-flash" width="450" height="356"></embed></object></center>
Compatibility falls under functionality, but it's more focused because it's impossible for a small indie team without major publisher backing to get together a wide range of PC hardware and software to test that the game/engine works in most scenarios. A Public Beta gives you a much better chance at catching these issues, which will frequently entail crashes and in some cases zero game functionality. In terms of the above categories, functionality is more important that compatibility because functionality has the ability to affect all players (that get the game to work). Compatibility on the other hand can have a massive negative impact but is usually restricted to a small percentage of users.
Stress testing is obvious: a small studio without any dedicated testers cannot do reliable stress testing in-house, and they probably can't afford to do it out of house unless they do some sort of Beta.
Balance comes last because it's usually easy to fix, easy to spot (given the right test scenarios), and because with internal testers under a good QA Lead, you can probably get a good level of general balance before a Public Beta depending on the complexity of the game.
You are making the massive assumption that this game is going to be free of major functionality issues when it reaches Public Beta. There are so many things to indicate this won't be the case: an entirely new engine, a complete lack of a dedicated QA team, a very small dev team, a very strict dev timeline. Even much bigger companies struggle with having fully functional public betas. Relic, for example, has big publisher backing and resources, multiple teams of first party and second party testers, but most of the bugs being reported in the DoW2 Beta were compatibility issues that caused graphical/audio corruptions, caused the game to crash frequently or even not boot at all. Those issues are way more serious than uncovering a dominant strategy.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->Balance is also one of the major reasons most people stop playing a game.<!--QuoteEnd--></div><!--QuoteEEnd-->No. The main reason someone stops playing a game is that it doesn't work, or if it is not enjoyable at all (i.e. inability to play or boredom). Only extreme balance issues can have this latter effect, and only a very small percentage of players will not find enjoyment in a game because of minor or medium imbalances: look at how many people still love playing Nancy and Ayumi despite them being highly biased to one side. Only a minority will quit a game because of balance issues that can only be spotted at a high level, because only a minority of players globally will ever play at that high a level. You are arguing that the game should primarily be tested to keep this minority of players happy with the game, I am arguing that there are bigger problems that affect all players regardless of skill that a 10 person dev team with no dedicated testers cannot find on their own.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->The bottom line: if it's so easy to fix, it shouldn't have made it to release anyway.<!--QuoteEnd--></div><!--QuoteEEnd-->The bottom line is that in reality, there are much bigger fish to fry than balance issues in the last few months prior to release.
Basically if the game doesn't work you can't enjoy the game. If the game is working and the balance isn't perfect, only a very small minority of players will notice this and stop enjoying the game.
The Australian NS community totally ignored (and looked down on a bit) custom server players/community for a long time, and if we were to have playtested a beta back then, mod integration isn't something our competitive community would have been looking for. The custom servers are still alive and kicking though all these years later, so shouldn't they (for example) also be sought for feedback on things I don't care at all about - since they managed to have an active community playing the game for a long time. I hope playtesting is allocated to all kinds of players as a means to making ns2 a great game for everyone. Competitive players, custom noobs, brand new players, and any other categories of players.
Not to nitpick, I never said in this thread that one needs a "high level" of skill to test a game. I stated that players need a "competent" amoung of skill to be able to "compete" (i.e. put up resistance) against those that exhibit superior skill most of the time.
What I also stated that is that while communication is great and important it is not the end all be all of a quality tester. You can have someone that writes great reports and can communicate openly with the developers, but if they do not understand the funadmentals of design or does not show regularly, or gets rolled everytime they play, he or she's perspective on balance will be significantly skewed, but well communicated.
Perhaps we are discussing a few different things. I am going based on the OP title of "beta" testing meaning that a significant portion of the bugs are replaced. My understanding is that beta testing would be appropriate for testing the features and the balance on a much larger scale than an alpha test, similar to the previous testing involving the constellation members and the veterans. These members would require the attributes that I recommended previously in this thread.
A player’s mentality is a pretty good indicator of their skill. Maybe not in every case, but in most, if a player is competitive then they’re looking to constantly improve themselves therefore they’ll become better. If a player is skilled, chances are they are care about the game they’re playing in and want to see their team win.
You don’t need a degree in professional game testing to see that a group of college frat boys playing a round of halo and laughing hysterically every time someone gets blow apart by a rocket while chugging beers will have different skill levels than a clan practicing over headsets for an upcoming match. The groups will play the game differently, have different skill levels, and may perceive balance differently as well. Clinging onto this “it’s not the skill it’s the mentality†argument is factually correct, but there’s such a clear relationship between these two factors that your argument just doesn’t make sense in the real world.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->You are talking about public beta testing.<!--QuoteEnd--></div><!--QuoteEEnd-->
No, I’m not. I’m talking about beta. Since when does “balance testing during beta†translate into “public beta?†I must have missed out on Starcraft and Warcraft 3’s public beta tests, since that’s where all of their balance testing must have taken place. Semantics are not relevant. Call ‘beta’ whatever you’d like, and call it ‘mentality’ instead of ‘skill.’ In the end, a retail game must be balanced as well as possible before release. To do this, you need competent testers who are able to discover overpowered strategies and are skilled enough to not let all the other play testers roll over them.
This seems to me to be somewhat of a red herring, you might state there is a strong relationship between game playing skills and maturity (which is a seperate discussion), but it isn't an argument against that communication skills and maturity are more important than being amoung the most proeficient in the game.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->You can have someone that writes great reports and can communicate openly with the developers, but if they do not understand the funadmentals of design or does not show regularly, or gets rolled everytime they play, he or she's perspective on balance will be significantly skewed, but well communicated.<!--QuoteEnd--></div><!--QuoteEEnd-->
I don't think Crispy was arguing against the importance of having people understanding the fundamentals of design, and especially setting the importance of communication over people bothering showing up. You're refuting views nobody even holds!
Honestly, interesting discussion but it's drowning in fallacies.
The reason I say this is because all of your arguments seem to be based on experience of doing third-party Beta testing of commercial games (i.e. Public Beta) and maybe second-party Beta testing of non-commercial games (i.e. private mod Beta tests). The difference between your first Beta build and your first Public Beta build is quite substantial. Either we talk about 2nd-party testing of NS2 at Beta stage, or we talk about 3rd-party testing of NS2 at Beta stage. There's a big difference in the types of tester you would want for one or the other, even if you're doing a closed Public Beta.
It also seems a lot of your arguments revolve around the assumption that NS2 will fail if it's not balanced to perfection on release. This is the single biggest flaw in your argument and seems to be wholly based on your personal and very idealised view of what makes a shippable multiplayer game. If money grew on trees and if you could turn back time, this would be a reasonable expectation of a multiplayer release, but in reality there is never enough time to perfect a game for release unless you have next to unlimited resources. I'm not advocating that a game be completely broken in terms of balance on release, but there are far more important issues to focus on in the first instance before you can turn your attention to truly perfecting the balance - the sorts of issues that will turn you off a game within the very first few minutes, hours and days of booting it up - not the sort of issues that only present themselves weeks or months into the game's lifetime.
The team is very small and it has a very limited budget which in turn means a very limited deadline. Valve took many, many years to 'perfect' TF2 and since then it has constantly been tweaked for better or worse to improve the balance. There are two beauties of Steam patches: the first is that they are easy to upload and can be applied universally instantaneously, the second is there is no certification process, so as soon as a patch is ready it can be uploaded. This is why small script changes that on their own would amount to miniscule patches can be made quickly and efficiently post-release to fine-tune the balance based on widescale and prolonged experience of the game. The key word there is 'prolonged': the team probably does not have the time to be focusing on perfecting the balance you would prefer to see, but that definitely doesn't mean the game will be 'broken' on release.
Actually that is not what he said at all. Mindset and maturity often go hand in hand but are exclusive. Steve was talking about mindset meaning a particular player's attitude toward the game. Some players just play a game and do not care about the result and just want to have fun. Others take the game more seriously and want to enjoy the game on a deeper level. So they develop a mindset that allows them to accept their own mistakes and correct them to improve his or her own game. I would also argue that understanding how the developers want the game to turn out would be of a higher priority than purely being able to communicate.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->Honestly, interesting discussion but it's drowning in fallacies.<!--QuoteEnd--></div><!--QuoteEEnd-->
I agree, please read more carefully before pointing out fallacies.
If you are referring to me, my statements are to benefit the entire community, not just the competitive one as we all know there is a symbiotic relationship between both casual and competitive players. The fact that I have never commercially tested games has very little to do with this argument, as I have playtested NS1, and I know the funadmentals of the game design, and I have a pretty good idea what the developers are planning. Why do you assume I am only for the competitive community? What do you feel that you have learned as your 15 months of being a "professional" games tester that I have not learned in the 2+ years of being involved with NS playtest builds? If the game is not as balanced as possible I do not believe many people will purchase the game because there will be a few dominating strategies that will ruin the game for some people. If 1.04 was a commercial release (no offense flayra and co) it would have been a commercial failure. Yet they followed a similar model to the one you are suggesting. It was relatively bug free (few exceptions) but it was a BROKEN game.
Also to add: A lot of multiplayer FPS games are not as "balance sensitive" as NS is. You do not hear many people crying imbalance in CS and CS:S or CoD4 for that matter. NS is unique with regards to combining FPS and RTS features which are mostly bound to the "RTS" side of things that need to be addressed prior to release.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->It also seems a lot of your arguments revolve around the assumption that NS2 will fail if it's not balanced to perfection on release. This is the single biggest flaw in your argument and seems to be wholly based on your personal and very idealised view of what makes a shippable multiplayer game. If money grew on trees and if you could turn back time, this would be a reasonable expectation of a multiplayer release, but in reality there is never enough time to perfect a game for release unless you have next to unlimited resources. I'm not advocating that a game be completely broken in terms of balance on release, but there are far more important issues to focus on in the first instance before you can turn your attention to truly perfecting the balance - the sorts of issues that will turn you off a game within the very first few minutes, hours and days of booting it up - not the sort of issues that only present themselves weeks or months into the game's lifetime.<!--QuoteEnd--></div><!--QuoteEEnd-->
Again, I'm just using experience from other games, I do not want to see NS2 become the ET:QW of the NS series. No game is perfect, but it does not have to ship broken either. It IS reasonable to test any extreme strategies and potentially abusive strategies on both sides before shipping. This is not an old console game that needs to be perfect on release, however if the game is not balanced, it will probably not be fun for a majority of the players looking for tight/close games that often times produce the most fun. If the game is not fun it will not sell.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->The team is very small and it has a very limited budget which in turn means a very limited deadline. Valve took many, many years to 'perfect' TF2 and since then it has constantly been tweaked for better or worse to improve the balance. There are two beauties of Steam patches: the first is that they are easy to upload and can be applied universally instantaneously, the second is there is no certification process, so as soon as a patch is ready it can be uploaded. This is why small script changes that on their own would amount to miniscule patches can be made quickly and efficiently post-release to fine-tune the balance based on widescale and prolonged experience of the game. The key word there is 'prolonged': the team probably does not have the time to be focusing on perfecting the balance you would prefer to see, but that definitely doesn't mean the game will be 'broken' on release.<!--QuoteEnd--></div><!--QuoteEEnd-->
True but they also do not have to pay the playtesters for helping them develop the game. That is the beauty of developing a sequel of a the most popular 3rd party half-life mod; There is a huge community to pick potential testers from and most of them will do it for free. Its also awesome that the devs are using LUA scripts to tweak everything; This allows for a more efficient playtest design as values can be changed on the fly and reset at any point. This would permit the developers to do more balance testing whilst not ignoring features and bugs due to the new method of coding.
I'm not sure where you got this, but at no point did I ever say "ignore other areas." By the time a game is ready to have people beta test the majority of game-ending bugs, exploits, and stability issues should be resolved. Obviously, if a tester finds a bug grants unlimited ammo during a play test, they aren't going to keep quite about it. The impression I got from you, and what lead me to post, was that you seem content on letting balance slide, even into release.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->It also seems a lot of your arguments revolve around the assumption that NS2 will fail if it's not balanced to perfection on release<!--QuoteEnd--></div><!--QuoteEEnd-->
Yet again, I'm not sure how you take what I can say then bring it to an extreme that doesn't make sense. Balance perfection is incredibly difficult to achieve throughout the life of a game yet alone just in a beta. I've made my stance fairly clear, and I don't really care for the semantics of 'beta' as opposed to 'play test' since I'm not a super 'professional game tester,' so I'll just quote myself and hope I'm clear enough:
<!--quoteo(post=0:date=:name=SentrySteve)--><div class='quotetop'>QUOTE (SentrySteve)</div><div class='quotemain'><!--quotec-->Call ‘beta’ whatever you’d like, and call it ‘mentality’ instead of ‘skill.’ In the end, a retail game must be balanced as well as possible before release. To do this, you need competent testers who are able to discover overpowered strategies and are skilled enough to not let all the other play testers roll over them.<!--QuoteEnd--></div><!--QuoteEEnd-->
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->Yet they followed a similar model to the one you are suggesting. It was relatively bug free (few exceptions) but it was a BROKEN game.<!--QuoteEnd--></div><!--QuoteEEnd-->Please quote the full comments I made supporting this argument if you are so brazen as to suggest a link. If you read what I say you will see I am not suggesting that balance be ignored in any way. I simply said that a competant QA Lead can account for these things and organise for testers to protect against them. I even said a dedicated team of testers should be put in a competitive environment to test all strats. What I didn't say was that all testers need to have these qualities. There is your distinction.
<b>@Steve:</b>
The arguments presented bvy FW, which you agreed with, called for a high priority on skill for ALL testers allowed into any NS2 beta. He even went so far to say that if this wasn't evident then communication skill would be worthless. This is what I disagree with. I said, quite clearly, that you should have different teams focusing on different things. One team would focus on the balance, and would be made up of high skilled players that would play against eachother in a competitive setting to eek out the hardcore ctrategies that can make or break the game. Other teams or segments of players would focus on other things. Ideally the hardcore balance testers would be involved earlier on in the process before a public beta. That is how I'd run it. All I'm saying is that you should not focus your Betas around balance when there are far more problems that can ruin your game. FW's ideal was for high-skilled players, which is simply not necessary.
The distinction between early and late Beta is important, because the game changes a lot between those stages. You need a different calibre and class of tester for those two periods of testing. You need to be looking for different individual tester qualities.
I personally think that Global Agenda has been doing a really good job with their Alpha testing, and they're prepping for their closed Beta tests. They are accessible (the entire alpha team is Friended on Steam) so turnarounds are really quick. They've mostly been doing functionality tests while keeping some semblance of balance as they add in new stuff. Of course, most of the testing is in-house for major things (like new game modes) before being dumped on the alpha team.
@ the topic on hand, I feel that betas should try to keep both functionality and balance in mind. Perhaps the first and foremost testing in early beta builds should be functionality. Stress the system and find the leaks. If a borken strat comes out (insta win!), try to compensate so that people try other features, but don't worry too much about the finer points of balance. Later builds that are feature complete should start to get a bead on the balance issues. It's nearly impossible to perfect balance, even with a crack team of professional devs and players. However, for a serious release of a competitive multiplayer game there should be at the very least a decent semblance of balance.
While it is true that NS relies heavily on the team dynamics and RTS factors, plus the asymmetry between the teams, thus pushing up the importance of balance, I personally am turned off more by broken mechanics than balance. Minor glitches are OK (small level glitches, rare wtf glitches, but not something like the infinite melee in L4D), but stupid bugs really make the game feel inferior. In some regards, balance is a game mechanic, so a perfectly polished but very unbalanced game also makes me run very far away. I didn't come for the eye candy, I came to play others and have fun.
I guess anyone could be considered professional if they get paid. I mean I once sold toys at Kay B toys, I guess you could say I was a professional salesman. I've also tested other games/mods but was never "professional" with regards to compensation. I disagree with the mod testing that is not focused, perhaps some of the mods you've tested were like that but I would disagree with that with regards to NS. I've always felt that NS (since Flayra was a professional game programmer prior to creating NS) ran a tight ship with regards to how to test the mod and run testing groups effectively. Bare in mind again, NS was very popular at the 2.01 stage because it was a very polished mod that a lot of people felt was professionally designed. Granted there have been a lot of dumpy mods that came out (or never did) that may have fit the mold that you have previously mentioned. However, since Flayra and Co always wanted NS to develop into something that could be commericially successful, I do not believe that the time playtesting for them would be that of a stereotypical mod.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->Please quote the full comments I made supporting this argument if you are so brazen as to suggest a link. If you read what I say you will see I am not suggesting that balance be ignored in any way. I simply said that a competant QA Lead can account for these things and organise for testers to protect against them. I even said a dedicated team of testers should be put in a competitive environment to test all strats. What I didn't say was that all testers need to have these qualities. There is your distinction.<!--QuoteEnd--></div><!--QuoteEEnd-->
Yes but you also stated that balance is not as high of a priority, and as far as your concerned could be shipped "broken" (i.e. bug-fee but unbalanced) as a result because of the lack of emphasis on balance testing. You have also stated that that UWE is a small studio with time constraints. Would it not make sense to have the same group of testers I have prioritized to test both bugs and balance? I can't imagine how much time it would take to organize these groups, seperate them effectively, run several different playtests for bugs, and balance etc... It seems to be an inefficient design with regards to testing. If you have one group that is competent enough to not get rolled during a game (thus reducing skill as an extraneous variable for balance testing purposes) that can also spend the time to find bugs and report them to the dev team, would that not be a more effective design?
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec--><b>@Steve:</b>
The arguments presented bvy FW, which you agreed with, called for a high priority on skill for ALL testers allowed into any NS2 beta. He even went so far to say that if this wasn't evident then communication skill would be worthless. This is what I disagree with. I said, quite clearly, that you should have different teams focusing on different things. One team would focus on the balance, and would be made up of high skilled players that would play against eachother in a competitive setting to eek out the hardcore ctrategies that can make or break the game. Other teams or segments of players would focus on other things. Ideally the hardcore balance testers would be involved earlier on in the process before a public beta. That is how I'd run it. All I'm saying is that you should not focus your Betas around balance when there are far more problems that can ruin your game. FW's ideal was for high-skilled players, which is simply not necessary.<!--QuoteEnd--></div><!--QuoteEEnd-->
Again I guess the question I have to ask is why have seperate teams, why not just have the competant players do the bug testing and balance testing? And again, please read CORRECTLY. I said competent players, not HIGH SKILLED. There is a significant difference. Please quote any reference in this thread where I stated that the playtesters had to be high skilled.
<!--quoteo--><div class='quotetop'>QUOTE </div><div class='quotemain'><!--quotec-->The distinction between early and late Beta is important, because the game changes a lot between those stages. You need a different calibre and class of tester for those two periods of testing. You need to be looking for different individual tester qualities.<!--QuoteEnd--></div><!--QuoteEEnd-->
Again, this isn't Dungeons and Dragons where everyone specializes in one "class" that cannot perform the abilities well of another class. I still do not see a reason why the balance testing group cannot be combined with the bug testing group, to create one SUPER group of testers.