Pondering Rapid Iteration - Natural Selection 2
System
Join Date: 2013-01-29 Member: 182599Members, Super Administrators, Reinforced - Diamond
Join Date: 2013-01-29 Member: 182599Members, Super Administrators, Reinforced - Diamond
Pondering Rapid Iteration - Natural Selection 2
One of the things I’ve been rambling on about a lot lately is the ‘How’ of NS2 development. That right now the ‘How’ is more important than the ‘What.’ Specifically, I spend...
Comments
https://en.wikipedia.org/wiki/A/B_testing
No matter how small you plan to do changes, eventually you will break a mod. Now, the timing might mean you can kill off an entire server if it hits the specific server's mod, the entire server population if you hit a mod big enough (like NS2+), competitive servers (NSL/Compmod)... And then you will depend on said mod developers to be around to fix it, which, thanks to timezones and real life commitments, it could take a while.
Server ops need to be notified in advance. If a comp match is in progress and you update in the middle and someone crashes, he won't be able to rejoin.
And there's plenty more reasons why this will never work.
What specifically is the pay off from this change?
Because I am concerned that fast iteration for the sake of fast iteration isn't a pay off, really... I don't see it bringing in more players (why would it, it's not breaking any headlines nor is it something that attracts beyond "re entering development") and it definitely has potential to create issues for retention as you mentioned.
So I am genuinely curious from your POV @Hugh , because I don't see it yet, what is the pay off from this risky model?
From the sound of it, this has no place in a live game that people are playing. This only has a place in a separate "experimental branch" that people can opt into. Once a version is stable, those could pushed to the live version. However that won't work because this is a MULTIPLAYER game with LIVE servers. It's not a single player experience like Subnautica, where server population has no influence over testing a buggy version, you can choose if you want to test new stuff or not.
I can tell ya now, even if that was the idea. With this tiny community of ours, you can be certain server ops will not opt into an experimental branch. Having a stable and experimental branch would mean your players could choose which version they want, but in that case can't find servers because of version differences
>Right now it sounds like high risk with potential for disaster down the line at any give moment due to the minefield approach<
And @IronHorse brings up a good point, tiny patches don't impress new customers. So how does this lure in more players, because that was the entire point, right? Along with fast patching right along with having a liquid game, which can be molded very quickly. I just see a very high risk factor, which will piss of players with bugs/crashes/no servers. And no control specimen, like a stable build vs experimental build.
@Hugh, you've talked about "How", but not really going into detail as to... You know "How" this will be a controllable situation, while luring in new players...
This doesn't sound practical in a multiplayer environment.
It actually already happens. With NS2+.
Mendasp can update it as much as he wants, as often as he wants. As an example, if you put the latest build of NS2 into a mod on Steam, it will auto-update everyone & servers at the end of a round, or on map-change. And with more than 1 person developing this 'mod' (which is NS2) iteration can be faster, and bugs fixed quicker. A new Hive backend can be constructed so that if Steam fails, the 'mod' can always be got.
I'm not saying this is the best way to do it, but to me, it's the most practical. Emulate NS2+ and how Mendasp has updated and worked on his mod, and apply it to NS2's own development. He proved that it can work very well, it might be time to emulate his methods for NS2 itself.
Right now I just see this rapid prototyping being a means to ending the game rather than saving it.....
I also believe it will push developers away from risky changes with big payoffs that require extensive QA (I.e. all the hit reg fixes, all the hitches and file system changes, network improvements, engine bugs, etc.). These likely would have never happened had it not been for a build-test-break-fix iterative process, as the first builds have far too many bugs. In essence your taking the game back to a beta or alpha stage and making us all testers that paid for a final product.
When I get off work, the last thing I want to find is a game breaking bug preventing me from enjoying my favorite game. This method increases the chances of that happening as a regular occorance. I'd rather have what happened this week that you so vocally decried as a massive failure of the monolithic building process than what you are proposing.
Regardless, I'm addicted I'll probably stick around because I love the game and playing with many of the senior playerbase. However, when the scales tip from the game being fun to being an aggravation, I can see lots of loyal players leaving (the same ones who have kept the game alive since UWE left). Quite the gamble when I don't see a particularly big potential gain. Considering the up side hasn't been communicated to any of us...... Though I suppose communication has never been a strong point under UWE, of past performance is anything to go by.
That's for mods in the steam workshop, unless you plan on delivering updates to the game through the workshop then it's an entirely different matter.
But...That's what I said? It's certainly a way to do it. NS2+ already proved it can be done.
I guess we'd also love to hear an answer on this risk vs reward factor @Hugh is so keen about... We've been waiting for... What 2 or 3 weeks now? Would you please stop with the vagueness already
Can that be done with engine changes? Surely you'd have to restart regardless for some updates.
My concern is more the fact that I'm not interested in downloading new updates constantly. To me, the rapid iteration model, even if it works perfectly with no issues, is not anything I desire.
I can see the value in being able to hotfix rapidly. But other than that, I really prefer the slower in nature updates.
Oh well, we will have to see I guess - maybe this is a brilliant idea in practise, who knows.
NS2+ IS a product that rapid iteration works for, as it is mostly a set of features that sit on top of the more complicated product, users will see direct benefit from individual features being added iteratively.
It seems like @Hugh has argued against rapid iteration in his own blog post by admitting to the cons this style of development will have on NS2 without saying exactly what the pros will be. If the pro is simply that it allows for faster iteration, then yes I agree that generally speaking rapid iteration is awesome, it just won't work for NS2.
Surely UWE didn't sit up one day and say "We can do this, therefore lets do it!." And then come up with a long list of questions, talk about them, then ask the community about some more questions they might have.
We've posited questions on these forums, we've been asking these things since the initial vauge references were made, we want to know the answers to these questions. We don't want to know the answers after s*** hits the fan, if s*** hits the fan.
This is an interesting point, Perhaps something for you at UWE to consider is to make an Official "Update" Mod (Which I seem to remember being suggested like gazillions of times before), and update that as frequently as your hearts desire. Server ops can opt into having this mod active on their servers, nothing horrible breaks, and new content/ideas are delivered in realtime to the players. The mod can be disabled at any time if something terribad happens.
Limitations of this method are that major engine changes won't work (right?). And since those are the biggest baddest most possible to break everything changes, that's completely fine. Those changes go out to Playtesters and Beta build curious people in the processes that have been refined in the years since this game has been released untill they've been deemed stable.
At that point, you compile all the engine changes beta things, and all the stuff from the Mod you want to, and push out a new 'monolithic' build. Rinse and repeat.
Just like that you've made everything possible to happen quickly, efficently (as far as steam can move efficiently), and you've put the ultimate decision power in the hands of us, the people who play the game and run the servers. Until everything is stable and accepted, then you push things out permanently.
You haven't yet responded about the point behind this, but usually the point behind an agile development cycle is to test the product in the real world and see if the consumers like the changes/new features quickly. So ultimately your goal is to make the game better for us, the players/community right? So you're not going to ignore us when we cry out in dismay at a change and disable the mod on the servers until something is changed back, right? Just checking because it seems a lot of the time that we're talking at a one way mirror.
Rented servers are a thing, and they're not the easiest thing to update or control. Often times you have to get on the horn with someone at support to get things updated. It's been nice having advanced warning of such updates so you know when to yell at who, with rapid-fire (even if it's not every half-hour) updates will be even more of a pain for admins who don't have the ability to write/use scripts on their boxes to check for/update automatically.
Furthermore, every update to the game requires a complete shutdown/restart of the server, unlike a mod update, which is just a mapchange. This is basically a "Kiss your seeded server population goodbye" because once they see that redplug, they'll move on (assuming they're not a community based server, and even then). So there's that too.
Basically, it's rough enough as is to keep lights on in a rented server, increasing the game update rate will only make it tougher, and seeing as how there's no more UWE Official servers, some strong consideration to the server admins should be there.
What i propose is a public beta branch and the public will be invited to special public playtest events. Everything else wont work, especially not in 3 months.
You say people ask you if you actually want to kill ns2. You certainly dont want to, but im also convinced this will be the result. Also consider the experience those people have in software development, they might know what they talk about.
The Agile Manifesto
Everyone of us is very much aware that daily builds are not an option at the moment due to:
But if we find a way to solve some of those issues a faster release cycle (not directly daily builds) is possible to do.
What's the idea behind using agile development for NS2 is basically that the past has shown that often features which were developed over month had absolute no effect on the player base and went unnoticed/unused. From now on we do not want to invest any time on a feature that you neither want nor use in the end.
Secondly NS2 builds often got rather complex with many features depending onto each other. Then when one of those feature turned out to be broken we often had no other choice but to revert a complete set of features. That should not happen anymore. Of course we don't want to use the player base as testers but there will be always edge cases all our internal tests won't cover. Those issues have to be uncovered and reacted on much faster than they have been reacted on in the past 2 years.
@elodea Every update will come with a change log as we have to make sure you are actually aware what to give feedback on. We even already went a step further and made the ns2 repositories commit history public (still WIP): http://goo.gl/hDlYAU
It doesn't really matter what you call it... xP The idea outlined in the blog post hints at a "trialanderror" development mode, which to me means that devs dream up and code a change then push it to the public to see what will happen.
On the other hand, if the idea is still to thoroughly test any changes made to the game, why announce this in the first place? As far as I know, that's the current practice.
Don't get me wrong, I'd love to see the game getting even more awesome, but the fanbase is a bit fickle. You really don't want to mess up the playtime the few us have, otherwise the game will die if even 1/10 times there is some problem with an update...
But that is not what Hugh is suggesting. What has everybody up in arms is this:
"But really, I mean it. No more builds. Any developer can push any change to the public at any time. With, if they so choose, no testing."
This combined with the 2 builds per hour suggestion is what can not work.
Yes, there are certainly ways to improve the development process, but what the blogpost proposes is not that.
Also, agile software development is great. Its used in many developments of new software. But it is rarely used, for good reason, in software that is already released and has a stable userbase that relies on the software to work.
With agile development you accept the fact that stuff will break, often. Which is fine if the software does not has work all the time. But in the context of ns2 where you have to update the servers, the clients, then detect that something is broken, roll back, update servers and clients again, handle the bugreports of all people that try to connect to servers with the wrong version etc, thats not going to work.
Even right now with the "relatively" low update frequency you see techsupport questions on about every update that come from the fact that the player was caught somewhere in the update cycle. Maybe his steam repository did not yet update, but the server did etc. The first 4-5 hours of every update are always a chaos. Strangely enough, often the bugreports during that period can even be ignored, because they are often caused by client server mismatches, or steam causing some sort of issue when updating the client. Why is that dangerous? Because it makes identifying issues that need a fast roll back very hard.
Other than for trivial features i still don't see a good proposal on how this should work. 3 months is a tight schedule. You likely need at least 3 months to even set up a system that works reliably and does what you want to.
And even after all this, i highly question the "value" added by the system. Yes, ns2 updates did not come out all the time as fast as we liked, but considering the CDT is/was an unpaid team this is normal. Simply hiring the PDT and continuing on a similar path as we had now would have a much bigger impact on the game than what is proposed now.
You can not start a 3 months trial period during which you also want to radically change the way the game is developed. And its not just radically change, its doing something nobody else does, so you dont really have something to base your ideas on.
So, tl;dr. Yes, the development process can certainly be improved and should be, but not the way that is proposed in this blog post.
One approach to the "updates break" problematic would be something like this:
- At all times there are always two version of the game (a and b) on all clients and servers.
- If a clients joins, the server tells him which version he is currently running and that one gets used.
- If an update is release (while version b is considered stable), version a gets removed everywhere and the new version is added.
- Server admins can switch between both versions at will.
With that, if a release is broken server admins can just "rollback" to the previous version till everything is fixed.
Server admins can test a new version while they watch for mod problems and can give reports.
Modders have more time fixing broken mods, knowing that their mod still can used with the older version.
Of course that would involve some work on uwe side and some patches may not work with that system and have to be release normally.
Contact me, if I should elaborate.
As both a player and a server admin I am 100% against. I am not interested in the slightest.
While, in essence, having some bug fixed faster, I can not truly believe that stuff will not break. Mod compatibility does not have a good track record. It will break. If NS2 is patches that fast, it will break at times which are far from ideal.
Add to that, that server logs cant be read if the server is actively writing to it, and you have no way to check how happened 30mins ago with the gazilionth patch without shutting the server down.
This will just end with server admins stopping to fully check if their server is performing optimal, because it will change to soon regardless.
Also I can not imagine it would be benefitial form competitive either. This means the game some played 30mins ago is already different from the guy the other team is playing now. And if I understood the post correctly, the patch changes could be anything. Great. Im sure we will all love that. (yes thats sarcasm.)
I can see nothing but problems with this, and I truly hope it shall never ever happen.