224 tech changes, part 2
matso
Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
<div class="IPBDescription">"No Good Deed Goes Unpunished"</div>
The playtesters showed up in force, and we had a good server running. Everything was flowing along nicely at 12, 16 players, and then ... everyone started rubberbanding madly. Net_stats showed the server had dropped like a rock from 30 to 5-10 ticks. This was fairly early game, nothing built up, and the server was pretty hot and would easily have handled the player count in 223.
Now, that wasn't totally unexpected - the server runs all of the clients moves, so with 16 players pumping in about 70-90 fps, the server just couldn't deal with it. Fortunately, the maxfps command was available, so all the playtesters could set their maxfps to 60, allowing the server to recover and we could continue playtesting, finding some extra weird bugs in fringe places (like spectator, first person mode, or commander mode, clicking on minimap glitches out etc).
All in all, looked good and the bug list didn't look too bad. But ... there wasn't a server around that could run 224 even with a 8v8 game. So it was not possible to release it as is.
Solve one problem, create a new one.
After a bit of brainstorming, we tried lowering max fps to 60 as a temporary measure to allow playtesting to continue, but even that soon ran into server problems, with servers that used to be able to hold 30 ticks for 20 players endgame suffering tick-crashes in mid-game.
So we branched out. Max started to dig into how Source handle it, while I started looking into move-merging. Basically, with 100 moves per second generated, each move wouldn't do much - at 5 m/s, you only move 5cm per frame, so it should be possible to merge frames together and only run about 30 moves per second.
We were looking at doing an interim solution to get 224 out quickly, accepting that 224 might suffer a bit of client/server misprediction, then coming up with a better solution for the next patch.
Max and Dushan started looking into server-side move merging, where the server simply accumulates incoming moves and executes them 30 times per second. At first, I had also looked at server-side merging, but I found it unsatisfactory, as I had the choice of either letting it get really sloppy OR spending a lot of CPU to make sure the moves were mergeable - not a good idea if you are trying to SAVE CPU.
After a few days of struggling, I was taking a shower when I had a great idea - the client already had a world that was running moves in parallel to the main world; the prediction world. I should be able to use that to run the moves both as single moves and as merged moves, then compare the end-result and see if the merged move ended up being close enough and trivial enough to allow the merge.
All said and done, I reworked the PredictionWorld, adding support for merging moves as they came in from the Client. So the Client reads inputs, constructs a move, hands over a copy to the Predict/Merge world (checking if there is a fresh server update at the same time), then processes that move and uses the result to render.
At the same time, the Predict/Merge world processes the move in ITS world twice, once as a single move, and once by trying to merge it with the previous move. Callouts to Lua at strategic times allows the Lua code to allow/deny the merge, so the analyses on what is a mergeable move isn't part of the engine, but instead is part of the game.
Performance wise, this would actually be FASTER than not doing the client side merge. Assuming 100ms latency, 100ms interp and 100 fps, you are looking at at running 20 prediction moves for each server update, with the updates coming in 20 times per second for a total of 400 moves (in 223, you would be looking at 20 moves 100 times per second for a total of 2000 moves... - not that you could ever GET to 100 fps in 223 with 100ms latency). Merging those 20 moves into an average of 7 moves means that you would need to run 140 moves. Adding in the about 70 merge moves attempt you can expect to need, you are still coming out about 200 moves ahead. And with each move at 0.5-0.7ms, that's about 120ms or 12% cpu less, in the off-core thread.
A day or two of coding showed it working fairly nicely, with a few weird bugs. Worked plenty fine from first-person perspective, but there was an unforeseen problem - as the player now updated his position only 30 times per second, the player movement smoothing code didn't work well any longer, resulting in other players moving erratically on your screen.
Fortunately, Max had already thought of a solution for low-fps player movements smoothing. Implementing it wasn't really necessary when player update rate was at 40+, but now when suddenly everyone was "low-fps ######es" as a PT said about other playtesters with low fps, Max proceeded to rip apart the network protocol and put it back together again. That's something Max feels comfortable doing in a few hours or so...
Those weird bugs ... they turned out to be pretty difficult to find. Most visible was that sometimes when reloading the pistol and FT, the reloading animation played twice. Sometimes. It turned out that the animation engine is very sensitive to when an animation starts. In the Predict world, the animation engine is used for one thing; controlling the timing of things like attacks and reloads (it's also used to produce BoneCoords to put some spine into models for rendering and physics, but that's not used in prediction).
Analysis showed that starting a new animation had to block merging; merging in any animation with another move really confused things no end. Fortunately, on the client the move is first run on its own, so the Lua code could just save up the prior relevant state of the animation, wait for the move to be run, and make sure that a new animation hadn't started before allowing a merge attempt.
While I had been working with the client side try, Max and Dushan had run into problems with the quick-fix server side merging; the same problems with animation synch and on the server, there was no CPU available to do any analysis of the moves, so mispredictions were large and hard to do anything about. So 224 was not getting out.
This had taken a week (+ weekend) of hard work, but by Wednesday the client-side merge was ready to be playtested, and after some heroic and epically long playtesting (7+ hours) and on-the-fly bugfixing by UWE, 224 was ready to be released.
So in the end, 224 brings you three insane upgrades in one package; both a new client multiworld/multithreaded system, an innovative dynamic merged move system and a new network protocol. Net effect should be about +40% client side fps (with the side effect of latency no longer lowering your fps), +20% server side player count and silky smooth player movement.
Enjoy!
The playtesters showed up in force, and we had a good server running. Everything was flowing along nicely at 12, 16 players, and then ... everyone started rubberbanding madly. Net_stats showed the server had dropped like a rock from 30 to 5-10 ticks. This was fairly early game, nothing built up, and the server was pretty hot and would easily have handled the player count in 223.
Now, that wasn't totally unexpected - the server runs all of the clients moves, so with 16 players pumping in about 70-90 fps, the server just couldn't deal with it. Fortunately, the maxfps command was available, so all the playtesters could set their maxfps to 60, allowing the server to recover and we could continue playtesting, finding some extra weird bugs in fringe places (like spectator, first person mode, or commander mode, clicking on minimap glitches out etc).
All in all, looked good and the bug list didn't look too bad. But ... there wasn't a server around that could run 224 even with a 8v8 game. So it was not possible to release it as is.
Solve one problem, create a new one.
After a bit of brainstorming, we tried lowering max fps to 60 as a temporary measure to allow playtesting to continue, but even that soon ran into server problems, with servers that used to be able to hold 30 ticks for 20 players endgame suffering tick-crashes in mid-game.
So we branched out. Max started to dig into how Source handle it, while I started looking into move-merging. Basically, with 100 moves per second generated, each move wouldn't do much - at 5 m/s, you only move 5cm per frame, so it should be possible to merge frames together and only run about 30 moves per second.
We were looking at doing an interim solution to get 224 out quickly, accepting that 224 might suffer a bit of client/server misprediction, then coming up with a better solution for the next patch.
Max and Dushan started looking into server-side move merging, where the server simply accumulates incoming moves and executes them 30 times per second. At first, I had also looked at server-side merging, but I found it unsatisfactory, as I had the choice of either letting it get really sloppy OR spending a lot of CPU to make sure the moves were mergeable - not a good idea if you are trying to SAVE CPU.
After a few days of struggling, I was taking a shower when I had a great idea - the client already had a world that was running moves in parallel to the main world; the prediction world. I should be able to use that to run the moves both as single moves and as merged moves, then compare the end-result and see if the merged move ended up being close enough and trivial enough to allow the merge.
All said and done, I reworked the PredictionWorld, adding support for merging moves as they came in from the Client. So the Client reads inputs, constructs a move, hands over a copy to the Predict/Merge world (checking if there is a fresh server update at the same time), then processes that move and uses the result to render.
At the same time, the Predict/Merge world processes the move in ITS world twice, once as a single move, and once by trying to merge it with the previous move. Callouts to Lua at strategic times allows the Lua code to allow/deny the merge, so the analyses on what is a mergeable move isn't part of the engine, but instead is part of the game.
Performance wise, this would actually be FASTER than not doing the client side merge. Assuming 100ms latency, 100ms interp and 100 fps, you are looking at at running 20 prediction moves for each server update, with the updates coming in 20 times per second for a total of 400 moves (in 223, you would be looking at 20 moves 100 times per second for a total of 2000 moves... - not that you could ever GET to 100 fps in 223 with 100ms latency). Merging those 20 moves into an average of 7 moves means that you would need to run 140 moves. Adding in the about 70 merge moves attempt you can expect to need, you are still coming out about 200 moves ahead. And with each move at 0.5-0.7ms, that's about 120ms or 12% cpu less, in the off-core thread.
A day or two of coding showed it working fairly nicely, with a few weird bugs. Worked plenty fine from first-person perspective, but there was an unforeseen problem - as the player now updated his position only 30 times per second, the player movement smoothing code didn't work well any longer, resulting in other players moving erratically on your screen.
Fortunately, Max had already thought of a solution for low-fps player movements smoothing. Implementing it wasn't really necessary when player update rate was at 40+, but now when suddenly everyone was "low-fps ######es" as a PT said about other playtesters with low fps, Max proceeded to rip apart the network protocol and put it back together again. That's something Max feels comfortable doing in a few hours or so...
Those weird bugs ... they turned out to be pretty difficult to find. Most visible was that sometimes when reloading the pistol and FT, the reloading animation played twice. Sometimes. It turned out that the animation engine is very sensitive to when an animation starts. In the Predict world, the animation engine is used for one thing; controlling the timing of things like attacks and reloads (it's also used to produce BoneCoords to put some spine into models for rendering and physics, but that's not used in prediction).
Analysis showed that starting a new animation had to block merging; merging in any animation with another move really confused things no end. Fortunately, on the client the move is first run on its own, so the Lua code could just save up the prior relevant state of the animation, wait for the move to be run, and make sure that a new animation hadn't started before allowing a merge attempt.
While I had been working with the client side try, Max and Dushan had run into problems with the quick-fix server side merging; the same problems with animation synch and on the server, there was no CPU available to do any analysis of the moves, so mispredictions were large and hard to do anything about. So 224 was not getting out.
This had taken a week (+ weekend) of hard work, but by Wednesday the client-side merge was ready to be playtested, and after some heroic and epically long playtesting (7+ hours) and on-the-fly bugfixing by UWE, 224 was ready to be released.
So in the end, 224 brings you three insane upgrades in one package; both a new client multiworld/multithreaded system, an innovative dynamic merged move system and a new network protocol. Net effect should be about +40% client side fps (with the side effect of latency no longer lowering your fps), +20% server side player count and silky smooth player movement.
Enjoy!
Comments
the birth of all great ideas
i thought that was hanging clocks over toilets
Lets see how this works out. :]
Love it when it happens and it works.
Hate it when it happens and only causes more problems.
I'm proud of you, the game and being part of this community - Wish you all the best for release (get some sleep afterwards)! ;)
No client FPS increase for me in this build whatsoever :p
I hope there's some room left for more improvements client side.
That's progress for ya.
This is why i first became a playtester, helping crushing down the bugs and have a direct impact on its devolopment.
And im glad that my pleas, constant whining and cheering about every single thing, that have raised or lowered my fps by 1, have been taken into account. So i, on my lowend machine can now enjoy games whit 15-40 fps, and a positiv KD ratio.
Over the last few months, i have gone from a max of 15 fps to now seeing 50+ in the RR. Absolutly fantastic job!
The Birth of the Architect.
<!--quoteo(post=1996886:date=Oct 25 2012, 02:58 PM:name=Vitdom)--><div class='quotetop'>QUOTE (Vitdom @ Oct 25 2012, 02:58 PM) <a href="index.php?act=findpost&pid=1996886"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->... Wouldn't it just be simpler and faster to have both the server and client running state update simulations/predictions, like the server frame updates synchronizes the client-server states and the S/C just keep on simulating/predicting the game sending synchronize data back and forth, instead of keeping a list of updates-not-part-of-the-latest-server-update and just predicting what will happen over and over again? If both the server and client has access to the same game state data (which it has), there would be no additional downsides, coming with the improvements. Why would it according to you be a "hitchy experience" when that way of doing things can be implemented in a very nice and well-working way?
Doesn't the Source engine work like that?<!--QuoteEnd--></div><!--QuoteEEnd-->
I just predicted a major part of "224 tech changes, part 2" and the network changes you made. xD