224 tech changes, part 2
matso Master of PatchesMembers, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer Join Date: 2002-11-05 Member: 7000Posts: 1,554 mod
"No Good Deed Goes Unpunished"
The playtesters showed up in force, and we had a good server running. Everything was flowing along nicely at 12, 16 players, and then ... everyone started rubberbanding madly. Net_stats showed the server had dropped like a rock from 30 to 5-10 ticks. This was fairly early game, nothing built up, and the server was pretty hot and would easily have handled the player count in 223.
Now, that wasn't totally unexpected - the server runs all of the clients moves, so with 16 players pumping in about 70-90 fps, the server just couldn't deal with it. Fortunately, the maxfps command was available, so all the playtesters could set their maxfps to 60, allowing the server to recover and we could continue playtesting, finding some extra weird bugs in fringe places (like spectator, first person mode, or commander mode, clicking on minimap glitches out etc).
All in all, looked good and the bug list didn't look too bad. But ... there wasn't a server around that could run 224 even with a 8v8 game. So it was not possible to release it as is.
Solve one problem, create a new one.
After a bit of brainstorming, we tried lowering max fps to 60 as a temporary measure to allow playtesting to continue, but even that soon ran into server problems, with servers that used to be able to hold 30 ticks for 20 players endgame suffering tick-crashes in mid-game.
So we branched out. Max started to dig into how Source handle it, while I started looking into move-merging. Basically, with 100 moves per second generated, each move wouldn't do much - at 5 m/s, you only move 5cm per frame, so it should be possible to merge frames together and only run about 30 moves per second.
We were looking at doing an interim solution to get 224 out quickly, accepting that 224 might suffer a bit of client/server misprediction, then coming up with a better solution for the next patch.
Max and Dushan started looking into server-side move merging, where the server simply accumulates incoming moves and executes them 30 times per second. At first, I had also looked at server-side merging, but I found it unsatisfactory, as I had the choice of either letting it get really sloppy OR spending a lot of CPU to make sure the moves were mergeable - not a good idea if you are trying to SAVE CPU.
After a few days of struggling, I was taking a shower when I had a great idea - the client already had a world that was running moves in parallel to the main world; the prediction world. I should be able to use that to run the moves both as single moves and as merged moves, then compare the end-result and see if the merged move ended up being close enough and trivial enough to allow the merge.
All said and done, I reworked the PredictionWorld, adding support for merging moves as they came in from the Client. So the Client reads inputs, constructs a move, hands over a copy to the Predict/Merge world (checking if there is a fresh server update at the same time), then processes that move and uses the result to render.
At the same time, the Predict/Merge world processes the move in ITS world twice, once as a single move, and once by trying to merge it with the previous move. Callouts to Lua at strategic times allows the Lua code to allow/deny the merge, so the analyses on what is a mergeable move isn't part of the engine, but instead is part of the game.
Performance wise, this would actually be FASTER than not doing the client side merge. Assuming 100ms latency, 100ms interp and 100 fps, you are looking at at running 20 prediction moves for each server update, with the updates coming in 20 times per second for a total of 400 moves (in 223, you would be looking at 20 moves 100 times per second for a total of 2000 moves... - not that you could ever GET to 100 fps in 223 with 100ms latency). Merging those 20 moves into an average of 7 moves means that you would need to run 140 moves. Adding in the about 70 merge moves attempt you can expect to need, you are still coming out about 200 moves ahead. And with each move at 0.5-0.7ms, that's about 120ms or 12% cpu less, in the off-core thread.
A day or two of coding showed it working fairly nicely, with a few weird bugs. Worked plenty fine from first-person perspective, but there was an unforeseen problem - as the player now updated his position only 30 times per second, the player movement smoothing code didn't work well any longer, resulting in other players moving erratically on your screen.
Fortunately, Max had already thought of a solution for low-fps player movements smoothing. Implementing it wasn't really necessary when player update rate was at 40+, but now when suddenly everyone was "low-fps ######es" as a PT said about other playtesters with low fps, Max proceeded to rip apart the network protocol and put it back together again. That's something Max feels comfortable doing in a few hours or so...
Those weird bugs ... they turned out to be pretty difficult to find. Most visible was that sometimes when reloading the pistol and FT, the reloading animation played twice. Sometimes. It turned out that the animation engine is very sensitive to when an animation starts. In the Predict world, the animation engine is used for one thing; controlling the timing of things like attacks and reloads (it's also used to produce BoneCoords to put some spine into models for rendering and physics, but that's not used in prediction).
Analysis showed that starting a new animation had to block merging; merging in any animation with another move really confused things no end. Fortunately, on the client the move is first run on its own, so the Lua code could just save up the prior relevant state of the animation, wait for the move to be run, and make sure that a new animation hadn't started before allowing a merge attempt.
While I had been working with the client side try, Max and Dushan had run into problems with the quick-fix server side merging; the same problems with animation synch and on the server, there was no CPU available to do any analysis of the moves, so mispredictions were large and hard to do anything about. So 224 was not getting out.
This had taken a week (+ weekend) of hard work, but by Wednesday the client-side merge was ready to be playtested, and after some heroic and epically long playtesting (7+ hours) and on-the-fly bugfixing by UWE, 224 was ready to be released.
So in the end, 224 brings you three insane upgrades in one package; both a new client multiworld/multithreaded system, an innovative dynamic merged move system and a new network protocol. Net effect should be about +40% client side fps (with the side effect of latency no longer lowering your fps), +20% server side player count and silky smooth player movement.
Member of CDT, Senior Spark Engine Hacker