224 tech changes, part 2

matsomatso Master of PatchesMembers, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer Join Date: 2002-11-05 Member: 7000Posts: 1,554 mod
"No Good Deed Goes Unpunished"

The playtesters showed up in force, and we had a good server running. Everything was flowing along nicely at 12, 16 players, and then ... everyone started rubberbanding madly. Net_stats showed the server had dropped like a rock from 30 to 5-10 ticks. This was fairly early game, nothing built up, and the server was pretty hot and would easily have handled the player count in 223.

Now, that wasn't totally unexpected - the server runs all of the clients moves, so with 16 players pumping in about 70-90 fps, the server just couldn't deal with it. Fortunately, the maxfps command was available, so all the playtesters could set their maxfps to 60, allowing the server to recover and we could continue playtesting, finding some extra weird bugs in fringe places (like spectator, first person mode, or commander mode, clicking on minimap glitches out etc).

All in all, looked good and the bug list didn't look too bad. But ... there wasn't a server around that could run 224 even with a 8v8 game. So it was not possible to release it as is.

Solve one problem, create a new one.

After a bit of brainstorming, we tried lowering max fps to 60 as a temporary measure to allow playtesting to continue, but even that soon ran into server problems, with servers that used to be able to hold 30 ticks for 20 players endgame suffering tick-crashes in mid-game.

So we branched out. Max started to dig into how Source handle it, while I started looking into move-merging. Basically, with 100 moves per second generated, each move wouldn't do much - at 5 m/s, you only move 5cm per frame, so it should be possible to merge frames together and only run about 30 moves per second.

We were looking at doing an interim solution to get 224 out quickly, accepting that 224 might suffer a bit of client/server misprediction, then coming up with a better solution for the next patch.

Max and Dushan started looking into server-side move merging, where the server simply accumulates incoming moves and executes them 30 times per second. At first, I had also looked at server-side merging, but I found it unsatisfactory, as I had the choice of either letting it get really sloppy OR spending a lot of CPU to make sure the moves were mergeable - not a good idea if you are trying to SAVE CPU.

After a few days of struggling, I was taking a shower when I had a great idea - the client already had a world that was running moves in parallel to the main world; the prediction world. I should be able to use that to run the moves both as single moves and as merged moves, then compare the end-result and see if the merged move ended up being close enough and trivial enough to allow the merge.

All said and done, I reworked the PredictionWorld, adding support for merging moves as they came in from the Client. So the Client reads inputs, constructs a move, hands over a copy to the Predict/Merge world (checking if there is a fresh server update at the same time), then processes that move and uses the result to render.

At the same time, the Predict/Merge world processes the move in ITS world twice, once as a single move, and once by trying to merge it with the previous move. Callouts to Lua at strategic times allows the Lua code to allow/deny the merge, so the analyses on what is a mergeable move isn't part of the engine, but instead is part of the game.

Performance wise, this would actually be FASTER than not doing the client side merge. Assuming 100ms latency, 100ms interp and 100 fps, you are looking at at running 20 prediction moves for each server update, with the updates coming in 20 times per second for a total of 400 moves (in 223, you would be looking at 20 moves 100 times per second for a total of 2000 moves... - not that you could ever GET to 100 fps in 223 with 100ms latency). Merging those 20 moves into an average of 7 moves means that you would need to run 140 moves. Adding in the about 70 merge moves attempt you can expect to need, you are still coming out about 200 moves ahead. And with each move at 0.5-0.7ms, that's about 120ms or 12% cpu less, in the off-core thread.

A day or two of coding showed it working fairly nicely, with a few weird bugs. Worked plenty fine from first-person perspective, but there was an unforeseen problem - as the player now updated his position only 30 times per second, the player movement smoothing code didn't work well any longer, resulting in other players moving erratically on your screen.

Fortunately, Max had already thought of a solution for low-fps player movements smoothing. Implementing it wasn't really necessary when player update rate was at 40+, but now when suddenly everyone was "low-fps ######es" as a PT said about other playtesters with low fps, Max proceeded to rip apart the network protocol and put it back together again. That's something Max feels comfortable doing in a few hours or so...

Those weird bugs ... they turned out to be pretty difficult to find. Most visible was that sometimes when reloading the pistol and FT, the reloading animation played twice. Sometimes. It turned out that the animation engine is very sensitive to when an animation starts. In the Predict world, the animation engine is used for one thing; controlling the timing of things like attacks and reloads (it's also used to produce BoneCoords to put some spine into models for rendering and physics, but that's not used in prediction).

Analysis showed that starting a new animation had to block merging; merging in any animation with another move really confused things no end. Fortunately, on the client the move is first run on its own, so the Lua code could just save up the prior relevant state of the animation, wait for the move to be run, and make sure that a new animation hadn't started before allowing a merge attempt.

While I had been working with the client side try, Max and Dushan had run into problems with the quick-fix server side merging; the same problems with animation synch and on the server, there was no CPU available to do any analysis of the moves, so mispredictions were large and hard to do anything about. So 224 was not getting out.

This had taken a week (+ weekend) of hard work, but by Wednesday the client-side merge was ready to be playtested, and after some heroic and epically long playtesting (7+ hours) and on-the-fly bugfixing by UWE, 224 was ready to be released.

So in the end, 224 brings you three insane upgrades in one package; both a new client multiworld/multithreaded system, an innovative dynamic merged move system and a new network protocol. Net effect should be about +40% client side fps (with the side effect of latency no longer lowering your fps), +20% server side player count and silky smooth player movement.

Enjoy!


Member of CDT, Senior Spark Engine Hacker

Comments

  • IndustryIndustry Esteemed Gentleman Members, Reinforced - Shadow, WC 2013 - Supporter Join Date: 2010-07-13 Member: 72344Posts: 698
    So uh.. you seeing anyone? /swoon
    THE IP ADDRESS IS COMING FROM INSIDE THE HOUSE.
  • GlissGliss Members, Constellation, NS2 Map Tester Join Date: 2003-03-23 Member: 14800Posts: 5,161 Fully active user
    QUOTE (matso @ Oct 25 2012, 09:19 AM) »
    After a few days of struggling, I was taking a shower when I had a great idea

    the birth of all great ideas
    Q: Is your goal to make everything picked the same amount of times? (from j.walsh)
    A: I don't think about it that way exactly. Sure it's good if there is a nice distribution and that is something to work towards, but really the goal in DotA's development is, first and foremost, to make the gameplay quality the best it can be. It is the combination of all the heroes, items and mechanics together that makes DotA what it is and not the individual elements alone. Sometimes that means certain things are carefully adjusted to be more frequent than others, sometimes it doesn't, but ultimately the main judge of a game's quality is not in some artificial balance metric, but in the overall gameplay quality and depth that it has.
  • VeNeMVeNeM Members Join Date: 2002-07-13 Member: 928Posts: 1,240 Fully active user
    QUOTE (Princess_of_Power @ Oct 25 2012, 12:27 PM) »
    the birth of all great ideas

    i thought that was hanging clocks over toilets
    Lethal Assassin
    user posted image
  • extolloextollo Ping Blip Members Join Date: 2010-07-16 Member: 72457Posts: 368 Advanced user
    sounds like a good topic for GDC 2013. :)
  • tk-421tk-421 Members, Reinforced - Shadow Join Date: 2006-11-03 Member: 58315Posts: 191
    Anyone see Carmack's last keynote at QuakeCon? That's how I picture this story being told...
  • SecuritySecurity Members, Constellation, Squad Five Blue Join Date: 2005-01-07 Member: 33133Posts: 279 Fully active user
    Very nice. Thanks for the insight!

    Lets see how this works out. :]
  • antacidantacid Members, NS2 Playtester Join Date: 2007-08-07 Member: 61821Posts: 483 Advanced user
    edited October 2012
    That is the one thing I love and hate about programming, the absolute need to basically invent new solutions to problems.

    Love it when it happens and it works.

    Hate it when it happens and only causes more problems.
  • Shrike3OShrike3O Members, Constellation Join Date: 2002-11-03 Member: 6678Posts: 718
  • ZekZek Members, NS1 Playtester, Constellation, Reinforced - Shadow Join Date: 2002-11-10 Member: 7962Posts: 4,690 Advanced user
    This is why I stick to web development :o Keep up the good work, you guys are rock stars.
  • falcfalc Members Join Date: 2011-03-18 Member: 87128Posts: 133
    Great work guys, at least the server performance looks much much much much much much :x more stable now (couldn't play the new build myself yet).

    I'm proud of you, the game and being part of this community - Wish you all the best for release (get some sleep afterwards)! ;)
    FRAGFEST.de | Steam Group | Twitter
    A german website and blog about games and stuff.
    NS2 Servers (i7 @ 3.8 GHz)
    Server #1 - 176.9.107.253:27015
    Server #2 - 176.9.107.253:27025
  • Laosh'RaLaosh'Ra Members Join Date: 2011-12-09 Member: 137232Posts: 905 Fully active user
    running significantly better for me now, thanks for the awesome work! :)
  • countbasiecountbasie Members Join Date: 2008-12-27 Member: 65884Posts: 824
    Thanks for these informations, I really appreciate how you Devs involve us in that kind of stuff. It's inspiring.

    No client FPS increase for me in this build whatsoever :p
    I hope there's some room left for more improvements client side.
  • Onii-chanOnii-chan Members Join Date: 2002-11-05 Member: 7164Posts: 503
    5 steps forward, 2 steps back.
    That's progress for ya.
    It's not like I want to gore you or anything!
  • TimMcTimMc Members Join Date: 2012-02-06 Member: 143945Posts: 1,425
    Kind of lost track what was going on at the end there, but I'm a web developer. Whats important is the fps gain, which is really nice. :) Thanks!
    My Steam Workshop:
    Arms Limitation Treaty - Whips, Sentries, ARCs and Hallucinations count towards MAC/Drifter limits.
    No Power Usage - Marine structures no longer require power. Power nodes only affect lighting, but are much weaker.
  • A_PajanderA_Pajander Members, Reinforced - Shadow Join Date: 2002-12-31 Member: 11695Posts: 172 Fully active user
    Pardon my ignorance, but which one is matso? :)
  • GISPGISP Battle Gorge DenmarkMembers, Playtest Lead, Forum Moderators, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, NS2 Map Tester, Reinforced - Onos, WC 2013 - Gold, Subnautica Playtester, Forum staff Join Date: 2004-03-20 Member: 27460Posts: 1,232 mod
    You guys have done AMAZING work optimizing.
    This is why i first became a playtester, helping crushing down the bugs and have a direct impact on its devolopment.
    And im glad that my pleas, constant whining and cheering about every single thing, that have raised or lowered my fps by 1, have been taken into account. So i, on my lowend machine can now enjoy games whit 15-40 fps, and a positiv KD ratio.
    Over the last few months, i have gone from a max of 15 fps to now seeing 50+ in the RR. Absolutly fantastic job!
    Forum Mod - UWE server admin - Have questions about SN or NS2? Need help promoting SN or NS2? Found a bug, or exploit? Or just want to say hi. PM me.
    Become a NS2 Playtester - Become a SN playtester - Buy NS2 here - Buy SN here - Official NS2 stream - SN Wiki
    - NS2 Wiki A Rainbow!
  • blinblin Members Join Date: 2011-07-20 Member: 111290Posts: 148
    feels like I just read a prequel to the Matrix.

    The Birth of the Architect.
  • VitdomVitdom Members, Reinforced - Supporter, Reinforced - Silver, Reinforced - Gold, Reinforced - Diamond, Reinforced - Shadow, WC 2013 - Shadow Join Date: 2012-04-30 Member: 151345Posts: 285 Advanced user
    edited October 2012
    Wow nice!

    QUOTE (Vitdom @ Oct 25 2012, 02:58 PM) »
    ... Wouldn't it just be simpler and faster to have both the server and client running state update simulations/predictions, like the server frame updates synchronizes the client-server states and the S/C just keep on simulating/predicting the game sending synchronize data back and forth, instead of keeping a list of updates-not-part-of-the-latest-server-update and just predicting what will happen over and over again? If both the server and client has access to the same game state data (which it has), there would be no additional downsides, coming with the improvements. Why would it according to you be a "hitchy experience" when that way of doing things can be implemented in a very nice and well-working way?

    Doesn't the Source engine work like that?

    I just predicted a major part of "224 tech changes, part 2" and the network changes you made. xD
    Steam profile
    Hyperdata for breakfast. Hyperdata for lunch. Hyperdata for dinner. Hyperdata for supper. Hyperdata as snacks!
    Dream hyperdata! Speak hyperdata! Live hyperdata! Be hyperdata!


    Nothing like the smell of distress bacon and egg lock in the morning! -- KwisatzHaderach
Sign In or Register to comment.