GameOvermind

player

A little bit about performance:

I've found performance to be at an absolutely unacceptable level, if the gamestate-recorder would be used on an extremely powerful system like endar's 5GHz PureAus, the tick-rate would still be in the single digits (<10). I concluded the main reason for this to be Luabind, the overhead it adds is immense. As you may know Max turned away from using Luabind quite some builds ago (around 171), but I continued to use it for GameOvermind, as the interaction between the game and the mod wasn't that significant, so I could get away with it. However v11 marks the turning point in that, the mod is now so deeply embedded into the game, the overhead Luabind introduces is simply killing the server.

To quantify, I've replaced a tiny bit of Luabind-code in a critical area of the mod with 'hand-rolled' Lua C-API calls, which yielded a good 10-15% performance increase right off the bat. So I've pretty much decided to do away with Luabind entirely and use manual C-API calls everywhere (maybe with very slight abstraction where the end-result will be identical). Additionally, I'm going to restrict the type-checking\value-evaluation and alike to the debug-build, because these also put load on the Lua-VM. These checks is what helped v10 of the mod to function as long as it did (and still does), and they certainly were justified at the time with NS2 being as unstable\unreliable as it used to be (I've seen some crazy ###### in those days). Now with NS2 slated for a summer-release, and the general stability improving, I think it should now be possible to depend on the engine\Lua-script to pass and accept correct values without further analysis on my part.

In practice this means that even a slight mismatch (say a differing function-prototype) will lead to a complete server-crash, whereas in v10 and prior that part of code would simply be non-functional. It's a tough decision, but I really need every bit of performance I can get (the gamestate-recorder is a rather heavy beast).

endar

Interesting read, I like progress updates that are wordy. Is most of whats above in reference to the recorder? I imagine for such a mammoth task (or what I imagine is a mammoth task) that you would have been focusing a lot more on this than the.. I guess core parts of GMOvermind.

What sort of performance impact does this have on a server not running any recording, is it still negligible like it is in V10?

player

Most of the new version's fundamentals were designed around the gamestate-recorder, as a result it is very much embedded into the game (keeping track of all entity activity\sounds\effects\networking-changes et cetera).

I indeed will make it turn off that infrastructure if the recording-functionality isn't used, and will try to make it use as little power as possible (I should be able to make it more efficient than v10, which uses Luabind still) to accommodate for servers who just want the barebone admin-functions.

endar

I think its only relevant in the short term, while performance is still so important. As release gets closer, overheads will become more manageable as the game code becomes more optimized.

I hope.

endar

Feature inquiry/request:

Is it possible to add notes to a player (steamID)? I'm thinking of more of a warning system for players who troll. If they alias and constantly change names, they can basically skip out on a ban, but if I were to om_players and see that ive already posted some notes about that player, I can save myself some time.

player

I could add a "Notes"-field for users, so you could add a griefer as a user, grant him guest-priviliges (the default for plebs), and add a note to it explaining. This way I won't have to add any new mechanisms, and can simply make use of the existing storage-system already in place for users.

endar

That sounds sufficient.

ScardyBob

So this might already be fixed for OMv11, but in OMv10 using the om_endround command lead to this error and a server crash:
<!--c1--><div class='codetop'>CODE</div><div class='codemain'><!--ec1-->Script Error: lua/NS2Gamerules.lua:798: attempt to call method 'GetTeamType' (a nil value)
    ConsoleCommand [C]:-1
    ConsoleCommand [string "function NS2GmOvrmind.Lua.ConsoleCommandLua
    [string "function NS2GmOvrmind.Lua.ConsoleCommand_HE..."]:138<!--c2--></div><!--ec2-->

player

I tried the command and it indeed threw some error-garbage in my face, but the command still functioned (the round was ended), and the server did not crash. Maybe you could run the command again to see if it indeed crashes the server, or that it was just a coincidence.

Though either way, it's not as if I will be putting out a fix for it, I'm too far ahead with v11 to prop up v10 at this point. Fingers crossed that it won't fall apart completely before v11.

ScardyBob

<!--quoteo(post=1900946:date=Feb 8 2012, 03:37 AM:name=player)--><div class='quotetop'>QUOTE (player @ Feb 8 2012, 03:37 AM) <a href="index.php?act=findpost&pid=1900946"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->I tried the command and it indeed threw some error-garbage in my face, but the command still functioned (the round was ended), and the server did not crash. Maybe you could run the command again to see if it indeed crashes the server, or that it was just a coincidence.

Though either way, it's not as if I will be putting out a fix for it, I'm too far ahead with v11 to prop up v10 at this point. Fingers crossed that it won't fall apart completely before v11.<!--QuoteEnd--></div><!--QuoteEEnd-->
It might have been a coincidence (the server was getting chunky at that point), but I'll see if I can reproduce. I also didn't expect a fix, but just wanted to let you know in case its something that may carry over into v11.

ScardyBob

So I just starting looking at some good server data collected via devicenull's automatic update mod and its recording impossible tickrates outputted from overmind (i.e. >35 and up to 48!). I was wondering how OMv10 determines the tickrate (i.e. it is just pulling it from the server or is it calculating it separately)?

endar

I've seen it go above 30 (up to 31 is kind of normal), but every time its done that, i think its been in an error state. The server has most likely crashed or failed to load something major and never works.

Were people playing during this time? I'd be curios to know if they had a tickrate from net_stats.

player

It sits in the game-loop and increments everytime it gets called, so if it says something like 40-45, it really was called 40-45 times the last second. But like endar noted, odd tick-rates often occur when the server is either empty or a phantom. As soon as players join it, the tick-rate normalizes to 30 again.

dePARA

Everytime i connect to my root over remotedesktop, my ns2-server crash. This problem is up for a while (dont know when it appears first time) but today i want to know whats going on there.

It seems the QtCore4.dll crashing the server. Here is the log (sorry its german, but maybe it helps)


Name der fehlerhaften Anwendung: server.exe, Version: 0.0.0.0, Zeitstempel: 0x4f3381a9
Name des fehlerhaften Moduls: QtCore4.dll, Version: 4.7.2.0, Zeitstempel: 0x4d83a805
Ausnahmecode: 0xc0000005
Fehleroffset: 0x001d1dc6
ID des fehlerhaften Prozesses: 0xea4
Startzeit der fehlerhaften Anwendung: 0x01cce7fdf9d453eb
Pfad der fehlerhaften Anwendung: C:\natural selection 2\server.exe
Pfad des fehlerhaften Moduls: C:\natural selection 2\QtCore4.dll
Berichtskennung: 889b6a88-5406-11e1-8ed5-f46d04736bff
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2012-02-10T16:44:44.000000000Z" />
<EventRecordID>2799</EventRecordID>
<Channel>Application</Channel>
<Computer>WIN-7C7PDFLJHDF</Computer>
<Security />
</System>
<EventData>
<Data>server.exe</Data>
<Data>0.0.0.0</Data>
<Data>4f3381a9</Data>
<Data>QtCore4.dll</Data>
<Data>4.7.2.0</Data>
<Data>4d83a805</Data>
<Data>c0000005</Data>
<Data>001d1dc6</Data>
<Data>ea4</Data>
<Data>01cce7fdf9d453eb</Data>
<Data>C:\natural selection 2\server.exe</Data>
<Data>C:\natural selection 2\QtCore4.dll</Data>
<Data>889b6a88-5406-11e1-8ed5-f46d04736bff</Data>
</EventData>
</Event>

ScardyBob

<!--quoteo(post=1901629:date=Feb 9 2012, 11:55 PM:name=player)--><div class='quotetop'>QUOTE (player @ Feb 9 2012, 11:55 PM) <a href="index.php?act=findpost&pid=1901629"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->It sits in the game-loop and increments everytime it gets called, so if it says something like 40-45, it really was called 40-45 times the last second. But like endar noted, odd tick-rates often occur when the server is either empty or a phantom. As soon as players join it, the tick-rate normalizes to 30 again.<!--QuoteEnd--></div><!--QuoteEEnd-->
Ah, ok, that looks to be the case for the above 30 tickrates. Here's a representative snippet of the log with that problem
<!--c1--><div class='codetop'>CODE</div><div class='codemain'><!--ec1-->09-10 02:52 root         DEBUG    CPU usage: 11, memory: 567 MB, players: 0 tickrate: 56
09-10 02:53 root         DEBUG    CPU usage: 6, memory: 567 MB, players: 0 tickrate: 56
09-10 02:54 root         DEBUG    CPU usage: 9, memory: 567 MB, players: 0 tickrate: 51
09-10 02:55 root         DEBUG    CPU usage: 7, memory: 567 MB, players: 0 tickrate: 56
09-10 02:56 root         DEBUG    CPU usage: 7, memory: 567 MB, players: 0 tickrate: 54<!--c2--></div><!--ec2-->
Though, I'll search through it some more to see if there are cases where players >0 and tickrate >30.

player

<!--quoteo(post=1901840:date=Feb 10 2012, 05:05 PM:name=dePARA)--><div class='quotetop'>QUOTE (dePARA @ Feb 10 2012, 05:05 PM) <a href="index.php?act=findpost&pid=1901840"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->Everytime i connect to my root over remotedesktop, my ns2-server crash. This problem is up for a while (dont know when it appears first time) but today i want to know whats going on there.

It seems the QtCore4.dll crashing the server. Here is the log (sorry its german, but maybe it helps)<!--QuoteEnd--></div><!--QuoteEEnd-->
Very odd, I don't see how an access-violation could be caused by something so unrelated. Anyway, it was probably sloppy coding on my part, v10 was very much a test-mod and lacked proper coding-conventions and runtime-safety (on the C++-side, Lua was very much boxed in). Based on the information you gave I can't do much, but I can tell you it's a lot less likely something like access-violations will happen in v11. Any idea when it started happening at all? v10 has been out for ages, so if you changed something in the mean time, that might be it. I don't have access to the exact source-code of v10 anymore (it's more v10.1 I have archived), else I might've been able to do more.

<!--quoteo(post=0:date=:name=ScardyBob)--><div class='quotetop'>QUOTE (ScardyBob)</div><div class='quotemain'><!--quotec-->Though, I'll search through it some more to see if there are cases where players >0 and tickrate >30.<!--QuoteEnd--></div><!--QuoteEEnd-->
I'm willing to bet there won't be (phantom-servers notwithstanding).

ScardyBob

So I spent a big more time searching and sorting the log and did find some >30 tickrate, >0 player selections. The most outrageous (i.e. full server, high tickrate) is below:
<!--c1--><div class='codetop'>CODE</div><div class='codemain'><!--ec1-->10-13 12:33 root         DEBUG    CPU usage: 93, memory: 629 MB, players: 17 tickrate: 50
10-13 12:34 root         DEBUG    CPU usage: 99, memory: 630 MB, players: 18 tickrate: 31
10-13 12:35 root         DEBUG    CPU usage: 100, memory: 630 MB, players: 18 tickrate: 47
10-13 12:36 root         DEBUG    CPU usage: 96, memory: 630 MB, players: 18 tickrate: 31
10-13 12:37 root         DEBUG    CPU usage: 98, memory: 630 MB, players: 18 tickrate: 50
10-13 12:38 root         DEBUG    Checking for server update...
10-13 12:38 root         DEBUG    CPU usage: 98, memory: 633 MB, players: 18 tickrate: 50
10-13 12:40 root         DEBUG    CPU usage: 100, memory: 635 MB, players: 17 tickrate: 50
10-13 12:41 root         DEBUG    CPU usage: 96, memory: 635 MB, players: 18 tickrate: 50
10-13 12:42 root         DEBUG    CPU usage: 99, memory: 636 MB, players: 18 tickrate: 49
10-13 12:43 root         DEBUG    CPU usage: 95, memory: 637 MB, players: 18 tickrate: 48
10-13 12:44 root         DEBUG    Checking for server update...
10-13 12:44 root         DEBUG    CPU usage: 100, memory: 637 MB, players: 18 tickrate: 50
10-13 12:45 root         DEBUG    CPU usage: 98, memory: 637 MB, players: 18 tickrate: 51
10-13 12:46 root         DEBUG    CPU usage: 95, memory: 636 MB, players: 17 tickrate: 47
10-13 12:47 root         DEBUG    CPU usage: 95, memory: 637 MB, players: 18 tickrate: 49
10-13 12:48 root         DEBUG    CPU usage: 98, memory: 637 MB, players: 18 tickrate: 50<!--c2--></div><!--ec2-->

These truly ridiculous numbers seem to have been limited to the B185-189 patches, so its a bit of a moot point now. However, I am concerned with getting some >30 tickrate with >0 player data in B194+.
<!--c1--><div class='codetop'>CODE</div><div class='codemain'><!--ec1-->02-06 00:49 root         DEBUG    CPU usage: 95, memory: 776 MB, players: 14 tickrate: 32
02-06 00:50 root         DEBUG    CPU usage: 93, memory: 776 MB, players: 16 tickrate: 33
02-06 00:51 root         DEBUG    CPU usage: 98, memory: 776 MB, players: 16 tickrate: 33
02-06 00:52 root         DEBUG    CPU usage: 99, memory: 776 MB, players: 15 tickrate: 33
02-06 00:53 root         DEBUG    CPU usage: 96, memory: 776 MB, players: 15 tickrate: 34<!--c2--></div><!--ec2-->

So my issue with this is that I'm wanting to use tickrate as a proxy for server performance so I can compare whether and how much its improved over the builds. If the tickrate can and does regularly go above the supposed max of 30, it makes this much harder. Got any idea why the tickrate would go above 30 in the latest builds?

player

Ahah, looks like a bug\feature? in a number of builds. I went back and checked what it did exactly to calculate it, and it really was as straight-forward as counting update-events, so I guess a 50 tick-rate really did happen in those builds. Something like 30-34 seems a lot more reasonable. Maybe it has something to do with the engine's pacing, and that it sometimes doesn't wait long enough before running the next tick, resulting in a slight variation. I do know that in-game, using net_stats, you can sometimes see the tick-rate spike to 31-32 after doing some heavy-work, so I'm guessing it may just be the engine trying to find equilibrium.

On a related note, I think it might be about time for the tick-rate to be unlocked to server-operators, this 30-ceiling (which isn't always 30 apparently, it also tends to drop to 1 when the server is empty) seems a bit out of place. If the game is to be outted in the summer, new experiences with strongly varying tick-rates are important.

Soul_Rider

How does tick affect NS2, as opposed to say CS:S. For example, you could really feel the reg difference between 33-66, not so much between 66-100

How different are the workload and update requirements in NS2 to a game like CS:S. I mean, if we were able to have server capable of performing 66tick, would it make any difference to the game? If so in what way? I am fairly certain a 66 tick server in the current build would probably kill the machine hosting it, btw :)

player

Well, a major complaint is the performance-impact the prediction-code has on clients when playing on a low tick-rate server. If the server gives out updates much more frequently, it'll save the client quite a bit of computing-work, which many folks will be thankful for. But as you noted, servers have a hard enough time to sustain a tick-rate of even 30, let alone 60 or even 100.

In terms of reg it kinda depends, netcode these days is intelligent enough to acknowledge a hit on the client as a hit on the server (within valid boundaries), so the difference wouldn't be day and night, but it'll increase the chances of a server approving a hit, as well as prevent prediction-errors causing spassed out animations.

endar

<!--quoteo(post=1901840:date=Feb 11 2012, 04:05 AM:name=dePARA)--><div class='quotetop'>QUOTE (dePARA @ Feb 11 2012, 04:05 AM) <a href="index.php?act=findpost&pid=1901840"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->Everytime i connect to my root over remotedesktop, my ns2-server crash. This problem is up for a while (dont know when it appears first time) but today i want to know whats going on there.
[log]<!--QuoteEnd--></div><!--QuoteEEnd-->


That does sound quite annoying. Are you saying it happens every time you connect, literally 100% of the time? What OS do you run? And I suppose it should be mentioned (since you didnt) does it happen when GM Overmind isnt running.

Floodinator

Is it possible that GMO can force a client to take a screenshot and upload it in a directory in the GMO directory? Would be interesting for admins. They could manually force a suspect player client to do a screenie to check for modded things like better flashlight, different models etc...

It would be a simple anticheat messure.

player

Nope. At this time there is no server-client synchronization of script-files, so there's pitifully little interaction I can do with clients, but even if there was, the Lua-sandbox offers no access to the render-pipeline, including the taking of screenshots (so UWE would have put in a screenshot-feature). The Lua-sandbox also offers no way of transporting said screenshots back to the server (that may be mitigated by allowing networking in the sandbox, and manually create a TCP-stream to upload it to the server, it's not brilliant, but possible somewhat).

Modded flashlights should be solved with the upcoming Lua-script hash-checks, as for modded content (models\textures), I'm assuming some kind of sv_consistency or even a similar set-up as the Lua-scripts (a server-side hash-evaluation) will be available.

[EDIT]
Actually I may be able to sneak screenshots through the data-stream of the NS2-client, probably won't be spiffingly fast but still.

dePARA

We have a new server since 1 week and there is excact the same issue that i postet before.

Server.exe crash at rdp-connect.
Our 2nd server without overmind didnt crash.

QtCore4.dll is involved.

MOOtant

<!--quoteo(post=1892874:date=Jan 6 2012, 06:56 PM:name=player)--><div class='quotetop'>QUOTE (player @ Jan 6 2012, 06:56 PM) <a href="index.php?act=findpost&pid=1892874"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->No it wasn't asked as such, but there are several good reasons for that, chief amongst them are that this mod isn't really Lua. When Server-to-Client synchronization is put in I'm sure I'll do some more features in Lua-script for that occasion, but as it stands it is virtually completely written in C++, and it will remain that way in the future. The reason this is an issue is because (I believe) NS2 was built with a moddable open-source Lua-script mantra in mind, and GameOvermind goes against that in large ways.

But truth be told I wouldn't have it any other way. Once it becomes a part of the official package, a mod loses a lot of its freedom (in terms of script having to be thoroughly checked\approved, not being able to easily modify certain things without some kind of collaboration with the rest of the scripters\dev-team). Things go more slowly\regulated in a professional environment, and with good reason.

Whilst I can simply code some spiffy new feature and throw it over the wall for you to try, and it's that freedom\casual-attitude I'm very comfortable with and allows for excellent productivity. I'll admit the downside here is that I may run into areas I can't get past without UWE's assistance, but I've been able to work around these issues so far (with increasingly creative\hack-ish approaches).<!--QuoteEnd--></div><!--QuoteEEnd-->
Let's be honest. That's bull######. Haskell packages (of all different sizes) are shared on github most of the time. Whenever someone wants to make a tiny change he clones and changes the code. If it happens to be a fix he makes pull request and boom all have the fix. Haskell "community" is more alive than ever thanks to that efficient sharing of code. Separate program downloads package to dev that's interested in last stable version

For instance Snap and Yesod frameworks completely exploded after people started making all these tiny changes to it adding what they needed.

player

Your point would be more valid if we were talking about an established code-base, but this is far from the case. Every day I make changes that break the code in many locations all over (something you may be less inclined to do if other people work on the same package), as well as doing constant changes in design that would throw some serious curve-balls if other people aren't informed. To put it differently, I'm experimenting a lot with the mod as a whole in trying to see how to get certain things to work properly. Right now for example, I'm tearing out Luabind completely and replacing it with hand-rolled C-API calls. As a result, the code-base is in shambles, with not much of anything working. It goes beyond changing a feature here or there, but involves everything at a fundamental level. Forking and then merging at a later time would plainly not work for obvious reasons (wholly different designs don't merge well). This is very much an ongoing experiment.

<!--quoteo(post=0:date=:name=dePARA)--><div class='quotetop'>QUOTE (dePARA)</div><div class='quotemain'><!--quotec-->We have a new server since 1 week and there is excact the same issue that i postet before.

Server.exe crash at rdp-connect.
Our 2nd server without overmind didnt crash.

QtCore4.dll is involved.<!--QuoteEnd--></div><!--QuoteEEnd-->
I can think of a few things that have been broken over the months, that may be causing this problem, but there isn't much I can do about it at this point. I suggest you sit it out until the new version is released.

player

<!--quoteo(post=1904300:date=Feb 17 2012, 06:05 PM:name=dePARA)--><div class='quotetop'>QUOTE (dePARA @ Feb 17 2012, 06:05 PM) <a href="index.php?act=findpost&pid=1904300"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->We have a new server since 1 week and there is excact the same issue that i postet before.

Server.exe crash at rdp-connect.
Our 2nd server without overmind didnt crash.

QtCore4.dll is involved.<!--QuoteEnd--></div><!--QuoteEEnd-->
Actually, I've thought about this a bit more, can you recall if this started happening at NS2 B191 (so since Jan 13th)?

MOOtant

I have another question: I'm thinking about writing some bits of my own admin mod and I wonder if I understand how you connected yours to NS2. Do you just loadlib() your DLL then make a call into its init() that replaces NS2's major functions with ones that do Overmind stuff and then call real NS2?

If so then what is the normal NS2's entry point?

player

The scripts are hierarchical in that they have to be included by a higher-level script, consequently there has to be a single script all the way at the top that initiates the whole chain, that script is defined in Game_Setup.xml (the server and client have a different top-level script). I supply my own script to be loaded, in it initialize the mod and place any hooks that need to be placed before NS2's 'normal' scripts are loaded, then I load the 'normal' top-level script from within the init-procedure, after which I can place additional hooks and overloads that need to occur after NS2's 'normal' scripts have been loaded. Then the init-procedure ends, control returns to the engine, and the server is running.

endar

I've just experienced the errors of server crashing when opening an RDP connection. For me, this has only started happening since B196 (just happened in B197 also). I run three separate servers under different user accounts, and they do not affect each other. It would be a shame for this to be the long awaited end to V10, anything we can do to help?

Faulting application name: server.exe, version: 0.0.0.0, time stamp: 0x4f3eb3a7
Faulting module name: QtCore4.dll, version: 4.7.2.0, time stamp: 0x4d83a805
Exception code: 0xc0000005
Fault offset: 0x001d1dc6
Faulting process id: 0x12b4
Faulting application start time: 0x01ccedcdc03e15a4
Faulting application path: C:\NS2Game\ns2f\server.exe
Faulting module path: C:\NS2Game\ns2f\QtCore4.dll
Report Id: 1520e3d3-59ce-11e1-8ceb-14dae929893c