GameOvermind

11213141618

Comments

  • OmNomNom!OmNomNom! Join Date: 2010-11-25 Member: 75266Members Posts: 33
    Still getting the below error when i'm running a server on a dedicated 2008 Server R2 box.
    No idea what causes it. The server can be fine for hours, but this probably happens a couple of times a day

    Faulting application name: server.exe, version: 0.0.0.0, time stamp: 0x4f6c95c0
    Faulting module name: QtCore4.dll_unloaded, version: 0.0.0.0, time stamp: 0x4d83a805
    Exception code: 0xc0000005
    Fault offset: 0x70130db0
    Faulting process id: 0x1260
    Faulting application start time: 0x01cd09d00546b735
    Faulting application path: C:\HLServer\NS2\server.exe
    Faulting module path: QtCore4.dll
    Report Id: de2c7d91-7605-11e1-9dff-1c6f65ce618c

  • playerplayer Join Date: 2010-09-12 Member: 73982Members Posts: 1,677
    Well what we do know is that it is possible for the (vanilla) server to crash all on its own, so there's that issue that clouds any diagnostics on GameOvermind. Also, the _unloaded bit significies that the problem occurred when it was (surprise) unloading the DLL, this means that the server was already on its way out so to speak (already crashed due to something else), and simply stumbled over GameOvermind not deinitializing properly. Certain exceptions (very well thrown by a problem with NS2) can cause the stack to unwind abruptly when GameOvermind isn't expecting nor able to deal with it.

    So I'm thinking as UWE keeps working on the game, this error might vanish all by itself. But I have taken note of the issue, and am spending quite a bit of time making the rewrite very much error-proof (like not having it throw exceptions around, ever).
  • OmNomNom!OmNomNom! Join Date: 2010-11-25 Member: 75266Members Posts: 33
    edited March 2012
    QUOTE (player @ Mar 25 2012, 12:15 PM) »
    Well what we do know is that it is possible for the (vanilla) server to crash all on its own, so there's that issue that clouds any diagnostics on GameOvermind. Also, the _unloaded bit significies that the problem occurred when it was (surprise) unloading the DLL, this means that the server was already on its way out so to speak (already crashed due to something else), and simply stumbled over GameOvermind not deinitializing properly. Certain exceptions (very well thrown by a problem with NS2) can cause the stack to unwind abruptly when GameOvermind isn't expecting nor able to deal with it.

    So I'm thinking as UWE keeps working on the game, this error might vanish all by itself. But I have taken note of the issue, and am spending quite a bit of time making the rewrite very much error-proof (like not having it throw exceptions around, ever).


    EDIT: My apologies, it seems like the server crashes were very likely due to system instability after all.
    Post edited by Unknown User on
  • ArgathorArgathor Join Date: 2011-07-18 Member: 110942Members, Squad Five Blue Posts: 682
    QUOTE (endar @ Mar 23 2012, 11:24 PM) »
    Thanks! Unfortunately i wont be able to have this updated for a few hours, im about to go out. Sorry to the Australian players.

    This isnt the first time that a new build has been incompatible with a old version of GM. There is nothing that player can do about it, I understand that, but why is there still NO playtesting without the mod?

    Put simply every real server uses this mod, but none of the playtesting does. It may not be UWE's problem, but it sure does look bad when all of the servers people play on dont work after a new patch. You get the idea.

    I second this, it doesn't make any sense. The mod is invaluable to server admin's and by the looks of it will be long past release. Perhaps it is time to begin incorporating it into the development cycle to ensure the best experience for admins and players alike.
    Arga | ns2hub.com
    "Someone's sitting in the shade today because someone planted a tree a long time ago."
  • BJHBnade_spammerBJHBnade_spammer Join Date: 2005-02-25 Member: 42431Members Posts: 464
    edited March 2012
    the reason they do that is for the reason that you guys mentioned so they dont get problems that arent related with the actual game.

    but i agree after they are done with testing for that build they should do testing with overmind just so they know if it will work properly with it

    my guess is that enough servers use it that we will find the bugs and let them know/ let player know about any problems
    and that is there reasoning behind why they dont test it
  • OmNomNom!OmNomNom! Join Date: 2010-11-25 Member: 75266Members Posts: 33
    Okay apologies for bringing this up again but I am still experiencing server instability. It is an overclocked server and I was thinking it might be because of that but after running many hours of stability tests without any errors i'm not so sure. The server is purely dedicated to running NS2 at the moment so I'm quite sure that it is nothing else interfering with the NS2 process.

    The game server is usually fine for hours before it crashes and then I get this error. I get no other errors around the same time as this in the event log so I have no idea what is causing it. I suppose the only way to test for sure is to leave the server running a few days without overmind :(

    Faulting application name: server.exe, version: 0.0.0.0, time stamp: 0x4f6c95c0
    Faulting module name: QtCore4.dll, version: 4.7.2.0, time stamp: 0x4d83a805
    Exception code: 0xc0000005
    Fault offset: 0x00190db0
    Faulting process id: 0xe00
    Faulting application start time: 0x01cd0d06f39b56e5
    Faulting application path: C:\HLServer\NS2\server.exe
    Faulting module path: C:\HLServer\NS2\QtCore4.dll
    Report Id: fbe67d0a-7913-11e1-bd4f-1c6f65ce618c

    Has anyone had a similar problem or can anyone suggest a possible fix or way to diagnose this problem?

    The server is a 2500k @ 4.8ghz 8GB RAM, has run OCCT for 4 hours stable.

    Thanks in advance for any advice guys
  • playerplayer Join Date: 2010-09-12 Member: 73982Members Posts: 1,677
    Run it vanilla, see what happens. It's just bloody hard to tell what's going on exactly, as we don't know whether or not NS2 is stable and reliable (well actually we do, it is not). I'm not sure if vanilla servers can actually run stable for large amounts of time while being stressed. I also don't know the story behind that RDP-issue (was or wasn't it something with NS2? and is it still going on?). Then there's the issue with the memory-leaks in NS2. That has been an ongoing for ages, and I still don't know if servers can run long enough without leaking memory and eventually crashing because of it (which may be happening to you, any idea how much memory was consumed just before the crash?). Devicenull has a mod to auto-restart it regularly, which mitigates that issue somewhat.

    We really need a seperate forum-section dedicated to server-operators, it's a completely jungle out there for whoever runs a server right now.

    Anyway, to close:
    I've about finished with the user\ban storage-abstraction, which constitutes the better half of 2 months of work, so that's something. Light at the end of the tunnel and such. No release-date\guestimates though!
  • OmNomNom!OmNomNom! Join Date: 2010-11-25 Member: 75266Members Posts: 33
    QUOTE (player @ Mar 29 2012, 03:46 PM) »
    Run it vanilla, see what happens. It's just bloody hard to tell what's going on exactly, as we don't know whether or not NS2 is stable and reliable (well actually we do, it is not). I'm not sure if vanilla servers can actually run stable for large amounts of time while being stressed. I also don't know the story behind that RDP-issue (was or wasn't it something with NS2? and is it still going on?). Then there's the issue with the memory-leaks in NS2. That has been an ongoing for ages, and I still don't know if servers can run long enough without leaking memory and eventually crashing because of it (which may be happening to you, any idea how much memory was consumed just before the crash?). Devicenull has a mod to auto-restart it regularly, which mitigates that issue somewhat.

    We really need a seperate forum-section dedicated to server-operators, it's a completely jungle out there for whoever runs a server right now.

    Anyway, to close:
    I've about finished with the user\ban storage-abstraction, which constitutes the better half of 2 months of work, so that's something. Light at the end of the tunnel and such. No release-date\guestimates though!


    That mod looks awesome, thanks for the heads up.
    Also, thanks for making this great mod. Without you a lot of server admins would be left in the lurch.

    Whenever I looked the server was fine, never using more than 600mb-1200mb although it has 8GB to play with so I doubt it'd run out before crashing for another reason. Also afaik it has never been a crash due to one of the common script errors (where you can see the server console getting spammed to hell). As for RDP, i tend not to use RDP. I use teamviewer instead and it does not tend to crash 'as i log in' for instance.

    Anyway, I'll shut up for now until I can get some more useful feedback. I'll also run it vanilla to see if i can get it to crash like that.

    Keep up the good work.
  • BJHBnade_spammerBJHBnade_spammer Join Date: 2005-02-25 Member: 42431Members Posts: 464
    dameware is another good program to use if you dont want to use rdp.
    i would suggest that server admins use something other than rdp as there is too much potential for problems
    direct access to the desktop is always better anyways
  • ScardyBobScardyBob ScardyBob Join Date: 2009-11-25 Member: 69528Forum Admins, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow Posts: 4,983 mod
    Any plans on including an AFK kicker in OM? AFK players has become a bit of a problem on servers recently, where I've been trying to kick them when I'm on. However, an automated process would be preferred.
  • playerplayer Join Date: 2010-09-12 Member: 73982Members Posts: 1,677
    Yea that's been on the todo-list for a while. I'll go ahead and do that for the rewrite even, it's a pretty simple feature.
  • SkieSkie Skulk Progenitor Join Date: 2003-10-18 Member: 21766Members, NS2 Playtester, Reinforced - Shadow Posts: 656
    Getting more strange QtCore4.dll crashes. I've noticed that the server.exe has crashed twice when I've opened "Libraries" in Windows 7 while a game has been running. Also changing resolutions in NS2 has caused it, while I've not been on the server.
  • playerplayer Join Date: 2010-09-12 Member: 73982Members Posts: 1,677
    Changing resolutions in NS2 crashes the server-process running on the same machine? I don't even... And do you mean "Libraries" as in the folder in windows-explorer? How on earth... Are you pulling my chain?
  • SkieSkie Skulk Progenitor Join Date: 2003-10-18 Member: 21766Members, NS2 Playtester, Reinforced - Shadow Posts: 656
    QUOTE (player @ Apr 3 2012, 05:07 PM) »
    Changing resolutions in NS2 crashes the server-process running on the same machine? I don't even... And do you mean "Libraries" as in the folder in windows-explorer? How on earth... Are you pulling my chain?

    I wish I were, but it's not April 1st anylonger. I had it just happen a third time, that when I opened "Libraries" in Windows 7 on the same machine that's running Gmovermind server, the server.exe crashed with an error in QtCore4.dll.

    It doesn't happen every time. It needs some players on the servers and they must have been playing for a while.
  • ArgathorArgathor Join Date: 2011-07-18 Member: 110942Members, Squad Five Blue Posts: 682
    I don't think its GMovermind related, but using almost any software on my PC (strangely, other than running NS2) crashes the server.

    It means that to host a server for gathers, I need to close every peice of software down on my PC (Skype, VLC, Dropbox, etc) and either play NS or not touch my PC.

    Web browsing seems ok, but just opening Windows Explorer windows to browse folders crashes the server.

    I have crashed far too many gather matches due to checking my downloads folder or opening the skype window!

    Im guessing its simply instability in the Server.exe but if anyone has any idea why it happens or any way to prevent it happening I would love to know!
    Arga | ns2hub.com
    "Someone's sitting in the shade today because someone planted a tree a long time ago."
  • playerplayer Join Date: 2010-09-12 Member: 73982Members Posts: 1,677
    Really? Those things seem so utterly unrelated. Anyway, Skie sent over some footage showing it happen, and I guess if you experienced similar events it may be true, though I'm rather perplexed as to the how and why. Needless to say I won't stop until it is resolved. The first step would be to figure who is causing this issue. The vanilla server-process is known to crash on its own, so right off the bat that clouds diagnostics. Additionally, if it says something like _unloaded, the problem occured only after the server had already crashed (or otherwise panicked) of something else. We have the RDP-issue, which I believe happened on vanilla servers too (and started after Max put in multi-threaded map-loading), anyone sent over a ticket regarding that to the dev-team yet? Has that been fixed in the mean time? Furthermore, I can't phantom GameOvermind crashing when there is activity on the server, and not when there isn't (as internally it doesn't really do anything different when there ARE players on), which makes my spidey-sense tell me that may be NS2.

    The only thing I can say for certain now is that the rewrite will have a lot tighter coding compared to what we have now (escpiaclly in the area of multi-threading, which didn't even exist in the server when I first wrote GameOvermind), so we may be able to better diagnose the problem then. I guess what would be helpful right now is any report of a vanilla server crashing, at least then we can pester the dev-team about it, and narrow down any issues that are specific to GameOvermind.

    And we really really need that bloody server-forumsection.
  • ArgathorArgathor Join Date: 2011-07-18 Member: 110942Members, Squad Five Blue Posts: 682
    It seems crazy to me too, especially simply opening my downloads folder, haha!

    I just tried without GM but could not get it to repeat the behavior (but with only me in the server). I will try and use the server for some gathers tomorrow (without GM) and see if I can repeat the behavior. Is there anything I can do to get more info about why it has crashed?
    Arga | ns2hub.com
    "Someone's sitting in the shade today because someone planted a tree a long time ago."
  • OmNomNom!OmNomNom! Join Date: 2010-11-25 Member: 75266Members Posts: 33
    Yes! This is what is happening to me as well. It was hard to identify at first as the server looked crashed as I logged in via Teamviewer. What seems to be happening is the server was crashing as I did something else on the PC, such as opening an explorer window.

    Shot in the dark but was wondering if it was a graphics driver thing related to bringing the monitor etc out of sleep.

    The problem is, this doesn't seem to be happening every time I log in so it is hard to reproduce.

    Would me posting the event log information help? Its usually QtCore4.dll that is the faulting module, but it was once MSVCR100.dll also.
  • playerplayer Join Date: 2010-09-12 Member: 73982Members Posts: 1,677
    Little tidbit of information to make you forget that I keep pushing the release-date back:

    With regards to the performance of the gamestate-recorder, as it stands currently, the recorder takes about 4ms per 100 entities per snapshot. Now in order to get smooth playback it is required to run the recorder on at least 20 snapshots per second. I should be able to bring this down when Server-Client Lua-sync is put in, because right now timing-issues arise due to the client running on a different clock than the server, causing desynced animations and assorted other stuff. This can be fixed by running some script client-side too, but the way I want to do it demands that both the server and client run the same script, which makes it no longer possible for vanilla-clients to join GameOvermind-servers, so obviously I'm waiting with that until automatic-downloading is put in. Anyway, to abate that timing-issue the server needs to send many snapshots per second to override if you will, any miscalculations the client is making. This point lies around 20, but I can bring that down later on to perhaps 5-10, so that's a performance-improvement waiting on the shelf.

    Now of course the recorder records more than just entities, but entities constitute almost all of the processing-time so I'm basing performance-numbers on that alone. I should also note that there is A LOT of performance-optimizations waiting to be done on those entities too, as right now I'm reading out far more data than I structly need to (most structures for example don't change position\angles after they've first spawned, so I only need to read out that data once, instead of every snapshot). Back to that 4ms, I'm guessing an average 6v6 game can contain up to 600 entities, so we're at 4*6=24ms per snapshot, add to that the 20 snapshots required for smooth playback, it brings us to 480ms per second (or about half of every second simply). So consider your server is able to run at 60 ticks (assuming there is no 30-limit), running the gamestate-recorder at 20 snaps would bring that down to 31.

    So the first release may not run fast enough to be used in matches just yet, but subsequent releases should see significant improvements. That, and of course it would be nice if NS2 itself wouldn't be as taxing as it is now. I'm guestimating the release-date of the rewrite (or a beta anyway) at late may. How things go after the release kinda depends on the performance-development of NS2, as I have to say the motivation to keep pouring time into this project is really disappating due to the terrible terrible performance of both the client, but more importantly the server. And I'd hate to spend time on a lost cause.
  • playerplayer Join Date: 2010-09-12 Member: 73982Members Posts: 1,677
    Forgot to mention, that 4ms applies to a 4GHz i7. If you happen to sit on a 4.5-5GHz sandybridge, it may be closer to 3.5ms.
  • Soul_RiderSoul_Rider Mod Bean Join Date: 2004-06-19 Member: 29388Members, Constellation, Squad Five Blue Posts: 4,264 Advanced user
    Are you making the gamestate recorder multithreaded or single threaded?

    Would there be less of a performance hit to the server/ns2 if it was multithreaded, or would the overhead of additional thread management increase the cost?

    Can the gamestate recorder even be made mutli-threaded?

    Excuse the questioning, I just have an enquiring mind :)
    Games:Subnautica Fan Game Mods: GorgeCraft & CTF & Proving Grounds Website: andysoulrider.uk
    Twitter: @AndySoulRider Twitch: Twitch.tv/SoulRider YouTube: Youtube.com/user/IamSoulRider/videos

    Capitalisation is the difference between helping your Uncle Jack off a horse and helping your uncle jack off a horse.
    Knowledge is gained through listening, Understanding is gained through discussion.
  • playerplayer Join Date: 2010-09-12 Member: 73982Members Posts: 1,677
    GameOvermind is as multi-threaded as I could make it. When it's all said and done, it will have about 14 threads during normal operation. As for the gamestate-recorder, it only spends as much time in the main-thread as it strictly has to. It reads out the raw data and immediately sends it off to another thread for further culling, formatting, compression, optionally encryption, and writing it to the filesystem.
  • oldassgamersoldassgamers Join Date: 2011-02-02 Member: 80033Members, Squad Five Blue, Reinforced - Shadow Posts: 553 Fully active user
    Just of ceriousity, will the new overmind version that you're currently working on affect server performance? As in, will the admin mod itself take cpu streanght?
    The abbreviation for Old Ass Gamers is OAG. OAG is something more then just a word, it makes us gamers understand the passion. The passion that you never want to stop playing video games. That age is just a myth and not even age will stop us.
  • playerplayer Join Date: 2010-09-12 Member: 73982Members Posts: 1,677
    Naturally the new version is a lot more feature-rich, but if you compare them in terms of similar functionality, the new version will be a lot faster (though that's not saying a lot, as the current one doesn't really cost resources to begin with). Any cost in terms of the new features (RCon\Gamestate-recorder\logging-stuff\SQL and assorted other things) is offloaded to other threads as much as possible. So if you intend to run the thing with everything on max, having part of another core available for the server-process is recommended.
  • ScardyBobScardyBob ScardyBob Join Date: 2009-11-25 Member: 69528Forum Admins, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow Posts: 4,983 mod
    How is the load on the other threads? I know many of the OCd NS2 servers are trying to recoup their cost by running multiple servers on the same machine (i.e. the rough number is # of servers = n-1 cores), such that you don't want too many other things running on the machine.
  • oldassgamersoldassgamers Join Date: 2011-02-02 Member: 80033Members, Squad Five Blue, Reinforced - Shadow Posts: 553 Fully active user
    edited April 2012
    QUOTE (player @ Apr 18 2012, 04:09 AM) »
    Naturally the new version is a lot more feature-rich, but if you compare them in terms of similar functionality, the new version will be a lot faster (though that's not saying a lot, as the current one doesn't really cost resources to begin with). Any cost in terms of the new features (RCon\Gamestate-recorder\logging-stuff\SQL and assorted other things) is offloaded to other threads as much as possible. So if you intend to run the thing with everything on max, having part of another core available for the server-process is recommended.


    So basically what your saying is that the adminmod itself will only require cpu streanght when gamestate-recorder is being activated and runs? But if you don't activated that feature the server tickrate will be the same while the admin mod is active ?
    The abbreviation for Old Ass Gamers is OAG. OAG is something more then just a word, it makes us gamers understand the passion. The passion that you never want to stop playing video games. That age is just a myth and not even age will stop us.
  • playerplayer Join Date: 2010-09-12 Member: 73982Members Posts: 1,677
    QUOTE (ScardyBob @ Apr 18 2012, 05:32 AM) »
    How is the load on the other threads? I know many of the OCd NS2 servers are trying to recoup their cost by running multiple servers on the same machine (i.e. the rough number is # of servers = n-1 cores), such that you don't want too many other things running on the machine.

    n-1 is a good rule-of-thumb, though you'll be pushing it if all 3 servers are recording\playing-back at once.

    QUOTE (oldassgamers)
    So basically what your saying is that the adminmod itself will only require cpu streanght when gamestate-recorder is being activated and runs? But if you don't activated that feature the server tickrate will be the same while the admin mod is active ?

    Oh of course it won't be scanning the gamestate if the recorder isn't running, that would be criminally inefficient.
  • Soul_RiderSoul_Rider Mod Bean Join Date: 2004-06-19 Member: 29388Members, Constellation, Squad Five Blue Posts: 4,264 Advanced user
    Hey Player,

    A quick question about Mod support. How flexible will your mod be when it comes to working with other client affecting lua files? For example, I'm looking at getting the mumble positional audio set-up. This needs to launch an alternative client.lua file. This particular mod isn't an issue, as GMOvmnd is server only, and this mod is client only, but if this mod was serverside too, I would struggle to use both mods. Is there anyway you can enable mods to run from within overmind even if they need their own specific launch lua files?
    Games:Subnautica Fan Game Mods: GorgeCraft & CTF & Proving Grounds Website: andysoulrider.uk
    Twitter: @AndySoulRider Twitch: Twitch.tv/SoulRider YouTube: Youtube.com/user/IamSoulRider/videos

    Capitalisation is the difference between helping your Uncle Jack off a horse and helping your uncle jack off a horse.
    Knowledge is gained through listening, Understanding is gained through discussion.
  • ScardyBobScardyBob ScardyBob Join Date: 2009-11-25 Member: 69528Forum Admins, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow Posts: 4,983 mod
    I just wanted to say thank you for your work on OM so far. I really hope you continue working on it and have been looking forward to your gamestate recorder for a while. I can't tell you how much us Team156s (who run OM on four of our servers) appreciate your mod.
  • playerplayer Join Date: 2010-09-12 Member: 73982Members Posts: 1,677
    QUOTE (Soul_Rider @ Apr 26 2012, 10:12 PM) »
    Hey Player,

    A quick question about Mod support. How flexible will your mod be when it comes to working with other client affecting lua files? For example, I'm looking at getting the mumble positional audio set-up. This needs to launch an alternative client.lua file. This particular mod isn't an issue, as GMOvmnd is server only, and this mod is client only, but if this mod was serverside too, I would struggle to use both mods. Is there anyway you can enable mods to run from within overmind even if they need their own specific launch lua files?

    When you wish to run multiple mods that each have their own entrypoint-scripts, you'll need to use 2-stage initialization (the stage before, and the stage after the game-scripts have been loaded). So say you have 3 mods you want to run, first initialize the 1st stage of every one of those 3 mods, then load the game-scripts on your own, then load the 2nd stage of every one of those 3 mods. The old version of GameOvermind (the one currently in use) already allows for this (may have been broken through time though, haven't actually tested it for over a year), a little excerpt of the library-loader:
    CODE
    --[[ Initializes the mod
        Remark: - The possible Mode-values are 0, 1 and 2
             - A Mode-value of 0 runs the mod in master-mode (it will initialize and load the NS2 Lua-files on it's own accord)
             - A Mode-value of 1 runs the mod in slave-mode (it will pre-initialize, to be done before the NS2 Lua-files are loaded)
             - A Mode-value of 2 runs the mod in slave-mode (it will post-initialize, to be done after the NS2 Lua-files are loaded)
            - In slave-mode the "InitFunc"-function has to be called with a Mode-value of 1, after which the NS2 Lua-files have to be manually loaded by the user, then the "InitFunc"-function has to be called again, this time with a Mode-value of 2
         In: INTEGER Mode
         Out: Nil ]]--
        NS2GmOvrmind.InitFunc(0); -- Execute the initialization-function (in master-mode)


    This does demand a little bit of work on the part of the server-operator, I would've expected after all this time for UWE to come out with a mod-loading system that does this, but that still hasn't happened. If by release-time there is still no such system in place, I guess I'll just package the excellent work fsfod has already done in this field with his modloader (in use in the menumod as we speak), and kinda make that the community-standard for mod interop.

    QUOTE (ScardyBob)
    I just wanted to say thank you for your work on OM so far. I really hope you continue working on it and have been looking forward to your gamestate recorder for a while. I can't tell you how much us Team156s (who run OM on four of our servers) appreciate your mod.

    You need not worry I'll ragequit on this mod before its initial release, too much time (an insame amount of time in fact) has gone into it for me to let that happen. Admittedly the progress of NS2 has worked incredibly demotivational, but there's not that much more work remaining before I can kick a beta out the door. Another month of work I guestimate, after that I'm sidetracking development, basically limiting myself to polishing and bug-fixing, and not add any more new features, and just observe where NS2 is going as it approaches its release-date.
Sign In or Register to comment.