Random server crashes (build 268)

AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
edited August 2014 in Server Discussion
Any other server admin getting this? I basically tried to fully reinstall all my mods and configurations and yet my server still keeps crashing. After hours and hours of constant testing I eventually sent a ticket to my server provider and they're fully aware of the issue, apparently waiting for a hotfix. Anyone else having the same problem? Sometimes it occurs during a map change while sometimes the server just redplugs in the middle of a match. Highly frustrating. :c

And no, logdir.txt doesn't really tell me any reason why it would crash all the suddenly. I just want to know if anyone else is having similar experiences after the new patch.
«1

Comments

  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    I get a lot of crashes just after map change on my servers. There is no error messages in the logs. Not sure what's causing it. Mod, bad luck or 268?
  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
    I really wished it would've been some certain mod but unfortunately that doesn't seem to be the case. Gotta be related with the new patch...didn't they change how servers loads maps now? Perhaps that's causing the issue?
  • the_tickthe_tick Netherlands Join Date: 2014-01-20 Member: 193352Members
    3rd crash so far today
  • dePARAdePARA Join Date: 2011-04-29 Member: 96321Members, Squad Five Blue
    I have the same crashes.
    But it looks like we have 4 admins here with modfied servers (serverrates, slotammount)

    Maybe the crashes are more often there, dont know.
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    Yesterday I had 4 crashes in 9 hours on one of my servers. Most of them just after a map change. It runs at default rates and without hacks.
  • GhoulofGSG9GhoulofGSG9 Join Date: 2013-03-31 Member: 184566Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, Reinforced - Supporter, WC 2013 - Supporter, Pistachionauts
    Anything in the log? Does it crash or freeze at mapchange?
  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
    I've had 4 crashes in the past 6 hours. Server does load up the next map but crashes immediately after. Leaving fast loading people with black screen that eventually crashes them too.
  • GhoulofGSG9GhoulofGSG9 Join Date: 2013-03-31 Member: 184566Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, Reinforced - Supporter, WC 2013 - Supporter, Pistachionauts
    Amad wrote: »
    I've had 4 crashes in the past 6 hours. Server does load up the next map but crashes immediately after. Leaving fast loading people with black screen that eventually crashes them too.

    So the server.exe does crash and not just freezed ( you can test this by checking if the webadmin is still reachable ).
  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
    Amad wrote: »
    I've had 4 crashes in the past 6 hours. Server does load up the next map but crashes immediately after. Leaving fast loading people with black screen that eventually crashes them too.

    So the server.exe does crash and not just freezed ( you can test this by checking if the webadmin is still reachable ).

    Webadmin is not reachable until the server finishes booting up.
  • dePARAdePARA Join Date: 2011-04-29 Member: 96321Members, Squad Five Blue
    I remember crashes @ mapchange caused by shine a while ago and it looks like that the server from Zero (Ibisgaming) didnt crash (he is using DAK)

    Would be interisting to now if an Vanilla server has the same issues.
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    edited August 2014
    Here is my log from the latest crash just five minutes ago
    [23:11:48]dino[98485725] ran command sh_vote with arguments: ns2_summit
    [23:11:49]P[47135802] ran command sh_vote with arguments: ns2_summit
    Downloading mods
    Finished downloading and installing mods
    Mounting mod from c:/thirsty1/mods/m706d242_1408976514/
    Mounting mod from c:/thirsty1/mods/m812f004_1408894912/
    Mounting mod from c:/thirsty1/mods/mc6fbbb0_1408734207/
    Mounting mod from c:/thirsty1/mods/maa8da06_1408729446/
    Loading Shine config...
    Loading extensions...
    bandwidth limit: 102400 bytes/sec per player
    - Extension 'basecommands' loaded.
    - Extension 'votesurrender' loaded.
    - Extension 'voterandom' loaded.
    - Extension 'unstuck' loaded.
    - Extension 'adverts' loaded.
    - Extension 'mapvote' loaded.
    - Extension 'namefilter' loaded.
    - Extension 'crossspawns' loaded.
    - Extension 'ns2stats' loaded.
    - Extension 'ban' loaded.
    - Extension 'logging' loaded.
    - Extension 'afkkick' loaded.
    - Extension 'adminmenubutton' loaded.
    - Extension 'pregameplus' loaded.
    - Extension 'serverswitch' loaded.
    - Extension 'motd' loaded.
    Completed loading Shine extensions.
    Loading Shine users...
    [23:12:18]Shine started up successfully.
    [23:12:18]Loading config://ServerConfig.json
    [23:12:18]Loading config://ReservedSlotsConfig.json
    [23:12:18]Loading config://ServerAdmin.json
    [23:12:18]Loading config://BannedPlayers.json
    [23:12:18]Loading config://MapCycle.json
    [23:12:18]Loading config://ConsistencyConfig.json
    [23:12:18]Hashed 1 game_setup.xml files for consistency
    [23:12:18]Hashed 980 *.lua files for consistency
    [23:12:18]Hashed 2737 materials/* files for consistency
    [23:12:18]Hashed 38 sound/* files for consistency
    [23:12:18]Hashed 12 maps/*.level files for consistency
    [23:12:18]Hashed 20 shaders/*.hlsl files for consistency
    [23:12:18]Hashed 24 models/*.surface_shader files for consistency
    [23:12:18]Hashed 1 shaders/Basic.surface_shader files for consistency
    [23:12:18]Hashed 1 shaders/Emissive.surface_shader files for consistency
    [23:12:18]Hashed 1 shaders/Level.surface_shader files for consistency
    [23:12:18]Hashed 1 shaders/Level_alpha.surface_shader files for consistency
    [23:12:18]Hashed 1 shaders/Level_emissive.surface_shader files for consistency
    [23:12:18]Hashed 1 shaders/Model.surface_shader files for consistency
    [23:12:18]Hashed 1 shaders/Model_alpha.surface_shader files for consistency
    [23:12:18]Hashed 1 shaders/Model_emissive.surface_shader files for consistency
    [23:12:18]Hashed 1 shaders/Model_emissive_alpha.surface_shader files for consistency
    [23:12:18]Hashed 30 models/alien/skulk/* files for consistency
    [23:12:18]Hashed 28 models/alien/lerk/* files for consistency
    [23:12:18]Hashed 20 models/alien/fade/* files for consistency
    [23:12:18]Hashed 15 models/alien/onos/* files for consistency
    [23:12:18]Hashed 20 models/alien/gorge/gorge* files for consistency
    [23:12:18]Skipped 100 *.py files for consistency
    [23:12:18]Skipped 75 materials/dev/* files for consistency
    [23:12:18]Skipped 21 materials/editor/* files for consistency
    [23:12:18]Skipped 15 materials/hud/* files for consistency
    [23:12:18]Skipped 85 materials/skybox/* files for consistency
    [23:12:18]Skipped 4 materials/system/* files for consistency
    [23:12:18]Skipped 19 materials/effects/mesh_effects/* files for consistency
    [23:12:18]Skipped 98 materials/effects/*.dds files for consistency
    [23:12:18]Skipped 0 materials/effects/*.py files for consistency
    [23:12:18]Skipped 100 materials/effects/*.polygons files for consistency
    [23:12:18]Skipped 0 cinematics/environment/* files for consistency
    [23:12:18]Skipped 0 cinematics/test/* files for consistency
    [23:12:18]Skipped 9 models/alien/skulk/*view* files for consistency
    [23:12:18]Skipped 13 models/alien/lerk/*view* files for consistency
    [23:12:18]Skipped 6 models/alien/fade/*view* files for consistency
    [23:12:18]Skipped 6 models/alien/onos/*view* files for consistency
    [23:12:18]Skipped 6 models/alien/gorge/gorge*view* files for consistency
    [23:12:18]Skipped 7 models/alien/fade/*vortex* files for consistency
    [23:12:18]Skipped 1 shaders/Blur.hlsl files for consistency
    [23:12:18]Skipped 1 shaders/DarkVision.hlsl files for consistency
    [23:12:18]Hashed 1 lua/entry/*.entry files for consistency
    [23:12:18]Consistency checking took 0 seconds
    [23:12:18][Elixer (Server)] Loading Utility Scripts v.1.72
    [23:12:18][Elixer (Server)] Using Utility Scripts v.1.72
    [23:12:18]Loading config://NS2PlusServerConfig.json
    [23:12:18][NS2+] Mod updater is: Enabled
    [9120.143] Worker 05 : Loading 'maps/ns2_summit.level'
    Building pathing mesh for level maps/ns2_summit.level
    Finished loading 'maps/ns2_summit.level'
    Client connected (81.27.12.236)
    [23:12:28]Client Authed. Steam ID: 11611431
    Strego connected.
    Client connected (84.251.27.140)
    [23:12:30]Client Authed. Steam ID: 13123664
    oh connected.
    Client connected (149.172.173.218)
    [23:12:31]Client Authed. Steam ID: 1839897
    Isoterik connected.
    Client connected (217.250.77.238)
    [23:12:32]Client Authed. Steam ID: 98485725
    dino connected.
    Client connected (85.224.182.104)
    [23:12:32]Client Authed. Steam ID: 36770742
    [OAG] HYAan connected.
    Client connected (193.83.228.148)
    [23:12:34]Client Authed. Steam ID: 75715334
    Flugstein connected.
    Client connected (85.179.188.7)
    [23:12:35]Client Authed. Steam ID: 46347502
    vertigo connected.
    Client connected (82.74.66.214)
    [23:12:35]Client Authed. Steam ID: 47135802
    P connected.
    Client connected (88.66.135.248)
    [23:12:39]Client Authed. Steam ID: 1663296
    Deanosaurus connected.
    Client connected (81.227.93.97)
    [23:12:40]Client Authed. Steam ID: 475075
    Mr Boz connected.
    Client connected (78.69.38.192)
    [23:12:42]Client Authed. Steam ID: 84783956
    Anakie connected.
    Client connected (89.2.9.68)
    [23:12:43]Client Authed. Steam ID: 21396444
    Euregan connected.
    Client connected (89.67.145.58)
    [23:12:46]Client Authed. Steam ID: 10534949
    Friend Computer connected.
    Client connected (77.179.199.118)
    [23:12:46]Client Authed. Steam ID: 165422768
    sta connected.
    Client connected (109.192.50.85)
    [23:12:47]Client Authed. Steam ID: 296310
    ~C|-|!LL3|2~ connected.
    Client connected (81.217.47.122)
    [23:12:47]Client Authed. Steam ID: 80592649
    Bluewind connected.
    [23:12:58]Bluewind[80592649] ran command sh_voterandom with no arguments.
    Client connected (176.199.153.26)
    [23:13:00]Client Authed. Steam ID: 108208717
    exe connected.
    [23:13:04]sta[165422768] ran command sh_voterandom with no arguments.
    [23:13:05]~C|-|!LL3|2~ was killed by oh
    [23:13:08]vertigo[46347502] ran command sh_voterandom with no arguments.
    [23:13:12]Euregan was killed by oh
    [23:13:12]Deanosaurus[1663296] ran command sh_voterandom with no arguments.
    [23:13:13]Mr Boz[475075] ran command sh_voterandom with no arguments.
    Client connected (94.30.48.182)
    [23:13:16]Client Authed. Steam ID: 45835001
    America connected.
    Client disconnected (213.64.0.238) Timeout
    [23:13:21]Strego was killed by oh
    [23:13:21]oh was killed by [OAG] HYAan
    [23:13:36]vertigo was killed by [OAG] HYAan
    [23:13:36]Bluewind was killed by Strego
    Client connected (83.89.38.70)
    [23:13:42]Client Authed. Steam ID: 70529433
    GanjaGhost connected.

    Sometimes it crashes right after map change and sometimes it takes a couple of minutes. The server runs NS2+, Shine admin, Shine NS2Stats and Shine extra plugins.
  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
    Hmm, compared to my own server log the only similar thing I can find is the voting. People started to vote for a map change and after a minute or so, the server crashed.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    @Supafred This is the one we call Pregame Crash. We've reported it a long time ago but every modded and/or 24+ server's issues get discarded at the iron wall. I.e. 95% of the servers . . .

    For us we started getting it some builds ago, I'd say 265 (don't hold me to that). And then it was fine with say 266. With 267 and beyond the Pregame Crash is back as it was before.

    Our prime suspects for this (hold on to your seat) are:
    - The randomizerr and voteforceeven votes from vanilla
    - The badgesrequest (see /lua/Badges_Server.lua line 6 set to false)

    Try disabling those three (we haven't gotten any further in figuring out which one it is because we, too, are struggling with crashes. However we try not to make a fuss since our server is borderline insane. Unless the world turns upside down, we will always see more crashes than smaller servers.)
  • dePARAdePARA Join Date: 2011-04-29 Member: 96321Members, Squad Five Blue
    I disabled the shine mapvote today for testing, after 1 round the default mapcyle changed the map.
    Crash

    The only error before the mapchange was:
    [7045.768] Main : Error: Failed to add an obstacle at -79.65, -100.89, 45.65!
  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
    ATF wrote: »
    Our prime suspects for this (hold on to your seat) are:
    - The randomizerr and voteforceeven votes from vanilla
    - The badgesrequest (see /lua/Badges_Server.lua line 6 set to false)

    Strange, I disabled the randomizer and voteforceeven votes and the crashes has been greatly reduced. Still getting them but not as often as before.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    Glad it helped. Keep a close eye on it tho as this is all speculation.
    We have the above mentioned three things disabled since a couple days and i don't think i've seen another pregame crash.

    Ah, yes, I should speculate in detail that the crash does not occur the moment the vote succeeds and puts players on the team. Instead, it occurs right when someone starts the vote and before the vote gets shown on clients screens and before it gets logged in the log!

    dePARA wrote: »
    I disabled the shine mapvote today for testing, after 1 round the default mapcyle changed the map.
    Crash

    The only error before the mapchange was:
    [7045.768] Main : Error: Failed to add an obstacle at -79.65, -100.89, 45.65!

    Shine does things to those votes (make them leave afks in rr) afaik. It's entirely possible that shine or shine's afk system (not just it's mapvote) causes it.
    Regrettably we can't disable shine for testing as we *still* need the extraips extension. A smaller server could, potentially disable shine entirely to test this.

    Your crash might be a different one, though, dePARA. If it does not get at least to the point where the first players have finished connecting, it's not the 'pregame crash' I speculate. Not that this helps... :-<

    The error you noted, we get 3-20 of those after every round and consider them 'normal'.
  • AmbivalentAmbiguityAmbivalentAmbiguity Miami, FL Join Date: 2014-02-18 Member: 194129Members, Reinforced - Shadow
    edited August 2014
    My servers are also both having crashes on map change. Server swaps to next map, and then crashes on loading the map. One is stock with Shine a few other mods, the other is a modified tick rate/interp/mr/sr/bwlimit


    @ATF
    ATF wrote: »
    Our prime suspects for this (hold on to your seat) are:
    - The randomizerr and voteforceeven votes from vanilla
    - The badgesrequest (see /lua/Badges_Server.lua line 6 set to false)

    How did you guys come to these conclusions? I'm probably going to take the advice but I'm just curious. Also I more so mean the badges_server change.

  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow
    ATF wrote: »
    Ah, yes, I should speculate in detail that the crash does not occur the moment the vote succeeds and puts players on the team. Instead, it occurs right when someone starts the vote and before the vote gets shown on clients screens and before it gets logged in the log!

    Exactly how I managed to crash my server the first time it occured, tried to start a map vote myself and what do you know, redplug. However, I haven't had a single crash in a whole day because of those little things you suggested. This seems like a possible work around though I'll keep monitoring the situation.

    Much appreciated, thank you.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    How did you guys come to these conclusions? I'm probably going to take the advice but I'm just curious. Also I more so mean the badges_server change.

    We are ancient magic wizards with silver grey hair and tall pointy hats!


    In all seriousness, it's a lot of guesswork! During pregame, the server essentially isn't doing anything. It should absolutely not crash at that time. So we tried to think of what are the things that do happen during pregame.. and that came up so trial and error...

    Regarding the badges request, that came from a different angle. Since a 42 player server pushes absolutely every limit, it also sent the Hive servers more requests than they accepted at the time.
    (When a player connects, it contacts UWE's Hive server and checks if that player has any badges that need displaying).
    This caused a lag spike right when players load after a mapcycle.
    We talked with Acedude who manages the Hive API server and he doubled their capacity. The lagspike was fixed.
    However, during all of that, we found a way to en-/disable the badgesrequest and since that was also at the time we had the pregame crashes, it -appeared to play a role- in it.
    The badges request is the other thing the server does during pregame, once for every player that connects / loads the map.

    Glad you asked ;)
  • dePARAdePARA Join Date: 2011-04-29 Member: 96321Members, Squad Five Blue
    Well, no, It cant be the randomizer or forceeventeams,
    I have disabled these features since ages and the server is crashing more since 267.
    And since 268 there can be crashes every 3rd mapchange.

    I was alone on the server @ one test using the sv_cyclemap map command and the server crashed at first mapchange.

    My suggestion is the new cache mechanic in 268.

    And again, we have still no feedback from an admin of an vanilla server.

  • GhoulofGSG9GhoulofGSG9 Join Date: 2013-03-31 Member: 184566Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, Reinforced - Supporter, WC 2013 - Supporter, Pistachionauts
    edited October 2014
    Ok,

    seems like there are some critical issues at map loading. To investigate into this we need to get crashdumps.

    How to enable crash dumping for your windows server:

    1. Open the cmd
    2. Type in regedit
    3. Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\
    4. Create a new key called LocalDumps

    5. Wait for the server.exe to crash
    6.Open run ( with win + r ) and type in %APPDATA%\CrashDumps now there should be a file.
    7. Zip it, upload it somewhere and post the link here

    Thanks for your help ;)
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    Any news on this? I am still having these kinds of crashes. I haven't been able to get a dump yet but I will if you still want it.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    @SupaFred How do you manage to not have dumps? Did you do the WER thing Ghoul said? The registry thing. You'll get a dump everytime it crashes. Zip it, get it to the devs.

    Did you try the three things I mentioned? Does it help, not help?
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    @ATF‌ No I have been on a trip and haven't had the time to enable crash dump yet. It seems like no one else have supplied a dump yet so I will try to produce one. I have disabled force even teams but not the badge requests.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    We're handing dumps in packs to the devs directly, just not publicly. We've had a lead playtester intentionally crash our server with malformed packets in the past. You get careful around their kind.

    Whenever we enable even one of the three things mentioned (2 votes & badgereq) we get the pregame crash. So if you disable just one... oh well see for yourself.
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    I have followed the instructions but I still can't get any crashdumps. I still have the problem with the servers crashing just after map change. Disabling badges didn't help.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    edited September 2014
    Here's an instruction that's a bit more detailed. We only use the "dumpfolder" value to set a custom folder.
    If you can't figure it out, add me on steam and I'll take a look (provided you have vnc installed).

    Regarding the pregamecrash, you need to disable all three things mentioned above, not just one.

    EDIT:
    As a matter of fact, we've just now had a pregame crash with 8-10 players on the server. And one player reported specifically to have hit the randomizerr vote just before. (We activate those after every build release to check if it's fixed. . .)
    We do have a dump of it and from the last dumps of this type, a dev said it's a luaj crash which likely originates from the older lua scripts and libs.
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    Thanks ATF. Strange thing is I had done everything right. Only key I didn't have was CustomDumpFlags. Will experiment more tomorrow.

    I am starting to suspect the built in voting system. I have had several players reporting server crashes just as they called a vote. Just now a reset vote was called and then the server crashed.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members
    XdP3ebr.jpg
    This is what works for us.

    Well if even the reset vote causes it, that would make sense. According to the devs all those votes call the votemanager parent class which checks for vote validity. The crash is then caused by an luaj call (taken from dumps). Only @fsfod can tell which luaj call it is, since he wrote the lua vms.

    Prepare your offerings. [-O<
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver
    I haven't had any pregame crashes since I disabled the built-in votes.
Sign In or Register to comment.