Random server crashes (build 268)

2»

Comments

  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow Posts: 62 Advanced user
    I haven't had any crashes at all since 269. All votes enabled etc.
  • The_RangerThe_Ranger So.Cali Join Date: 2003-01-27 Member: 12800Members, Constellation, WC 2013 - Shadow Posts: 210 Fully active user
    I had a few crashes in 269 while votes were on, 0 while off. I plan to turn them on again and see if if it stays the same, as I have also made a few other changes.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members Posts: 261
    We are currently testing with the shine afkkick extension disabled.
    Although looking at its lua it only hooks into VoteRandomizeRR, not VotingForceEvenTeams.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members Posts: 261
    dumping this here as I'm in a hurry:

    [372.322] Main : Error: lua/Team.lua:153: expected self tobe a Type
    [Server] Script Error #5: lua/Team.lua:153: expected self tobe a Type
    Call stack:
    #1: GetOwner [C]:-1
    #2: StartVote lua/Team.lua:153
    player = "VoteRandomizeRR"
    #3: lua/Voting.lua:114
    client = ServerClient { }
    message = {client_index=0, expireTime=0, voteId=0 }

    [375.670] Main : Error: lua/Team.lua:153: expected self tobe a Type
    [Server] Script Error #6: lua/Team.lua:153: expected self tobe a Type
    Call stack:
    #1: GetOwner [C]:-1
    #2: StartVote lua/Team.lua:153
    player = "VotingForceEvenTeams"
    #3: lua/Voting.lua:114
    client = ServerClient { }
    message = {client_index=0, expireTime=0, voteId=0 }

    Got those during a pregame, they might help point to the origin.
  • the_tickthe_tick Netherlands Join Date: 2014-01-20 Member: 193352Members Posts: 97
    I think this is a good spot to look at, just now, ns2_summit just loaded in. and after about 2 minutes ( which is also my afk kick timer) the server crashed
    Owner of the tick's battleground servers
  • ATFATF Join Date: 2014-05-09 Member: 195944Members Posts: 261
    The shine afkkick extension disabled does not stop the pregame crash, so we can rule it out as the culprit!

    What remains are the built-in votes. Those need to be looked at.
  • the_tickthe_tick Netherlands Join Date: 2014-01-20 Member: 193352Members Posts: 97
    I've disabled built in votes for 6 months or so now, can't be that...
    Owner of the tick's battleground servers
  • remiremi remedy [blu.knight] Join Date: 2003-11-18 Member: 23112Members, Super Administrators, Forum Admins, NS2 Developer, NS2 Playtester Posts: 2,689 admin
    ATF wrote: »
    dumping this here as I'm in a hurry:

    [372.322] Main : Error: lua/Team.lua:153: expected self tobe a Type
    [Server] Script Error #5: lua/Team.lua:153: expected self tobe a Type
    Call stack:
    #1: GetOwner [C]:-1
    #2: StartVote lua/Team.lua:153
    player = "VoteRandomizeRR"
    #3: lua/Voting.lua:114
    client = ServerClient { }
    message = {client_index=0, expireTime=0, voteId=0 }

    [375.670] Main : Error: lua/Team.lua:153: expected self tobe a Type
    [Server] Script Error #6: lua/Team.lua:153: expected self tobe a Type
    Call stack:
    #1: GetOwner [C]:-1
    #2: StartVote lua/Team.lua:153
    player = "VotingForceEvenTeams"
    #3: lua/Voting.lua:114
    client = ServerClient { }
    message = {client_index=0, expireTime=0, voteId=0 }

    Got those during a pregame, they might help point to the origin.

    That is a really odd call-stack. Wonder if it's some issue with luajit doing something weird.
    [ Rek | Reklys | remedy | remi.D | blu.knight | Psyke | Sky ]
    I have way too many names.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members Posts: 261
    I scrabbled about in our log archive and found some info on these:
    1. They started appearing on September 2nd.
    2. They appear during pregame only, maybe at a 3% rate. Meaning mostly they don't appear.
    3. There's not just those two but also for the other 3 votes. So it happens for all vanilla votes.

    There's a lot in common with the preconditions for the pregame crash. I'd say they are a hint.

    You are already in posession of 15-20 dumps of those crashes, and yes the last thing in those is a luaj call according to Ghoul.

    Additionally -and this is like a ghost story-, we've always noticed a slight tendancy for the pregame crash to happen on veil and veil five specifically. It also happens on random other maps, albeit much more rarely.
    Recently we've added veil five again and out of 8 times it got voted, 4 times pregame crash. Now the map is out again.
  • ATFATF Join Date: 2014-05-09 Member: 195944Members Posts: 261
    Finding a bunch of these in the log today:
    [9258.987] Main : Error: lua/MixinDispatcherBuilder.lua:230: attempt to call field '__towatch' (a nil value)
    [Server] Script Error #1: lua/MixinDispatcherBuilder.lua:230: attempt to call field '__towatch' (a nil value)
        Call stack:
        #1: StartVote lua/MixinDispatcherBuilder.lua:230
            functionName = "OnGetIsSelectable"
            allFunctionsTable =  {1=function, 2=function }
            self = "VoteKickPlayer"
            count = 2
            className = nil
        #2: lua/Voting.lua:114
            client = ServerClient { }
            message =  {client_index=0, expireTime=0, kick_client=14, voteId=0 }
    
  • ATFATF Join Date: 2014-05-09 Member: 195944Members Posts: 261
    So far no pregame crash since 270, with all but one ingame-vote enabled. :)>-
  • ATFATF Join Date: 2014-05-09 Member: 195944Members Posts: 261
    edited November 2014
    Having the pregame crash on ns2_yana. Yep, it's confirmed the exact same symptoms and repro. We're investigating.

    EDIT: Same on ns2_orbital.
    Post edited by ATF on
  • TermFragsTermFrags Tampa, FL Join Date: 2012-08-30 Member: 156877Members Posts: 13 Fully active user
    Had random crashes after the 270 update a few days ago. Debian Squeeze server running updated IA32 libs since stable are outdated. Checked server logs, no output. Just a random crash with no error. Checked the messages log, I found this:

    Nov 2 17:04:45 gemini kernel: [16123001.700977] server_linux32[18836]: segfault at 200354 ip 00000000f6be9e11 sp 00000000ff96cd78 error 4 in liblua51.so[f6b9b000+8e000]
    Nov 2 23:33:25 gemini kernel: [16146300.109188] server_linux32[20697]: segfault at 194 ip 00000000f7245fae sp 00000000ffcecd10 error 4 in libSpark_Core.so[f7040000+6d8000]
    Nov 3 00:29:29 gemini kernel: [16149660.478617] server_linux32[21446]: segfault at 0 ip 00000000f6b9d3db sp 00000000ff9059c0 error 4 in liblua51.so[f6b2c000+8e000]


    I suspected it was NS2Stats. After disabling it, no more crashes. I meant to post this earlier but I haven't had time, I've been busy playing on the server. ;)
  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow Posts: 62 Advanced user
    TermFrags wrote: »
    Had random crashes after the 270 update a few days ago. Debian Squeeze server running updated IA32 libs since stable are outdated. Checked server logs, no output. Just a random crash with no error. Checked the messages log, I found this:

    Nov 2 17:04:45 gemini kernel: [16123001.700977] server_linux32[18836]: segfault at 200354 ip 00000000f6be9e11 sp 00000000ff96cd78 error 4 in liblua51.so[f6b9b000+8e000]
    Nov 2 23:33:25 gemini kernel: [16146300.109188] server_linux32[20697]: segfault at 194 ip 00000000f7245fae sp 00000000ffcecd10 error 4 in libSpark_Core.so[f7040000+6d8000]
    Nov 3 00:29:29 gemini kernel: [16149660.478617] server_linux32[21446]: segfault at 0 ip 00000000f6b9d3db sp 00000000ff9059c0 error 4 in liblua51.so[f6b2c000+8e000]


    I suspected it was NS2Stats. After disabling it, no more crashes. I meant to post this earlier but I haven't had time, I've been busy playing on the server. ;)

    Ironically, I've also had tons of crashes in the past of couple of weeks. Especially during mapchanges. Decided to take a proper look today and what do you know, disabling [SHINE]NS2stats solved the problem. Not a single crash in over 24 hours. ;o
    SupaFred
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver Posts: 440 Advanced user
    @Amad‌ Same here. Ghoul runs NS2Stats without problems though. Perhaps it's one of the settings in ns2stats.json causing it?
    Owner of The Thirsty Onos servers
    Steam profile
    YouTube for NS2 stuff
  • GhoulofGSG9GhoulofGSG9 Join Date: 2013-03-31 Member: 184566Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, Reinforced - Supporter, WC 2013 - Supporter, Pistachionauts Posts: 2,819 admin
    edited November 2014
    I'm refactoring ns2stats just another time to see if it can fix it at my side. Fun note thanks to my new IDE tools ns2stats already runs now 4-5 times faster.

    But the io stream issues seem to still be there when it comes to reading larger json files.



    Developer, Modder and Server Admin of Survival of the Fattest - Ingame Nick: Ghoul
  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow Posts: 62 Advanced user
    SupaFred wrote: »
    @Amad‌ Same here. Ghoul runs NS2Stats without problems though. Perhaps it's one of the settings in ns2stats.json causing it?

    Have you tried changing the settings yet? I'm going to toy around with them tonight.
  • SupaFredSupaFred Join Date: 2013-03-03 Member: 183652Members, NS2 Playtester, Squad Five Blue, NS2 Map Tester, Reinforced - Supporter, Reinforced - Silver, Reinforced - Shadow, WC 2013 - Silver Posts: 440 Advanced user
    @Amad‌ No not yet. The settings I changed from default are
    "Awards":false,
    "LogChat":true,
    "StatusReport":true

    What settings did you use?
    Owner of The Thirsty Onos servers
    Steam profile
    YouTube for NS2 stuff
  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow Posts: 62 Advanced user
    The only settings I've ever changed from that .json file was awards, set it to false. Now I tried all different options but no go, server keeps crashing between map changes, randomly but crashes nevertheless. Can't use ns2stats at all. :/
  • GhoulofGSG9GhoulofGSG9 Join Date: 2013-03-31 Member: 184566Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, Reinforced - Supporter, WC 2013 - Supporter, Pistachionauts Posts: 2,819 admin
    Get me some crash dumps please ;)
    Developer, Modder and Server Admin of Survival of the Fattest - Ingame Nick: Ghoul
    ATF
  • AmadAmad Join Date: 2013-08-07 Member: 186626Members, Reinforced - Silver, Reinforced - Shadow Posts: 62 Advanced user
    So apparently Compmod doesn't like NS2stats. With compmod enabled and NS2stats disabled, everything works (Obviously works other way around too). These two mods enabled however...server will crash eventually between mapchanges.

    Saw this in the previous server log:

    [66295.006] Main : Error:
    EntityChangeMixin.optionalCallbacks =

    {

    OnEntityChange = "Called when an entity changes into another entity or is destroyed"

    }



    if Server then



    /**

    * Pass in Id of new Entity this Entity is turning into or nil if it's being deleted.

    */

    function EntityChangeMixin:SendEntityChanged(newId)



    // This happens during the game shutdown process, so don't force a new game

    // rules to be created if one doesn't already exist.

    if GetHasGameRules() then


    [Server] Script Error #1:
    EntityChangeMixin.optionalCallbacks =

    {

    OnEntityChange = "Called when an entity changes into another entity or is destroyed"

    }



    if Server then



    /**

    * Pass in Id of new Entity this Entity is turning into or nil if it's being deleted.

    */

    function EntityChangeMixin:SendEntityChanged(newId)



    // This happens during the game shutdown process, so don't force a new game

    // rules to be created if one doesn't already exist.

    if GetHasGameRules() then


    Call stack:
    #1: read [C]:-1
    #2: OnScriptLoaded lua/EventTester.lua:169
    fileName = "lua/EntityChangeMixin.lua"
    file = userdata
    #3: Load lua/EventTester.lua:193
    fileName = "lua/EntityChangeMixin.lua"
    reload = nil
    #4: lua/CompMod/NewTech/TunnelExit.lua:16
    #5: scriptLoad [C]:-1
    #6: Load lua/EventTester.lua:191
    fileName = "lua/CompMod/NewTech/TunnelExit.lua"
    reload = nil
    #7: lua/CompMod_Shared.lua:11
    ModFiles = {1="lua/CompMod/NewTech/GorgeTunnelEntranceAbility.lua", 2="lua/CompMod/NewTech/GorgeTunnelExitAbility.lua", 3="lua/CompMod/NewTech/HeavyMachineGun.lua", 4="lua/CompMod/NewTech/TunnelExit.lua", 5="lua/CompMod/NewTech/WalkMixin.lua", 6="lua/CompMod/NewTech/Welder.lua" }
    (for index) = 4
    (for limit) = 6
    (for step) = 1
    i = 4
    #8: scriptLoad [C]:-1
    #9: Load lua/EventTester.lua:191
    fileName = "lua/CompMod_Shared.lua"
    reload = nil
    #10: lua/CompMod_Server.lua:4
    #11: scriptLoad [C]:-1
    #12: Load lua/EventTester.lua:191
    fileName = "lua/CompMod_Server.lua"
    reload = nil
    #13: lua/ModLoader.lua:116
    modEntries = {1= {Client="lua/CompMod_Client.lua", ModName="CompMod", Predict="lua/CompMod_Predict.lua", Priority="100", Server="lua/CompMod_Server.lua" }, 2= {Client="lua/NS2Plus/CHUD_Client.lua", ModName="CustomHUD", Predict="lua/NS2Plus/CHUD_Predict.lua", Priority="50", Server="lua/NS2Plus/CHUD_Server.lua" }, 3= {Client="lua/Badges+_Client.lua", ModName="Badges+", Priority="4", Server="lua/Badges+_Server.lua" } }
    entryFiles = {1="lua/entry/Badges+.entry", 2="lua/entry/CompMod.entry", 3="lua/entry/CustomHUD.entry" }
    ParseEntryFile = function
    SortByModPriority = function
    (for index) = 1
    (for limit) = 3
    (for step) = 1
    i = 1
    modEntry = {Client="lua/CompMod_Client.lua", ModName="CompMod", Predict="lua/CompMod_Predict.lua", Priority="100", Server="lua/CompMod_Server.lua" }
    #14: scriptLoad [C]:-1
    #15: Load lua/EventTester.lua:191
    fileName = "lua/ModLoader.lua"
    reload = nil
    #16: lua/PostLoadMod.lua:12
    #17: scriptLoad [C]:-1
    #18: Load lua/EventTester.lua:191
    fileName = "lua/PostLoadMod.lua"
    reload = nil
    #19: lua/Server.lua:462
    chatMessageCount = 0
    reservedSlots = {amount=0, ids= { } }
    kMapEntityLoadPriorities = {ns2_gamerules=1 }
    GetMapEntityLoadPriority = function
    DumpServerEntity = function
    LoadServerMapEntity = function
    CheckForDuplicateLocations = function
    OnMapPostLoad = function
    OnCanPlayerHearPlayer = function
    OnCheckConnectionAllowed = function
2»
Sign In or Register to comment.