I had a few crashes in 269 while votes were on, 0 while off. I plan to turn them on again and see if if it stays the same, as I have also made a few other changes.
We are currently testing with the shine afkkick extension disabled.
Although looking at its lua it only hooks into VoteRandomizeRR, not VotingForceEvenTeams.
I think this is a good spot to look at, just now, ns2_summit just loaded in. and after about 2 minutes ( which is also my afk kick timer) the server crashed
I scrabbled about in our log archive and found some info on these:
1. They started appearing on September 2nd.
2. They appear during pregame only, maybe at a 3% rate. Meaning mostly they don't appear.
3. There's not just those two but also for the other 3 votes. So it happens for all vanilla votes.
There's a lot in common with the preconditions for the pregame crash. I'd say they are a hint.
You are already in posession of 15-20 dumps of those crashes, and yes the last thing in those is a luaj call according to Ghoul.
Additionally -and this is like a ghost story-, we've always noticed a slight tendancy for the pregame crash to happen on veil and veil five specifically. It also happens on random other maps, albeit much more rarely.
Recently we've added veil five again and out of 8 times it got voted, 4 times pregame crash. Now the map is out again.
Had random crashes after the 270 update a few days ago. Debian Squeeze server running updated IA32 libs since stable are outdated. Checked server logs, no output. Just a random crash with no error. Checked the messages log, I found this:
Nov 2 17:04:45 gemini kernel: [16123001.700977] server_linux32[18836]: segfault at 200354 ip 00000000f6be9e11 sp 00000000ff96cd78 error 4 in liblua51.so[f6b9b000+8e000]
Nov 2 23:33:25 gemini kernel: [16146300.109188] server_linux32[20697]: segfault at 194 ip 00000000f7245fae sp 00000000ffcecd10 error 4 in libSpark_Core.so[f7040000+6d8000]
Nov 3 00:29:29 gemini kernel: [16149660.478617] server_linux32[21446]: segfault at 0 ip 00000000f6b9d3db sp 00000000ff9059c0 error 4 in liblua51.so[f6b2c000+8e000]
I suspected it was NS2Stats. After disabling it, no more crashes. I meant to post this earlier but I haven't had time, I've been busy playing on the server.
Had random crashes after the 270 update a few days ago. Debian Squeeze server running updated IA32 libs since stable are outdated. Checked server logs, no output. Just a random crash with no error. Checked the messages log, I found this:
Nov 2 17:04:45 gemini kernel: [16123001.700977] server_linux32[18836]: segfault at 200354 ip 00000000f6be9e11 sp 00000000ff96cd78 error 4 in liblua51.so[f6b9b000+8e000]
Nov 2 23:33:25 gemini kernel: [16146300.109188] server_linux32[20697]: segfault at 194 ip 00000000f7245fae sp 00000000ffcecd10 error 4 in libSpark_Core.so[f7040000+6d8000]
Nov 3 00:29:29 gemini kernel: [16149660.478617] server_linux32[21446]: segfault at 0 ip 00000000f6b9d3db sp 00000000ff9059c0 error 4 in liblua51.so[f6b2c000+8e000]
I suspected it was NS2Stats. After disabling it, no more crashes. I meant to post this earlier but I haven't had time, I've been busy playing on the server.
Ironically, I've also had tons of crashes in the past of couple of weeks. Especially during mapchanges. Decided to take a proper look today and what do you know, disabling [SHINE]NS2stats solved the problem. Not a single crash in over 24 hours. ;o
GhoulofGSG9Join Date: 2013-03-31Member: 184566Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Squad Five Silver, Reinforced - Supporter, WC 2013 - Supporter, Pistachionauts
edited November 2014
I'm refactoring ns2stats just another time to see if it can fix it at my side. Fun note thanks to my new IDE tools ns2stats already runs now 4-5 times faster.
But the io stream issues seem to still be there when it comes to reading larger json files.
The only settings I've ever changed from that .json file was awards, set it to false. Now I tried all different options but no go, server keeps crashing between map changes, randomly but crashes nevertheless. Can't use ns2stats at all.
So apparently Compmod doesn't like NS2stats. With compmod enabled and NS2stats disabled, everything works (Obviously works other way around too). These two mods enabled however...server will crash eventually between mapchanges.
Saw this in the previous server log:
[66295.006] Main : Error:
EntityChangeMixin.optionalCallbacks =
{
OnEntityChange = "Called when an entity changes into another entity or is destroyed"
}
if Server then
/**
* Pass in Id of new Entity this Entity is turning into or nil if it's being deleted.
*/
function EntityChangeMixin:SendEntityChanged(newId)
// This happens during the game shutdown process, so don't force a new game
// rules to be created if one doesn't already exist.
Newest intel on this one is that it appears to be dependant on the server hardware. Or at least, when you run the very same server on two different boxes, there will be different maps affected.
let's say there's server [A] and it's crashing in pregame on maps like veil veil_five mostly and oddly on some customs +summit.
We move server [A] to hardware and suddenly there's yana, orbital that crash everytime and oddly uplift.
These are real observations.
Here comes server [C] and crashes in pregame a lot on mineshaft, almost always when one of the ingame votes is called/succeeds. Disabling all votes helps... a bit.
It's just like how the now fixed (worked around) workshop issues affected servers (in alternating geographical locations) differently.
Comments
Although looking at its lua it only hooks into VoteRandomizeRR, not VotingForceEvenTeams.
[372.322] Main : Error: lua/Team.lua:153: expected self tobe a Type
[Server] Script Error #5: lua/Team.lua:153: expected self tobe a Type
Call stack:
#1: GetOwner [C]:-1
#2: StartVote lua/Team.lua:153
player = "VoteRandomizeRR"
#3: lua/Voting.lua:114
client = ServerClient { }
message = {client_index=0, expireTime=0, voteId=0 }
[375.670] Main : Error: lua/Team.lua:153: expected self tobe a Type
[Server] Script Error #6: lua/Team.lua:153: expected self tobe a Type
Call stack:
#1: GetOwner [C]:-1
#2: StartVote lua/Team.lua:153
player = "VotingForceEvenTeams"
#3: lua/Voting.lua:114
client = ServerClient { }
message = {client_index=0, expireTime=0, voteId=0 }
Got those during a pregame, they might help point to the origin.
What remains are the built-in votes. Those need to be looked at.
That is a really odd call-stack. Wonder if it's some issue with luajit doing something weird.
1. They started appearing on September 2nd.
2. They appear during pregame only, maybe at a 3% rate. Meaning mostly they don't appear.
3. There's not just those two but also for the other 3 votes. So it happens for all vanilla votes.
There's a lot in common with the preconditions for the pregame crash. I'd say they are a hint.
You are already in posession of 15-20 dumps of those crashes, and yes the last thing in those is a luaj call according to Ghoul.
Additionally -and this is like a ghost story-, we've always noticed a slight tendancy for the pregame crash to happen on veil and veil five specifically. It also happens on random other maps, albeit much more rarely.
Recently we've added veil five again and out of 8 times it got voted, 4 times pregame crash. Now the map is out again.
[9258.987] Main : Error: lua/MixinDispatcherBuilder.lua:230: attempt to call field '__towatch' (a nil value) [Server] Script Error #1: lua/MixinDispatcherBuilder.lua:230: attempt to call field '__towatch' (a nil value) Call stack: #1: StartVote lua/MixinDispatcherBuilder.lua:230 functionName = "OnGetIsSelectable" allFunctionsTable = {1=function, 2=function } self = "VoteKickPlayer" count = 2 className = nil #2: lua/Voting.lua:114 client = ServerClient { } message = {client_index=0, expireTime=0, kick_client=14, voteId=0 }EDIT: Same on ns2_orbital.
Nov 2 17:04:45 gemini kernel: [16123001.700977] server_linux32[18836]: segfault at 200354 ip 00000000f6be9e11 sp 00000000ff96cd78 error 4 in liblua51.so[f6b9b000+8e000]
Nov 2 23:33:25 gemini kernel: [16146300.109188] server_linux32[20697]: segfault at 194 ip 00000000f7245fae sp 00000000ffcecd10 error 4 in libSpark_Core.so[f7040000+6d8000]
Nov 3 00:29:29 gemini kernel: [16149660.478617] server_linux32[21446]: segfault at 0 ip 00000000f6b9d3db sp 00000000ff9059c0 error 4 in liblua51.so[f6b2c000+8e000]
I suspected it was NS2Stats. After disabling it, no more crashes. I meant to post this earlier but I haven't had time, I've been busy playing on the server.
Ironically, I've also had tons of crashes in the past of couple of weeks. Especially during mapchanges. Decided to take a proper look today and what do you know, disabling [SHINE]NS2stats solved the problem. Not a single crash in over 24 hours. ;o
But the io stream issues seem to still be there when it comes to reading larger json files.
Have you tried changing the settings yet? I'm going to toy around with them tonight.
"Awards":false,
"LogChat":true,
"StatusReport":true
What settings did you use?
Saw this in the previous server log:
[66295.006] Main : Error:
EntityChangeMixin.optionalCallbacks =
{
OnEntityChange = "Called when an entity changes into another entity or is destroyed"
}
if Server then
/**
* Pass in Id of new Entity this Entity is turning into or nil if it's being deleted.
*/
function EntityChangeMixin:SendEntityChanged(newId)
// This happens during the game shutdown process, so don't force a new game
// rules to be created if one doesn't already exist.
if GetHasGameRules() then
[Server] Script Error #1:
EntityChangeMixin.optionalCallbacks =
{
OnEntityChange = "Called when an entity changes into another entity or is destroyed"
}
if Server then
/**
* Pass in Id of new Entity this Entity is turning into or nil if it's being deleted.
*/
function EntityChangeMixin:SendEntityChanged(newId)
// This happens during the game shutdown process, so don't force a new game
// rules to be created if one doesn't already exist.
if GetHasGameRules() then
Call stack:
#1: read [C]:-1
#2: OnScriptLoaded lua/EventTester.lua:169
fileName = "lua/EntityChangeMixin.lua"
file = userdata
#3: Load lua/EventTester.lua:193
fileName = "lua/EntityChangeMixin.lua"
reload = nil
#4: lua/CompMod/NewTech/TunnelExit.lua:16
#5: scriptLoad [C]:-1
#6: Load lua/EventTester.lua:191
fileName = "lua/CompMod/NewTech/TunnelExit.lua"
reload = nil
#7: lua/CompMod_Shared.lua:11
ModFiles = {1="lua/CompMod/NewTech/GorgeTunnelEntranceAbility.lua", 2="lua/CompMod/NewTech/GorgeTunnelExitAbility.lua", 3="lua/CompMod/NewTech/HeavyMachineGun.lua", 4="lua/CompMod/NewTech/TunnelExit.lua", 5="lua/CompMod/NewTech/WalkMixin.lua", 6="lua/CompMod/NewTech/Welder.lua" }
(for index) = 4
(for limit) = 6
(for step) = 1
i = 4
#8: scriptLoad [C]:-1
#9: Load lua/EventTester.lua:191
fileName = "lua/CompMod_Shared.lua"
reload = nil
#10: lua/CompMod_Server.lua:4
#11: scriptLoad [C]:-1
#12: Load lua/EventTester.lua:191
fileName = "lua/CompMod_Server.lua"
reload = nil
#13: lua/ModLoader.lua:116
modEntries = {1= {Client="lua/CompMod_Client.lua", ModName="CompMod", Predict="lua/CompMod_Predict.lua", Priority="100", Server="lua/CompMod_Server.lua" }, 2= {Client="lua/NS2Plus/CHUD_Client.lua", ModName="CustomHUD", Predict="lua/NS2Plus/CHUD_Predict.lua", Priority="50", Server="lua/NS2Plus/CHUD_Server.lua" }, 3= {Client="lua/Badges+_Client.lua", ModName="Badges+", Priority="4", Server="lua/Badges+_Server.lua" } }
entryFiles = {1="lua/entry/Badges+.entry", 2="lua/entry/CompMod.entry", 3="lua/entry/CustomHUD.entry" }
ParseEntryFile = function
SortByModPriority = function
(for index) = 1
(for limit) = 3
(for step) = 1
i = 1
modEntry = {Client="lua/CompMod_Client.lua", ModName="CompMod", Predict="lua/CompMod_Predict.lua", Priority="100", Server="lua/CompMod_Server.lua" }
#14: scriptLoad [C]:-1
#15: Load lua/EventTester.lua:191
fileName = "lua/ModLoader.lua"
reload = nil
#16: lua/PostLoadMod.lua:12
#17: scriptLoad [C]:-1
#18: Load lua/EventTester.lua:191
fileName = "lua/PostLoadMod.lua"
reload = nil
#19: lua/Server.lua:462
chatMessageCount = 0
reservedSlots = {amount=0, ids= { } }
kMapEntityLoadPriorities = {ns2_gamerules=1 }
GetMapEntityLoadPriority = function
DumpServerEntity = function
LoadServerMapEntity = function
CheckForDuplicateLocations = function
OnMapPostLoad = function
OnCanPlayerHearPlayer = function
OnCheckConnectionAllowed = function
let's say there's server [A] and it's crashing in pregame on maps like veil veil_five mostly and oddly on some customs +summit.
We move server [A] to hardware and suddenly there's yana, orbital that crash everytime and oddly uplift.
These are real observations.
Here comes server [C] and crashes in pregame a lot on mineshaft, almost always when one of the ingame votes is called/succeeds. Disabling all votes helps... a bit.
It's just like how the now fixed (worked around) workshop issues affected servers (in alternating geographical locations) differently.