Revenge of the Failed to allocate ? bytes crash

krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
What happens: I am in started game (this time I was spectating, crash before that I just evolved Onos). Suddenly red plug appears(often I still hear other people communicating, sometimes I still see other people moving chopily, but I am out of sync - can move but cannot do anything else) and after about half a minute game crashes with the Fatal error: failed to allocate X bytes message. X is I think random number. Happens rather rarely (every ~10 h) and after extended session I think (> 1 h after running game). Somtimes alt-tabing works poorly, and actually shows the error message with invisible text.
My reporter dump (game was still running, showing the fatal error message): https://dropbox.com/s/zfz947sfmlu1wxl/tech_support.zip?dl=0
«1

Comments

  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    That server, Wooza's playground, is using more than just a few server mods - it is also running a hacked executable (and is using a lower tickrate than default) to increase the playercount beyond what the game was intended for. As a result, there is a good chance that the issues you have been experiencing are from playing there. Test on other servers and report back what you experience.

    If the game crashes, run that techsupport.exe with NS2 still open in the background.



    p.s.
    "The Celeron is a family of microprocessors from Intel targeted at the low-end consumer market" - Intel Wiki
    Your CPU is more than 6 times less powerful than CPU's that were released in the same month. It barely passes the minimum requirements for the game!
    Which is a shame considering how relatively nice your GPU (video card) is.. Very odd combination of parts, there. ;) I highly recommend upgrading your CPU to take full advantage of the rest of your system.
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    Oh. Ok, I will watch if that happens on other servers too from now on. But common software engineering sense tells me that server should never be able to crash a client (aka robustness), not to mention security, if server can cause faulty memory allocation.

    Dont judge me. STOP IT. I can feel you judging me right now :) My notebook died and I needed cheap PC quick, I was lazy to upgrade it to proper thing yet. Actually the game runs quite well, if there's not 20 people in the same place mashing themselves chaotically (I avoid that anyway, I suck in furball fight), so THANKS for that Devs (Actually I accidentally run it with bad Windows power plan for some time, cutting CPU to 80%, and still good, on vannilla < 24 players servers no performance problems whatsoever). I am talking just about FPS of course. Could poor CPU cause those annoyng instakills, by any chance (does the game have simulations per second not tied to FPS)? That is do I get worse physics, without noticing it in game smoothness?

    Also: I did run the techsupport.exe while ns2.exe still open with the fatal error message. It clearly stated it detects it and is making dump. But the zip does not contain anything I would consider a memory dump, which I think you need. The log.txt just shows some lua stack trace of some error....
    Also: After the fatal error message the automatic error reporter start telling me to check log.txt for private things before send, I guess. But the send takes too long to be just log.txt (several minutes). Is it sending other data?
    Also: I noticed the installation folder contains 200 MB of *.pdb files. Does it make sense for me as end user to have theese in my installation.
    Are those three bug report worthy?
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    edited February 2015
    Hard to tell for sure whether your CPU could be causing some of those issues, but it is definitely possible.

    Yea techsupport.exe should make a folder called dump or minidump inside the zip when it's done, and of course will be a large file size.
    If it's not... well.. that makes things slightly more difficult, as you have to run the 32 bit version of Windows Task Manager (it's hidden away in a windows folder) and manually click on the process and create a dump. (which will be larger than the one techsupport.exe makes..)

    Dont bother with the automatic error reporter. As the Community Development Team we do not have access to where it is sent, unfortunately. So we can't guarantee anyone is reviewing what you send..
    Don't worry about the pdb files , and yes all information is worthy. ;)
  • MephillesMephilles Germany Join Date: 2013-08-07 Member: 186634Members, NS2 Map Tester, NS2 Community Developer
    btw I have the same problem and get this on all sorts of servers regulary.
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    I hate it when things are hard to tell :)

    Well it didn't. I can certanly make dump file your way, if thats useful to the team. By "hidden" I assume you mean SysWOW64 folder.

    Well I am honored to store the games pdb debug symbols. :D
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    @Mephiles Then follow the instructions I posted here, we need data to solve it.

    @krOoze That's the one, thanks :)
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    edited February 2015
    Ok, I got another crash at Woozas. Too bad, I was playing at another server a minute before that...
    Anyway the reporter lied again about making a dump:
    dump.PNG?raw=1

    To be exact, I am sure it made a dump (process froze for a few seconds), but I don't know what it has done with it. It's not in the zip...

    report: https://dropbox.com/s/f6rfoei3121zpmd/tech_support2.zip?dl=0
    windows dump file: https://dropbox.com/s/njx766s51mp8d6e/NS2.7z?dl=0
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    edited February 2015
    IronHorse wrote: »
    Thanks!
    Of course, have fun with it. I hope you will have "luck" catching this one this time. It would be logisticaly easier for me to do a minidump. Is that sufficient to you, if there is a next time I post?

    @Doctik Well thanks for forcing me to read the last thread in entirety - that led nowhere fast. My network is rock stable, thanks for asking. Actually I haven't got any crashes at the time of the old thread and it was many patches ago - so lets not zombie that one, ok? (Hence the "Revenge of". Get a hint, man :) ) My problems started I thing after the optimization patches. The servers stoped crashing, hitching and that funny restarting to your old position thing(thanks devs). My guess would be that you run out of 32-bit address space, your in process network stack fails. Then it is a race what happens sooner timeout or memalloc. OR the network stack disconnect you for no reason whatsoever starts leaking memory(or interp/rendering leaks - your oponents fly into infinity without connection, who knows what happens when that happens). Than the error reporter is 32-bit too so if it tries copying game memory it fails to allocate too, and fails to report it. But you know: Never tell developers your naive speculations, but facts only. And keep this thread clean this time, pleeease <kitty image />.
  • woozawooza Switzerland Join Date: 2013-11-21 Member: 189496Members, Squad Five Blue
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    edited February 2015
    @wooza Maybe a little more interpretation. Is the claim true/false? Should/shouldn't it cause client crash or at least increased memory usage?
  • woozawooza Switzerland Join Date: 2013-11-21 Member: 189496Members, Squad Five Blue
    @krOoze The Server rates are back to default like other servers have (30 Tickrate is default, we run it before with 25 instead to save some server CPU performance). So yes the claim is false.

    With 42Players there is definetly more to display / calculate for your gpu/cpu so yes, it may use a bit more memory usage than on other servers i guess.

    About your problem, it happened to me too when i started playing the game the first times (But really rarely). I don't know exatlcy what i did to fix this but i remember i had a defective ram at this time. I used the programm MemTest. If you using it, start multiple instances so that almost every megabyte is checked
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    Uhh.. on the dates he reported having issues (Feb 13th and 15th) you were using less than the default tick rate as well as a hacked executable to increase the maximum player count.... So how exactly is what I said false?

    http://ns2servers.devicenull.org/servers/1530213/85.6.202.192:27016/?date=02/13/2015

    You have changed it since then, but not the hacked server executable. (or mods I'm assuming)

    Unsurprisingly, your server consistently produces false positives for tech support related issues. Therefore, it cannot be used as a testing environment.

    @krOoze please continue to test long sessions on other servers to attempt to reproduce the issue. I've forwarded your dumps to some devs, just waiting on what they found, if anything.
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    @wooza Unlikely, but what the hell... Did "Extended test" on RAM. No errors...
    @IronHorse I'm on it! :) I must get some sleep, but I will try to as a lil experiment let it run overnight with some bots. Maybe it will crash too....

    PS Woozas servers are always full, even when others are not and I had no (other) problems with them, so maybe make them legit in the long run (that is without hack, you speak of)? You already know about them and warn about potential performance issues in game anyway. Sometimes there is not even many alternatives where to play...
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    Yea... we've already worked with them a bunch in the past to help fix other issues and bottlenecks that are specific to their modifications - it should be a lot better now - but there are still more issues of course, just like there are with any server side modifications. The issue is that we have very limited resources as is, so dedicating more time to testing and fixing issues that are from modifications that are outside of our scope, and preferably should not occur in the first place, (and potentially change constantly anyways) is just not realistic....
    Not to mention we don't have 42 users consistently to even begin testing such a thing :)

    We definitely try, and wish to foster all environments, despite any non-preferable change in gameplay or balance, we do test with mods and attempt to support all - but maintaining stability for the mod authors is unfortunately just not realistic - we already have our hands very full with the vanilla version of NS2. ;)

    Thanks again for the helpful info, i'll update you when I have more.
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    edited February 2015
    Another crash, but at "House of awesome". This time empty "Fatal Error" message appeared with empty button. After collecting techreport and memdump I clicked the button. The same message apperared several times in a loop and then the game seemed to recover to main menu. The log still contains "failed to allocate", so I guess this is still good thread to post it.
    It was again preceded by minute long red plug. Again I could hear well other players speaking and observed no performance drop during that time, but could only walk around desynced.

    On first Fatal Error message:
    TR zip: tech_support3.zip
    dump: maxidump3.7z

    After recovery and exit:
    TR zip: tech_support4.zip
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    Shouldn't have complained - the game hates me now :D . Another crash. Server Alliance Franco-Belge. tech_support5.zip, maxidump5.7z
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    So supposedly what is happening for you is a bunch of memory leaks.
    Pointing towards dx9 and network related.. but no promises.
    At least we have the data on hand, though, thanks again.

    I will update when we have more
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    edited February 2015
    mem.PNG?raw=1
    Yea, actually I investigated it a bit myself. When the red plug appears, the quick runaway memory leak starts (or vice versa). Alt-tabing seems to slow it down. It crashes with the bad memalloc error when it reaches 4GB (for the obvious reasons). Sometimes it just ends with ingame timeout message or "verify your cache" message (like in the graph above), especially if you alt-tab during the red-plug time. (I verified and nuke reinstalled it several times now.)

    Don't hesitate to ask if you need me to test something, or further data.

    PS: Oh didn't realize that everybody doesn't speak czech. Red is processor utilization, green is working set of the NS2 process.
  • DC_DarklingDC_Darkling Join Date: 2003-07-10 Member: 18068Members, Constellation, Squad Five Blue, Squad Five Silver
    I strongly wonder how your ns2 can even hit 4GB.. Im not saying it does not, im wondering HOW.
    Im playing on quite high settings on 1080p, and I purposely attempted to hit 4GB to break it, and never got close.

    WHATS happening in your executable that its hitting 4GB, makes me wonder.
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    edited February 2015
    @DC_Darkling Read the graph in previous post, and watch the timeline. It does it by allocating memory uncontrolably due to some bug or other. Arguably the new memory doesn't contain anything useful, so it is not dependent on memory requirements of resolution or quality settings. It's relatively easy to make coding error like that (or use library that has such an error). It is odd that not everybody has the error, but not inexplicable (only those with certain HW and OS and running environment, game settings or corrupted settings files enforced/undeletable by Steam Cloud could be exposed to the faulty codepath). Also I could be doing unwitingly something the game is not prepared for too and corrupting files in the process (like alt-tabing too much or alt-f4 out of game). Also, you know, every public server uses tons of mods, which could be faulty.
    My guess would be that the devs would have hard time finding the culprit, since in the dumps they probably would see only the last memory allocation, that tipped it over 4GB, which likely would not be the cause of the error in the first place. They would need to dig through it to see the actual error source, if they cannot reproduce it themselves on their machines. The red plug could be the biggest hint and finding what stalls the netcode could mean a fix soon (fingers crossed :) ). Hope I sated your curiosity.

    But I kinda wonder how it is the game always uses ~1.5 GB during normal behavior, no matter what I do :).
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    We think we know what it is in general, but not specifically.
    Definitely looking into it, though.
  • DC_DarklingDC_Darkling Join Date: 2003-07-10 Member: 18068Members, Constellation, Squad Five Blue, Squad Five Silver
    aha, I see the edit now of what the green and red lines mean.
    The values in the graph still mean nothing to me, but I guess I get the general idea.

    You could attempt to keep a eye live on memory usage and try to find out 'what' is eating the memory.
    Sadly that would only boil it down to a handle or ddl so I am unsure if that is useful info.

    Also debugging that externally can get quite tricky. IF you are unfamiliar with it, im not sure how helpful it is if I try to explain it.
    Best I can do is link you to the article explaining it, but note that it will not be of much use if you do not understand the underlying information:
    http://blogs.technet.com/b/markrussinovich/archive/2009/03/26/3211216.aspx
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    edited February 2015
    The values in the graph still mean nothing to me, but I guess I get the general idea.
    Green line and Y axis are total memory usage of NS2 in MB (not MiB). Red line should be promile usage of CPU by NS2 (don't ask me why 100 % is around 1600). X axis is time in hh:mm:ss. In the circle the red plug appears. At the CPU drops I was alt-tabed to desktop. The memory drop at the end is when the game recovered with timeout message that time (sorry, did not catch actual full blown crash when I was measuring). It's not exactly rocket science.
    You could attempt to keep a eye live on memory usage and try to find out 'what' is eating the memory.
    Way ahead of ya. I had only one two crashes from last time and tried "memory" command in console. For that crashes it seems the program spammed someting it calls NetworkMemory of a ClientWorld, and every memory seems to be allocated by Spark engine, so no point looking which dll does that. So better leave it to the devs, they will see that and much more, since they have both the full dump from me and game source code. I am not quite so desperate yet to try to fix it in asembly language.

    I uninstalled Fraps (which I was not using, but it still pushes its meddling dll into everything) and had peace from crashes for now (perhaps statistical fluke, perhaps I did something else. will try to reinstall it in a week or so, if I get none till then). So I sugest to anyone having the same problems to uninstall Fraps and similar software and report back too.
  • DC_DarklingDC_Darkling Join Date: 2003-07-10 Member: 18068Members, Constellation, Squad Five Blue, Squad Five Silver
    Ok I dived into my own dxdiag to be sure... yours is... off.

    To start the available memory in dxdiag itself is to be ignored. It basicly says how much memory the system has and how much the OS can use, not how much there is left. (I was in doubt as I never check memory by dxdiag but thats how its viewed)
    In this regard so far so good. You have 8GB of memory and almost all is used for the OS, meaning no driver is going nuts on reservations for hardware.

    The odd part is your page file. Many rumors exist about the size of a pagefile but it truly boils down to 'what you need'.
    Your very first post has a dxdiag containing the following:
    Page File: 7735MB used, 666MB available.

    Thats well over 7GB of paged memory, not accounting for the non paged memory which ideally should be pretty full also. To compare, idle my pc pulls a 2.5GB of paged memory.

    While thats impressive memory usage, which makes me wonder where it all goes as ns2 cant use THAT much GB even with large address awareness, the real question is.. why do you have 'only' 666MB left? Was it only 666 during the crash, or less?



    I now have a small assumption / theory that you may be hitting either the commit limit or the paged pool limit (which are almost identical).
    Does not mean ns2 is not at fault, but this can not be ns2 only.
    Start be severely raising your commit limit by setting a bigger pagefile manually, as something in your pc wants all that paged memory. (I know its auto due to its changing max size)
  • Perman12Perman12 Campuchia Join Date: 2015-01-31 Member: 201130Members
    I have REd Plug + Crash "Failed to allocate 20047 bytes and now will be terminated"

    After play NS2 for about exactly 30 min ( or map change one) >> Crash to destop=='

    P/s: Using Window7 32bit
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    edited February 2015
    @DC_Darkling :
    First of all dxdiag shows nonsense. What it shows is not "Page File". It is known! :)
    It is most likely backed virtual space(RAM + page file - not acounting for possible page file expansion) and commited memory(=not only used, but reserved and cached too, possibly drivers' adress spaces also).
    Windows warns when what you describe happens. Besides I have page file expansion on and free disk space.
    Something within NS2 is clearly at fault. It is using 1.5 GB, then something breaks, then it starts allocating memory linearily untill it's stopped by something (e.g. running out of 32bit Large address aware space = 4GB). And my dumps I sent are 4GB so no doubt there.

    https://dropbox.com/s/pcnbosxv946mpl6/tech_support7.zip?dl=1
    I actually managed to catch it in log using "memory" command (and "verbose 3", so it shows some weird silenced errors too) Thats how I know it bloats NetworkMemory of a ClientWorld and possibly another unaccounted memory not managed by Spark engine e.g. DirectX. Coincidentaly:
    IronHorse wrote: »
    Pointing towards dx9 and network related..
    So we seem to be little behind the devs in our conspiracy theories. :)

  • DC_DarklingDC_Darkling Join Date: 2003-07-10 Member: 18068Members, Constellation, Squad Five Blue, Squad Five Silver
    yeh I know virtual ram is both.. and yet again its not, depends if its paged.. But seeing as you seem to know also lets not get in that one haha. ;)
    I was just wondering as your virtual memory usage is... quite high.

    Yeh if the dumps are 4GB... didnt check their size. I only tried some memory commands which obviously failed as its not a system dump but a program dump.
    I should really not try to be helpful when im ill and not fully lucid. :D


    Hmm I see what you mean, network memory is quite high.. (I should really just run verbose 3 myself a few times, been to long)
    Which is why I am not sure of the following but damn.. thats a lot of errors on the prediction, with network memory rising fast.
    Did you try dx11 or is your amd to old for that? (just curious, I know dx11 has input lag in ns2)
  • krOozekrOoze Join Date: 2014-04-24 Member: 195593Members
    edited February 2015
    Well the bogus value in dxdiag shows 2.5 in idle. If you add the 4 for crashed ns2 it does +- match what is in the log.

    Best ideas usually comes, when a person is not lucid. :D

    Nah, my GPU is new and shiny. I played on dx11 some time ago. It was not exactly stable either and run slightly slower. I think the frame pacing was better though.
    I still think the Fraps could possibly trigger that memory leak somehow. I forgot I installed AMD Gaming Evolved/raptr, which does the same thing. (Un)fortunately I do not get the crashes frequently and regularily enough to test it easily.
Sign In or Register to comment.