[Linux] Natural Selection 2 keeps crashing when shooting.

CommanderAlchemyCommanderAlchemy Sweden Join Date: 2013-12-04 Member: 189780Members
edited August 2014 in Technical Support
Right i know that there was another thread but was later closed as fixed. I posted about this and some other crashing issues on another thread but it was related to optimization.

Well after latest update i removed all config files related to NS2 and downloaded the whole game again from Steam the game keeps crashing when shooting weapons or hatching. I decided to do a video of the occurrence to just clarify the issue and try to collect the most amount of info i can about this. If there is something missing please tell.

I also posted this on nVidia forums to maybe get a response from their devs located here:
https://devtalk.nvidia.com/default/topic/770053/linux/natural-selection-2-keeps-crashing-when-shooting-

The game was running the lowest settings and effects that could be turned off in the graphics options. Everything else is the stock settings from a fresh install.

Info/Logs
Steam systeminfo:
https://gist.github.com/CommanderAlchemy/83c70f2deffb678192c3

Dmesg error:
[77133.261726] ns2_linux32[15536]: segfault at 0 ip 00000000f151f3e7 sp 00000000ffac1680 error 4 in libnvidia-glcore.so.343.13[efdc8000+1925000]

nVidia-bug-report:
Check the devtalk.nvidia.com post i did, linked above it contains allot of information and logs if the steam systeminfo is not enough.

log.txt (from ~/.config/....
https://gist.github.com/CommanderAlchemy/9cbde0308b5814da0ef4


Video of the issue
«13

Comments

  • ZaggyZaggy NullPointerException The Netherlands Join Date: 2003-12-10 Member: 24214Forum Moderators, NS2 Playtester, Reinforced - Onos, Subnautica Playtester
    edited August 2014
    [170344.823198] ns2_linux32[26840]: segfault at 0 ip 00000000f15d34e6 sp 00000000ffb1c120 error 4 in libnvidia-glcore.so.340.32[efeb8000+18e9000]

    Also when biting with skulk:

    [170953.562559] ns2_linux32[27104]: segfault at 0 ip 00000000f15ff4d1 sp 00000000ff7f7260 error 4 in libnvidia-glcore.so.340.32[efee4000+18e9000]
  • CommanderAlchemyCommanderAlchemy Sweden Join Date: 2013-12-04 Member: 189780Members
    Do you have issues aswell when shooting different guns etc? I notcied same crash when hatching and showing you got different driver it seems that this is more of a general issue? I wonder if running this game in wine with opengl would have the same issue.
  • CommanderAlchemyCommanderAlchemy Sweden Join Date: 2013-12-04 Member: 189780Members
    Just tried v269 issue seems to remain. Though it seems that nVidia devs are looking into the issue atleast it's in their backlog.
    https://devtalk.nvidia.com/default/topic/770053/linux/natural-selection-2-keeps-crashing-when-shooting-/?offset=6#4303486
  • WeTDreamzWeTDreamz Join Date: 2013-08-30 Member: 187247Members
    This also happens to me. I tried playing offline today and the second the rifle shot, the game crashed. I'm using an AMD video card which is running in software mode for some reason. I doubt this is an nvidia problem.
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    Paging more Linux savvy members than myself @Asraniel‌ and @murray‌
  • CommanderAlchemyCommanderAlchemy Sweden Join Date: 2013-12-04 Member: 189780Members
    edited September 2014
    WeTDreamz wrote: »
    This also happens to me. I tried playing offline today and the second the rifle shot, the game crashed. I'm using an AMD video card which is running in software mode for some reason. I doubt this is an nvidia problem.
    Dunno seems that nVidia devs found some issues and are wroking on fix. If its the same issue you got, driver issue that triggers both nVidia and AMD drivers, or maybe more general "game" issue dunno :/

    Though kudos for nVidia and sandipt for responding and working so fast with this issue since I posted it on their forums. Lets hope it's "the" issue that causes crashes and not some yet another bug :P
    sandipt wrote:
    We are able to reproduce this issue and our engineers are working to fix it
    https://devtalk.nvidia.com/default/topic/770053/linux/natural-selection-2-keeps-crashing-when-shooting-/?offset=6#4303486
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
    Well that's good news :)
    Thanks @CommanderAlchemy‌ for doing that
  • ZeitgeistZeitgeist FFM Join Date: 2014-05-20 Member: 196148Members
    Good job. I can't really play the game either it crashes or mouse movement is laggy or both. Loading times are way to long. But it has improved since the last patch.
  • CommanderAlchemyCommanderAlchemy Sweden Join Date: 2013-12-04 Member: 189780Members
    @IronHorse‌ @Asraniel‌ @murray‌

    There is an answer on the nvidia thread about the crashing issue, seems it's the game doing some bad things that it should not do.
    ahuillet wrote:
    We have investigated the issue and found that the crash is due to an application bug. It crashes upon calling glDrawElementsInstanced() because it has incorrectly set the vertex attribute pointers to NULL.
    Below is a trace of the OpenGL command stream that leads to the crash. You will note that no VBO is bound, and the application gives a NULL pointer to attribute 6, which by itself is enough to trigger the crash. The application also gets GL_INVALID_OPERATION for the next attributes because of this part of the OpenGL specification:
    `An INVALID_OPERATION error is generated if a non-zero vertex arrayobject is bound, no buffer is bound to ARRAY_BUFFER, and pointer is not NULL.`

    Looking at the command stream, it's likely that the call that unbinds the VBO should simply not be there: experiments on our side show that ignoring this call prevents the crash and doesn't appear to yield rendering artifacts. Of course such experiments are not spec conformant and cannot be integrated to the NVIDIA driver. An end-user might want to try intercepting glBindBuffer(GL_ARRAY_BUFFER, 0) with LD_PRELOAD, and ignoring this call, to work around the issue until the application is fixed.


    glBindBuffer(GL_ARRAY_BUFFER, 0);
    glVertexAttribPointer(6, 3, GL_FLOAT, 0, 76, (nil));
    glVertexAttribDivisor(6, 1);
    glVertexAttribPointer(7, 3, GL_FLOAT, 0, 76, 0xc);
    // **********
    // ERROR: 1282 = 0x502 (GL_INVALID_OPERATION)
    // **********
    glVertexAttribDivisor(7, 1);
    glEnableVertexAttribArray(8);
    glVertexAttribPointer(8, 3, GL_FLOAT, 0, 76, 0x18);
    // **********
    // ERROR: 1282 = 0x502 (GL_INVALID_OPERATION)
    // **********
    glVertexAttribDivisor(8, 1);
    glEnableVertexAttribArray(9);
    glVertexAttribPointer(9, 3, GL_FLOAT, 0, 76, 0x24);
    // **********
    // ERROR: 1282 = 0x502 (GL_INVALID_OPERATION)
    // **********
    glVertexAttribDivisor(9, 1);
    glEnableVertexAttribArray(10);
    glVertexAttribPointer(10, 3, GL_FLOAT, 0, 76, 0x30);
    // **********
    // ERROR: 1282 = 0x502 (GL_INVALID_OPERATION)
    // **********
    glVertexAttribDivisor(10, 1);
    glDrawElementsInstanced(GL_TRIANGLES, 84, GL_UNSIGNED_SHORT, (nil), 1);
    https://devtalk.nvidia.com/default/topic/770053/linux/natural-selection-2-keeps-crashing-when-shooting-/
  • develdevel Join Date: 2014-09-13 Member: 198444Members
    Have you encountered similar crashes while not shooting?

    I'm having relativly rare random crashes. Want to know if it is the same thing.
    (will post details later)
  • CommanderAlchemyCommanderAlchemy Sweden Join Date: 2013-12-04 Member: 189780Members
    devel wrote: »
    Have you encountered similar crashes while not shooting?

    I'm having relativly rare random crashes. Want to know if it is the same thing.
    (will post details later)

    Cannot say, nVidia found some issues with the game rendering code so it could possibly affect other things and probably other vendors.
  • IronHorseIronHorse Developer, QA Manager, Technical Support & contributor Join Date: 2010-05-08 Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
  • JektJekt Join Date: 2012-02-05 Member: 143714Members, Squad Five Blue, Reinforced - Shadow
    Daaaaamn nVidia linux support. That's awesome.
  • rkfgrkfg Russia Join Date: 2013-09-03 Member: 187744Members, Reinforced - Supporter, Reinforced - Silver, Reinforced - Gold, Reinforced - Diamond, Reinforced - Shadow
    I'm yet to see another vendor who actually finds bugs in a 3rd party proprietary game just like that. Purely awesome!

    BTW, it's really as I assumed — a null pointer passed to the driver's function. Not that I'm bragging about it... ah, ok, I'm bragging. Though it was an obvious outcome anyway.
  • ViVnetViVnet Join Date: 2004-05-27 Member: 28947Members, Reinforced - Shadow
    I can confirm the workaround suggested by nvidia appears to correct the issue.

    I have not been able to reproduce the crash after several hours of gameplay.
  • warpedwarped Join Date: 2013-08-07 Member: 186617Members
    ViVnet wrote: »
    I can confirm the workaround suggested by nvidia appears to correct the issue.

    I have not been able to reproduce the crash after several hours of gameplay.

    Awesome news.

    Would it be possible to provide some quick information on setting this up?
    I am vaguely familiar with LD_PRELOAD, but not in terms of intercepting this.
  • ViVnetViVnet Join Date: 2004-05-27 Member: 28947Members, Reinforced - Shadow
    edited October 2014
    It can be done by preloading the following code. At least, it appears to be working for me.

    You should be willing to take any risk that may be associated with this, certainly not recommended for the average user. In my case I do not crash at all when using this and I also do not notice any graphics or performance issues.

    I was also told that this specific code is not a cheat and VAC will not take action against it. (By who I'm 99% sure is a Valve employee, but this information should still be considered speculative and risky)

    Compile it with:
    gcc -fPIC -c -o preload.o preload.c
    gcc -shared -o preload.so preload.o -ldl

    Load it by adding the following to NS2's launch options in steam:
    LD_PRELOAD=$LD_PRELOAD:/path/to/preload.so %command%
    // preload.c
    
    #include <dlfcn.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <GL/gl.h>
    
    void __attribute__ ((constructor)) load(void);
    
    // Called when the library is loaded and before dlopen() returns
    void load(void)
    {
        fprintf(stderr, "PRELOAD: Intercepting glBindBuffer(GL_ARRAY_BUFFER, 0)\n");
    }
    
    
    typedef void (* glBindBuffer_func)(GLenum target,  GLuint buffer);
    
    void glBindBuffer(GLenum target, GLuint buffer) {
    
        void *gl_handle;
    
        if(target != GL_ARRAY_BUFFER || buffer != 0) {
            gl_handle = dlopen("libGL.so", RTLD_LAZY);
            glBindBuffer_func gbb = dlsym(gl_handle, "glBindBuffer");
            gbb(target, buffer);
        }
    }
    
  • rkfgrkfg Russia Join Date: 2013-09-03 Member: 187744Members, Reinforced - Supporter, Reinforced - Silver, Reinforced - Gold, Reinforced - Diamond, Reinforced - Shadow
    This function pointer should be probably cached so it's not being looked up every time the function is called. I suppose this function call happens quite often.
  • ViVnetViVnet Join Date: 2004-05-27 Member: 28947Members, Reinforced - Shadow
    Propose some changes if you wish, this was kindly provided to me by a couple friendly users on IRC/freenode. (I didn't write it) Community FTW!
  • rkfgrkfg Russia Join Date: 2013-09-03 Member: 187744Members, Reinforced - Supporter, Reinforced - Silver, Reinforced - Gold, Reinforced - Diamond, Reinforced - Shadow
    edited October 2014
    I'd rewrite it like this:
    // preload.c
    
    #include <dlfcn.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <GL/gl.h>
    
    void __attribute__ ((constructor)) load(void);
    
    // Called when the library is loaded and before dlopen() returns
    void load(void)
    {
        fprintf(stderr, "PRELOAD: Intercepting glBindBuffer(GL_ARRAY_BUFFER, 0)\n");
    }
    
    typedef void (* glBindBuffer_func)(GLenum target,  GLuint buffer);
    glBindBuffer_func gbb = NULL;
    
    void glBindBuffer(GLenum target, GLuint buffer) {
    
        void *gl_handle;
    
        if(target != GL_ARRAY_BUFFER || buffer != 0) {
            if (gbb == NULL) {
                gl_handle = dlopen("libGL.so", RTLD_LAZY);
                gbb = dlsym(gl_handle, "glBindBuffer");
            }
            gbb(target, buffer);
        }
    }
    
    Performance matters! In this case we only check if the function pointer is null and if it is, we search the actual pointer and store it in that var. This way the overhead should be negligible.

    And the compilation could be done in one pass with gcc -fPIC -shared -o preload.so preload.c
    I also have to add -m32 argument so it produces a 32-bit binary which is necessary as NS2 is 32-bit only and won't accept a 64-bit library.

    About the usage: place the preload.so file to the NS2 directory. Set the launch options for the game in Steam to this:
    env LD_PRELOAD=preload.so %command%
    
    You can then attach with gdb and check if the library is loaded with "info sharedlibrary" command. I'll be playing with it from now on and I'll report if it doesn't help. Crashes happen not so often for me so it may take a couple of weeks to confirm.
  • develdevel Join Date: 2014-09-13 Member: 198444Members
    Or declare-initialize gl_handle and gbb as "static" inside this function.
  • ViVnetViVnet Join Date: 2004-05-27 Member: 28947Members, Reinforced - Shadow
    edited October 2014
    We should note that I was directed to load the library in a way that does not clobber steam's preloading (consider the overlay).

    LD_PRELOAD=$LD_PRELOAD:/path/to/object.so
    rkfg wrote: »
    About the usage: place the preload.so file to the NS2 directory. Set the launch options for the game in Steam to this:
    env LD_PRELOAD=preload.so %command%
    

    For me, I can definitely say it works. If I stop loading it I crash within an average of 5-10 minutes of play. I've played more in the last two days than I've been able to play in the last six months thanks to nvidia!

    Also, this is the current version, you can find it here: https://github.com/ali1234/nvidiathing/blob/master/preload.c

    It seems he has also made the changes you suggested. Additionally, it should be using RTLD_NEXT instead of RTLD_LAZY. However, RTLD_NEXT requires '#define _GNU_SOURCE' before the dlfcn.h include (which isn't there) and even then it produces compiler warnings.

    I wasn't required to use the -m32 option.
  • rkfgrkfg Russia Join Date: 2013-09-03 Member: 187744Members, Reinforced - Supporter, Reinforced - Silver, Reinforced - Gold, Reinforced - Diamond, Reinforced - Shadow
    I suppose it should become more readable with the early return:
    if(target == GL_ARRAY_BUFFER && buffer == 0) {
            return;
    }
    if(!gbb) {
        gl_handle = dlopen("libGL.so", RTLD_NEXT);
        gbb = dlsym(gl_handle, "glBindBuffer");
    }
    gbb(target, buffer);
    
    This amount of refactoring for such a trivial and small piece of code amuses me.
  • AsranielAsraniel Join Date: 2002-06-03 Member: 724Members, Playtest Lead, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow, Subnautica Playtester, Retired Community Developer
    Great find, this looks very promising. Can you comment on any side effects? I would be worried that since some buffer no longer gets unbound, there might be a memory leak on the gpu. Do you see increasing memory usage or even crashing after some time due to that? The workaround looks promising and could be implemented directly in the engine, provided we find somebody with enough opengl knowledge to make sure that there are no sideeffects to this.
  • warpedwarped Join Date: 2013-08-07 Member: 186617Members
    Unfortunately did not work for me. Below is the journal log. Tried all variations discussed.
    Oct 10 03:27:36 kernel: ns2_linux32[1088]: segfault at 0 ip 00000000f15e0066 sp 00000000ff90c6c0 error 4 in libnvidia-glcore.so.343.22[efe82000+192c000]
    

    It is loading the PRELOAD script however as shown in the terminal line
    Setting breakpad minidump AppID = 4920
    PRELOAD: Intercepting glBindBuffer(GL_ARRAY_BUFFER, 0)
    warning: The VAD has been replaced by a hack pending a complete rewrite
    warning: The VAD has been replaced by a hack pending a complete rewrite
    Installing breakpad exception handler for appid(steam)/version(1412803545)
    Installing breakpad exception handler for appid(ns2_linux32)/version(1.0)
    warning: The VAD has been replaced by a hack pending a complete rewrite
    Game update: AppID 4920 "Natural Selection 2", ProcID 1088
    warning: The VAD has been replaced by a hack pending a complete rewrite
    warning: The VAD has been replaced by a hack pending a complete rewrite
    Installing breakpad exception handler for appid(steam)/version(1412803545)
    Game removed: AppID 4920 "Natural Selection 2", ProcID 1088
    

    Strange thing is, in pub servers, I can play for an hour or so without a problem, using shotguns the whole time. However, when I start a sandbox server up, it crashes without fail on the very first shot.
  • rkfgrkfg Russia Join Date: 2013-09-03 Member: 187744Members, Reinforced - Supporter, Reinforced - Silver, Reinforced - Gold, Reinforced - Diamond, Reinforced - Shadow
    Can you capture a backtrace then if it's reproducible so well? There are also great tools for tracing OpenGL calls like API Trace or VOGL. I suppose the NVIDIA guy used something like that to find the culprit.
  • rkfgrkfg Russia Join Date: 2013-09-03 Member: 187744Members, Reinforced - Supporter, Reinforced - Silver, Reinforced - Gold, Reinforced - Diamond, Reinforced - Shadow
    The crash is still reproducible in Sandbox mode, even when the library is preloaded. Not every time when I start the game but quite often. The problem is I can't capture it with apitrace, it doesn't happen when the game is running under it, FPS is also low (10-15) so it may be related. Like, because of a big time delta the frame with the invalid GL-call is just skipped and crash is avoided. Tried the same with VOGL, doesn't crash. Tried about 10 times without a single result, when I run the game without those hooks it crashes on first or second launch in Sandbox when I shoot the shotgun.
  • rkfgrkfg Russia Join Date: 2013-09-03 Member: 187744Members, Reinforced - Supporter, Reinforced - Silver, Reinforced - Gold, Reinforced - Diamond, Reinforced - Shadow
    Hmm, I also doubt the preload hack actually works. I added some debug output and it was never triggered throughout the game. The intro message ("preload hack loaded...) is there though and the library is shown in GDB as preloaded.
  • CommanderAlchemyCommanderAlchemy Sweden Join Date: 2013-12-04 Member: 189780Members
    I tried the code from the github.
    compiled with : gcc -fPIC -shared -o preload.so preload.c
    I preload it with: LD_PRELOAD=preload.so %command%
    Having the file in game dir.

    It prints out that it loaded but.
    It crashes at sandbox the way it did before. I added also a fprintf just before it should return in the and it never prints out.
      if(target == GL_ARRAY_BUFFER && buffer == 0) {
        fprintf(stderr, "ns2 is a lie\n");
        return;
      }
    


    Also note that it was one of the issues of the rendering code.

    From the nvidia dev,
    "It crashes upon calling glDrawElementsInstanced() because it has incorrectly set the vertex attribute pointers to NULL."

    "You will note that no VBO is bound, and the application gives a NULL pointer to attribute 6, which by itself is enough to trigger the crash"

    So i guess we would also need to do something about
    glVertexAttribPointer(6, 3, GL_FLOAT, 0, 76, (nil));


  • rkfgrkfg Russia Join Date: 2013-09-03 Member: 187744Members, Reinforced - Supporter, Reinforced - Silver, Reinforced - Gold, Reinforced - Diamond, Reinforced - Shadow
    The problem is, if you put that fprintf outside the if statement, i.e. to the very beginning of the hooked function, you won't see any output either. As I suppose there should be plenty of calls but either there are really none (which is not true according to the apitrace log, there are dozens of calls though with a bit different params) or they are not being hooked (which I suppose to be happening).
Sign In or Register to comment.