Great find, this looks very promising. Can you comment on any side effects? I would be worried that since some buffer no longer gets unbound, there might be a memory leak on the gpu. Do you see increasing memory usage or even crashing after some time due to that? The workaround looks promising and could be implemented directly in the engine, provided we find somebody with enough opengl knowledge to make sure that there are no sideeffects to this.
Coulnt you guys try to remove the nullpointers and the calls with null and then maybe release a testing "linux" branch on steam, Letting us test it if it works to get something moving?
Or maybe try contacting nVidia to get a better explanation of the issue.
If you guys don't have anyone with good OpenGL knowledge anymore maybe start a forum post here or contact http://www.phoronix.com/scan.php?page=home or maybe http://www.gamingonlinux.com/ and let them advertise that you guys are looking for a OpenGL dev. They will probably do it to help a game on Linux.
Btw is there anyone on Linux that is able to play this game anymore because of the faulty OpenGL code? Maybe update the information on Steam etc, about the Linux state to what it really is?
I bought this game because it "supported" Linux and honestly i'm really disappointed on the current state and that this issue has not been resolved in more than a year.
I'm playing daily on Debian (the details are in the signature) for several hours in a row, sometimes I get the shotgun crash but not even close to the point when I ragequit the game. Once per 2 weeks, maybe? I can crash almost instantly in the sandbox mode but that's it. The regular games are fine most of the time. Today, for example, a friend of mine had the game frozen completely and it even didn't allow to get to the desktop. She's playing on Windows though. I also hear often that it crashes from time to time on Windows, I even think that it's overall more stable on Linux (for me, of course). I have several other issues with cyrillics and mic and much bigger loading times, and it all was reported several times, but I can't say the stability is what bothers me. Quite the opposite. It's sad that it's not true for everyone.
Yeah this game has been quite disappointing for myself. I can play it from time to time without crashing, thanks to a 4.5 Ghz CPU.
Unfortunately, none of my friends have such high end hardware. As a result, they struggle to run the game whether they crash or not. Combine this with an elongated learning curve and I have zero friends to play it with as they never put in the time to learn the game because "playing" is subjective for them (mostly just lagging).
Did you guys try the original code I posted (before it got edited 25 times)? Because it was definitely working for me: playing while preloading and crashing almost immediately while not.
I think the concerns for modifying it were "Performance matters," but in all honesty it does not if you just crash to the desktop. The original version was also looked at by PLAG (Valve) and verified that it would not trigger VAC and he also helped to write it in IRC.
Did you guys try the original code I posted (before it got edited 25 times)? Because it was definitely working for me: playing while preloading and crashing almost immediately while not.
I've tried every variation that's been posted. All without success.
Have you tried it within the Sandbox training mode, where myself and rfkg can reproduce the crash every time? I haven't had a chance to setup apitrace or VOGL yet, but I will sometime today or tomorrow with any luck.
I have a feeling there is more than just the glBindBuffer call causing crashes. As my game does also crash when skulking or sometimes just standing still. Not as frequent as the shotgun crash, but it still happens.
Loading freeze happened in online mod too. But after several games it becomes ok (maybe mods slow it down so it doesn't freeze).
PS Had another online shotgun crash even with that workaround. And it seems that overall the crashes (not the shotgun ones, but the random ones; dunno about shotgun) are more frequent when bigger texture size used.
Loading freeze happened in online mod too. But after several games it becomes ok (maybe mods slow it down so it doesn't freeze).
PS Had another online shotgun crash even with that workaround. And it seems that overall the crashes (not the shotgun ones, but the random ones; dunno about shotgun) are more frequent when bigger texture size used.
I can also confirm that on my side the issue appears almost each time I try to shoot with the shotgun as marines or randomly every 15 minutes when graphisms are set to high. However, once settings are disabled or set to low, the issue seems to disappear.
AsranielJoin Date: 2002-06-03Member: 724Members, Playtest Lead, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow, Subnautica Playtester, Retired Community Developer
There is no shotgun specific bugfix, as we need somebody with opengl knowledge. But there have been other critical linux fixes. Somebody mentioned lowering particle quality helps. Does that work for you?
AsranielJoin Date: 2002-06-03Member: 724Members, Playtest Lead, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow, Subnautica Playtester, Retired Community Developer
There was a threading issue leading to a black screen when loading maps. Also the previously outdated libawesomium and luajit version caused several crashes.
There is no shotgun specific bugfix, as we need somebody with opengl knowledge. But there have been other critical linux fixes. Somebody mentioned lowering particle quality helps. Does that work for you?
Is there any plan to change this?
Or should be just see this as wontfix and don't expect any progress on the OpenGL renderer?
Never see anything as wontfix. But if people could confirm that the crash is fixed with minimal quality particles, that would already help
It is not, as you see in my video i have the lowest settings possible and it still crashes like clockwork.
You also seen what nVidia developers responded about this crashing, that the command stream from the OpenGL renderer is not following OpenGL spec on several occasions (even gave you a trace on what calls causing this crash) and that they cannot ignore the calls as a fix because that would mean an really ugly hack from their side.
Don't take this wrong but as it looks like now , the OpenGL code is badly broken doing things that it really should not do and the only solution is to modify it. Thus as you say only an OpenGL programmer can fix this crashing issue.....
Everything else is like painting a pig trying to make it look good.
AsranielJoin Date: 2002-06-03Member: 724Members, Playtest Lead, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow, Subnautica Playtester, Retired Community Developer
You probably misunderstood me. Yes there is a big bug in the opengl code, and yes, there is currently nobody with the knowledge and time to fix it. I would not say that its "badly" broken though, its very likely just one error, causing those crashes. The rest of the opengl code seems to work fine.
That said, searching for a workaround and better documentation of the bug is the next best thing to do in the meantime. Specially as workarounds often help to identify the correct way to solve it. If for example the minimal particle workaround had worked, a temporary hack would have been to enforce this setting on the linux client until the real issue can be fixed. While not ideal, its much better than crashing.
Believe me, if anybody is pushing people to fix this its me. I had a few tries at fixing it myself, but with no luck so far (but i wont give up, planning to take another go at it in the next days, but my opengl knownledge is limited).
So no, there is no intent to make a pig look good just trying to approach the problem from all possible angles.
There is no shotgun specific bugfix, as we need somebody with opengl knowledge. But there have been other critical linux fixes. Somebody mentioned lowering particle quality helps. Does that work for you?
In the training mission it is 100% crash when shooting the lerk with a shotgun on max particles (tried 2 times on 269 and one time on 270).
No crashing on training lerk with low particle quality (tried 2 times on 270).
Also noticed in multiplayer on 269 that around 4 of 6 crashes were while shooting lerks with a shotgun. (but in multiplayer it doesn't crash every time you shoot)
It crashed in multiplayer on 270 with low particle quality when shooting a lerk.
I have tried all the different graphic settings as well as tons of googlin' to find a solution myself. the game is broken at the moment such that I can not play.
Do we have anyone working on this yet? Is there any debug logging I can do that might help? The boomstick instantly crashes my client every time.
GTX 770, amd fx 8350, Mint 17 w/ v334 nvidia drivers.
I have GTX 770, too, though I'm on Debian Jessie (details in the signature) and crashes are rare for me. I play on the daily basis (1-2 hours per day) and have one crash maybe in 1-2 weeks. It's quite strange that almost similar configurations behave so differently. Maybe there are settings outside the game that influence it.
Here's my nvidia-settings OGL page:
I also force VSync off via the app profiles page:
TheFall.x86 is unrelated, it's another game (very good btw) and forcing VSync doesn't help; Unity3D bug, same as in Wasteland 2.
I don't have any compositing enabled and/or running, it's super easy when you have no DM hence gaining full control over the desktop part of the system. Try to set up your system somehow close to what I have and check if the game still crashes.
You probably misunderstood me. Yes there is a big bug in the opengl code, and yes, there is currently nobody with the knowledge and time to fix it. I would not say that its "badly" broken though, its very likely just one error, causing those crashes. The rest of the opengl code seems to work fine.
That said, searching for a workaround and better documentation of the bug is the next best thing to do in the meantime. Specially as workarounds often help to identify the correct way to solve it. If for example the minimal particle workaround had worked, a temporary hack would have been to enforce this setting on the linux client until the real issue can be fixed. While not ideal, its much better than crashing.
Believe me, if anybody is pushing people to fix this its me. I had a few tries at fixing it myself, but with no luck so far (but i wont give up, planning to take another go at it in the next days, but my opengl knownledge is limited).
So no, there is no intent to make a pig look good just trying to approach the problem from all possible angles.
I figured out how to work around the issue 100% of the time.
Here is a partial call stack in which the issue occurs. Commenting out any line of the 'stack' will stop the crash from occuring:
./Natural Selection 2/ns2/lua/Weapons/Marine/Shotgun.lua
function Shotgun:FirePrimary(player)
(no targets)
252: self:ApplyBulletGameplayEffects(player, nil, impactPoint, direction, 0, trace.surface, showTracer)
(targets in range)
264: self:ApplyBulletGameplayEffects(player, target, hitPoint - hitOffset, direction, kShotgunDamage, "", showTracer and i == numTargets)
./Natural Selection 2/ns2/lua/Weapons/BulletsMixin.lua
function BulletsMixin:ApplyBulletGameplayEffects(player, target, endPoint, direction, damage, surface, showTracer)
34: self:DoDamage(damage, target, endPoint, direction, surface, false, showTracer)
./Natural Selection 2/ns2/lua/DamageMixin.lua
function DamageMixin:DoDamage(damage, target, point, direction, surface, altMode, showtracer)
229: HandleHitEffect(point, doer, surface, target, showtracer, altMode, damage, direction)
./Natural Selection 2/ns2/lua/NS2Utility.lua
function HandleHitEffect(position, doer, surface, target, showtracer, altMode, damage, direction)
235: GetEffectManager():TriggerEffects("damage", tableParams)
./Natural Selection 2/ns2/lua/EffectManager.lua
467: InternalTriggerMatchingEffects(self, data, triggeringEntity, effectName, tableParams, cachedTableParams)
(I stopped traversing the stack at this point)
If you want to test this yourself, you'll want to do the following:
1. Add "res_workers 4" to your launch args
2. Disable consistency checking, add "*" to the ignore list at ./Natural Selection 2/core/lua/ConsistencyConfig.lua
3. Set particle effects to maximum
4. Comment out the line you want to test.
5. Open ns2, launch sandbox mode, buy a shotgun and fire it. You can also spawn a lerk with "spawn lerk" in the console.
You should be able to use this to create a workaround (e.g. disable the shotgun hit effect when running on linux/openGL), and it could be helpful in fixing one of the real causes.
Please implement a workaround for the next patch if possible since I want to be able to play marines again.
Well, this is the shotgun's high level logic, Lua scripts can't cause a crash inside the video driver. If you disable this codepath, you'll possibly render the shotgun non-working (or not displaying the bullets particles). The real error is much deeper, in the renderer's code. I may prefer the invisible bullets over crashes but it's more like a workaround than the fix. Still, a great advancement, all we've had before is the binary callstack, now there's a bit of Lua.
And btw, how did you figure this out? I'm just curious. I've become a real pro in getting GDB backtraces thanks to NS2 but gaining some knowledge in Lua tracing can't hurt.
AsranielJoin Date: 2002-06-03Member: 724Members, Playtest Lead, Forum Moderators, NS2 Playtester, Squad Five Blue, Reinforced - Shadow, WC 2013 - Shadow, Subnautica Playtester, Retired Community Developer
I would also like to know how you figured this out.
THAT said, i was able to track the crash down to one specific cinematic that is triggered by the shotgun. Its the cinematics/materials/metal/ricochetHeavy.cinematic (or likely all ricochetHeavy.cinematic files). Changing the shotgun to use the normal riccochet fixes the issue.
Now its just a question of what the correct fix it. But thank you very much for finding this, i'm still unsure how you managed to do that.
Well, completely unexpected resolution as I didn't expect it to happen that fast. I really hope this will be patched ASAP and it will work for real. Probably, it's the most anticipated fix for me for this year. Even more than hitching fix.
Well, this is the shotgun's high level logic, Lua scripts can't cause a crash inside the video driver. If you disable this codepath, you'll possibly render the shotgun non-working (or not displaying the bullets particles). The real error is much deeper, in the renderer's code. I may prefer the invisible bullets over crashes but it's more like a workaround than the fix. Still, a great advancement, all we've had before is the binary callstack, now there's a bit of Lua.
I tested this during debugging and depending on which call you comment out, you still get damage and most particle effects.
I would also like to know how you figured this out. In any case, my intetion was to help provide a lot more debugging information so that a proper fix can be created, which seems to finally be in sight now.
THAT said, i was able to track the crash down to one specific cinematic that is triggered by the shotgun. Its the cinematics/materials/metal/ricochetHeavy.cinematic (or likely all ricochetHeavy.cinematic files). Changing the shotgun to use the normal riccochet fixes the issue.
Now its just a question of what the correct fix it. But thank you very much for finding this, i'm still unsure how you managed to do that.
That's great to hear! I hope we can get this fixed in the next patch.
As for how I figured this out, I just used the simple tried and true method of debugging - commenting out code, which is useful if you don't know the codebase very well (or hardly at all in my case). I used the fact that lowering the particle quality decreases the likelihood of the crash as a hint and assumed that the issue must be related to some effect specific to the shotgun. I then basically commented out calls relating to effects that occur aftering shooting the shotgun (hence the FirePrimary function), checked to see the effect (fixed crash or not) and continued doing this up the call stack. In some functions, I did a binary search by commenting out blocks of function calls to save time debugging.
After each commenting out attempt, I reloaded the game and used my reproducible crash (high particle quality, sandbox, shoot the shotgun) as opposed to the tutorial one as that takes too long. I also used the res_workers cvar to significantly decrease load times. In the end, it only took me a bit over an hour to produce the call stack.
Just amazing! It's really easier to debug such issues with at least half of the game code at hand (the engine is closed source but it seems that it has enough exported hadnles to play with). I conceded (yep) after trying to find the exact crash source with various api tracers but didn't think that it can be worked around at higher levels. My bad.
That aside, we can make a mod meanwhile that replaces the lua/DamageEffects.lua. I only found the ricochetHeavy.cinematic reference in that file. I suppose this file is consistency-checked so it should be a server mod but I believe I can easily get it installed on our "Russia [PRO] #N" servers. Even if it only benefits me and does no harm to others.
Just amazing! It's really easier to debug such issues with at least half of the game code at hand (the engine is closed source but it seems that it has enough exported hadnles to play with). I conceded (yep) after trying to find the exact crash source with various api tracers but didn't think that it can be worked around at higher levels. My bad.
That aside, we can make a mod meanwhile that replaces the lua/DamageEffects.lua. I only found the ricochetHeavy.cinematic reference in that file. I suppose this file is consistency-checked so it should be a server mod but I believe I can easily get it installed on our "Russia [PRO] #N" servers. Even if it only benefits me and does no harm to others.
Indeed, it was very fortunate that the LUA source is available as otherwise I suspect this issue would probably remain unfixed.
Yeah I was thinking that you could make a mod to workaround this, but it's not ideal as servers have to have it as you mentioned. Hopefully we can get a proper fix for this soon.
Comments
Or maybe try contacting nVidia to get a better explanation of the issue.
If you guys don't have anyone with good OpenGL knowledge anymore maybe start a forum post here or contact http://www.phoronix.com/scan.php?page=home or maybe http://www.gamingonlinux.com/ and let them advertise that you guys are looking for a OpenGL dev. They will probably do it to help a game on Linux.
Btw is there anyone on Linux that is able to play this game anymore because of the faulty OpenGL code? Maybe update the information on Steam etc, about the Linux state to what it really is?
I bought this game because it "supported" Linux and honestly i'm really disappointed on the current state and that this issue has not been resolved in more than a year.
Unfortunately, none of my friends have such high end hardware. As a result, they struggle to run the game whether they crash or not. Combine this with an elongated learning curve and I have zero friends to play it with as they never put in the time to learn the game because "playing" is subjective for them (mostly just lagging).
Did you guys try the original code I posted (before it got edited 25 times)? Because it was definitely working for me: playing while preloading and crashing almost immediately while not.
I think the concerns for modifying it were "Performance matters," but in all honesty it does not if you just crash to the desktop. The original version was also looked at by PLAG (Valve) and verified that it would not trigger VAC and he also helped to write it in IRC.
I've tried every variation that's been posted. All without success.
Have you tried it within the Sandbox training mode, where myself and rfkg can reproduce the crash every time? I haven't had a chance to setup apitrace or VOGL yet, but I will sometime today or tomorrow with any luck.
I have a feeling there is more than just the glBindBuffer call causing crashes. As my game does also crash when skulking or sometimes just standing still. Not as frequent as the shotgun crash, but it still happens.
PS Had another online shotgun crash even with that workaround. And it seems that overall the crashes (not the shotgun ones, but the random ones; dunno about shotgun) are more frequent when bigger texture size used.
I can also confirm that on my side the issue appears almost each time I try to shoot with the shotgun as marines or randomly every 15 minutes when graphisms are set to high. However, once settings are disabled or set to low, the issue seems to disappear.
Is there any plan to change this?
Or should be just see this as wontfix and don't expect any progress on the OpenGL renderer?
It is not, as you see in my video i have the lowest settings possible and it still crashes like clockwork.
You also seen what nVidia developers responded about this crashing, that the command stream from the OpenGL renderer is not following OpenGL spec on several occasions (even gave you a trace on what calls causing this crash) and that they cannot ignore the calls as a fix because that would mean an really ugly hack from their side.
Don't take this wrong but as it looks like now , the OpenGL code is badly broken doing things that it really should not do and the only solution is to modify it. Thus as you say only an OpenGL programmer can fix this crashing issue.....
Everything else is like painting a pig trying to make it look good.
That said, searching for a workaround and better documentation of the bug is the next best thing to do in the meantime. Specially as workarounds often help to identify the correct way to solve it. If for example the minimal particle workaround had worked, a temporary hack would have been to enforce this setting on the linux client until the real issue can be fixed. While not ideal, its much better than crashing.
Believe me, if anybody is pushing people to fix this its me. I had a few tries at fixing it myself, but with no luck so far (but i wont give up, planning to take another go at it in the next days, but my opengl knownledge is limited).
So no, there is no intent to make a pig look good
In the training mission it is 100% crash when shooting the lerk with a shotgun on max particles (tried 2 times on 269 and one time on 270).
No crashing on training lerk with low particle quality (tried 2 times on 270).
Also noticed in multiplayer on 269 that around 4 of 6 crashes were while shooting lerks with a shotgun. (but in multiplayer it doesn't crash every time you shoot)
It crashed in multiplayer on 270 with low particle quality when shooting a lerk.
100% reproducible; must be super easy to fix.
Do we have anyone working on this yet? Is there any debug logging I can do that might help? The boomstick instantly crashes my client every time.
GTX 770, amd fx 8350, Mint 17 w/ v334 nvidia drivers.
Here's my nvidia-settings OGL page:
I also force VSync off via the app profiles page:
TheFall.x86 is unrelated, it's another game (very good btw) and forcing VSync doesn't help; Unity3D bug, same as in Wasteland 2.
I don't have any compositing enabled and/or running, it's super easy when you have no DM hence gaining full control over the desktop part of the system. Try to set up your system somehow close to what I have and check if the game still crashes.
I figured out how to work around the issue 100% of the time.
Here is a partial call stack in which the issue occurs. Commenting out any line of the 'stack' will stop the crash from occuring:
./Natural Selection 2/ns2/lua/Weapons/Marine/Shotgun.lua function Shotgun:FirePrimary(player) (no targets) 252: self:ApplyBulletGameplayEffects(player, nil, impactPoint, direction, 0, trace.surface, showTracer) (targets in range) 264: self:ApplyBulletGameplayEffects(player, target, hitPoint - hitOffset, direction, kShotgunDamage, "", showTracer and i == numTargets) ./Natural Selection 2/ns2/lua/Weapons/BulletsMixin.lua function BulletsMixin:ApplyBulletGameplayEffects(player, target, endPoint, direction, damage, surface, showTracer) 34: self:DoDamage(damage, target, endPoint, direction, surface, false, showTracer) ./Natural Selection 2/ns2/lua/DamageMixin.lua function DamageMixin:DoDamage(damage, target, point, direction, surface, altMode, showtracer) 229: HandleHitEffect(point, doer, surface, target, showtracer, altMode, damage, direction) ./Natural Selection 2/ns2/lua/NS2Utility.lua function HandleHitEffect(position, doer, surface, target, showtracer, altMode, damage, direction) 235: GetEffectManager():TriggerEffects("damage", tableParams) ./Natural Selection 2/ns2/lua/EffectManager.lua 467: InternalTriggerMatchingEffects(self, data, triggeringEntity, effectName, tableParams, cachedTableParams) (I stopped traversing the stack at this point)If you want to test this yourself, you'll want to do the following:
1. Add "res_workers 4" to your launch args
2. Disable consistency checking, add "*" to the ignore list at ./Natural Selection 2/core/lua/ConsistencyConfig.lua
3. Set particle effects to maximum
4. Comment out the line you want to test.
5. Open ns2, launch sandbox mode, buy a shotgun and fire it. You can also spawn a lerk with "spawn lerk" in the console.
You should be able to use this to create a workaround (e.g. disable the shotgun hit effect when running on linux/openGL), and it could be helpful in fixing one of the real causes.
Please implement a workaround for the next patch if possible since I want to be able to play marines again.
And btw, how did you figure this out? I'm just curious. I've become a real pro in getting GDB backtraces thanks to NS2 but gaining some knowledge in Lua tracing can't hurt.
THAT said, i was able to track the crash down to one specific cinematic that is triggered by the shotgun. Its the cinematics/materials/metal/ricochetHeavy.cinematic (or likely all ricochetHeavy.cinematic files). Changing the shotgun to use the normal riccochet fixes the issue.
Now its just a question of what the correct fix it. But thank you very much for finding this, i'm still unsure how you managed to do that.
Many thanks to both of you!
That's great to hear! I hope we can get this fixed in the next patch.
As for how I figured this out, I just used the simple tried and true method of debugging - commenting out code, which is useful if you don't know the codebase very well (or hardly at all in my case). I used the fact that lowering the particle quality decreases the likelihood of the crash as a hint and assumed that the issue must be related to some effect specific to the shotgun. I then basically commented out calls relating to effects that occur aftering shooting the shotgun (hence the FirePrimary function), checked to see the effect (fixed crash or not) and continued doing this up the call stack. In some functions, I did a binary search by commenting out blocks of function calls to save time debugging.
After each commenting out attempt, I reloaded the game and used my reproducible crash (high particle quality, sandbox, shoot the shotgun) as opposed to the tutorial one as that takes too long. I also used the res_workers cvar to significantly decrease load times. In the end, it only took me a bit over an hour to produce the call stack.
That aside, we can make a mod meanwhile that replaces the lua/DamageEffects.lua. I only found the ricochetHeavy.cinematic reference in that file. I suppose this file is consistency-checked so it should be a server mod but I believe I can easily get it installed on our "Russia [PRO] #N" servers. Even if it only benefits me and does no harm to others.
Yeah I was thinking that you could make a mod to workaround this, but it's not ideal as servers have to have it as you mentioned. Hopefully we can get a proper fix for this soon.
That said, i would like to figure out what exactly goes wrong with that cinematic, as other cinematics might have the same issue.
Going to check with the real gameplay programmers on what the actual fix should be for the next patch. no promises yet, but its looking good.