Build 273 - Lags with SG

MauvaisVitrier

I've been experiencing weird lags when using the shotgun since the last update came out. It only happens with the shotgun, when shooting. I then have each bullet I shoot, a sort of roll-back-warp thingy with an about .5 sc delay. I've no issue whatsoever on a local server. But this happens now every time during online games.
This seems to be even worse when there's stuff on the screen (even just a growing hive). I've no 'lags' (I call them lags because I don't really know what they are) with the lmg even when there's lots of stuff (pve, enemies) in a room.
I might record a little demo for you to see what I mean. I've heard people complaining about unusual lags since the last update but nothing regarding the shotgun.
So I'm wondering where does this problem come from.

ps : I'll update when I get something on tape for you to see. In a couple of hours max

Scout255

I have the exact same issue, after a shotgun blast (either from me, or another marine using one) I get a sizable and large hitch. Makes timing followup shots very difficult and is incredibly annoying.

Had no such issues with the last build. Seems like a lot of people are experiencing it based on mumble chat. Kinda takes the fun out of the game at the moment which is a shame.

AMD R290X Graphic card, I5-2500K @ 4.8 Ghz, 16 Gig Ram, and an SSD. Have not updated drivers recently so it shouldn't be that. Seems to be caused by the most recent patch.

IronHorse

Please follow my instructions here:
http://forums.unknownworlds.com/discussion/comment/2230001/#Comment_2230001

MauvaisVitrier

Thanks, I will do that.

Btw, has anyone reported you a HUGE fps decrease with cluster grenades ? I don't if this is new (I mean last build or not), but I usually have 140 fps or so, and when a cluster grenade I threw explodes, my fps go down to around 15 for maybe 1.5 to 2 seconds. Never experienced that too before.

IronHorse

Alternatively, here is a very quick way to get us the data we need:

Open console at the menu (~ key) and type ns2_summit. Once you spawn in the ready room, type in the console:

cheats 1
j2
switch
give shotgun
profile (press spacebar a few times to make sure its running and is not paused)

Now find and shoot a cyst and immediately press spacebar to pause the profiler when you do
Use the arrow keys to highlight the tallest yellow frame at the bottom of screen
Then expand the longest horizontal purple bars at the top of the screen by clicking on the top (longest) bar until you cant go down any further
Press F12 for screenshot
Upload to imgur and link it here.

Thank you to everyone providing data!
@MauvaisVitrier‌ maybe do the above ^ with cluster grenades, that could help us.

BeigeAlert

I wasn't able to repro this on a local server, but I tried it on one of the Tactical Freedom servers, and it appeared immediately. Of course, this doesn't rule out server mods.

rkfg

Good, good. The same function that I have in plog/profiler so it's not platform-specific. Looks to me like it doesn't cache the loaded particle files and reloads them again and again. As I assumed here, mods make it more visible because they add more directories to the game's virtual filesystem so it should scan them as well for every file. This makes any file search proportionally slower. I hope devs implement a smarter system one day. For example, it could scan mod directories once on startup and create a dictionary (a hash map) of files in them. Mods content doesn't change during the game so it's only required on startup. Further lookups would start with searching the dictionary, which is most of the time an O(1) operation and doesn't require any slow I/O, and then loading the file from the base directory or from a mod's one. This will drastically reduce the overall loading time and hitches like this one, though it's better to have them as they point to much more serious engine bugs. Especially on rotary disks.

Why does this happen? I suppose playtesters only play on vanilla servers so everything is fast and smooth. I suggest creating a bunch of empty mods just to stress this VFS thing. It won't change the game logic (mods are empty) but it will add more directories to look files in and hence constant I/O bugs will become visible with a naked eye. A win-win situation.

devel

rkfg wrote: »

For example, it could scan mod directories once on startup and create a dictionary (a hash map) of files in them. Mods content doesn't change during the game so it's only required on startup.

It is already like that. To enable constant filesystem scanning, pass "-hotload" launch parameter.

(btw, there is decent fs notify subsystem at least in Linux. If used, it would allow hotload dynamic behavior without penalty)

rkfg

Hotloading is looking to be not what I expect. It allows mods reloading on the fly. The description is pretty vague like "use -hotload parameter to enable hotloading". So it's meant to be used by developers, not regular players. My proposition doesn't seem to break or replace consistency check (though it should be thought out), it's just for marking the source of the file (should this particular model or texture be loaded from the base NS2 directory or from this mod?) so there are no excessive filesystem scans. Even though Linux caches both files and directories, it doesn't help when the game hits it hundreds times per second.

rkfg

Actually, from what you describe I can say that -hotload does quite the opposite — it enables constant FS scanning while I expect it to be as low as possible. For now playing on 12x12 server with 5 mods is nearly impossible in late game. Even welder slows game to a crawl, 15 FPS at most. LMG creates various lags with 500 ms freezes. When you're moving with your teammates and you all shoot it becomes a nightmare.

Unfortunately, for me this is the worst build amongst all since Sep 2013. Even when the game crashed a lot, it was at least playable for a round or so. Now even joining is pointless.

devel

rkfg wrote: »

Actually, from what you describe I can say that -hotload does quite the opposite — it enables constant FS scanning while I expect it to be as low as possible.

Yes, let me google-translate my post to you:
Это уже так. Чтобы включить сканирование постоянный файловой системы, проходят "-hotload" параметр запуска.

So, when there are any problems with frequent OpenFile calls - it means that either the files are missing (we've already seen broken maps and mods), or a bug clears some part "a dictionary (a hash map)" of loaded resouces.

rkfg

I read and understand English pretty well, thanks, maybe I've poorly described it in my previous posts so you misunderstood it.

The render function that somehow related to particles tries to find and (probably) load some files. It does so very often and the worst part that it searches for those files not only in the main NS2 resource directory but also in all mod folders. The more mods are active, the more time it requires, obviously. I see no reason to scan all those directories once the needed file was found. Its location can be marked internally so further load attempts won't require such excessive search. It would only require to traverse 1 directory instead of 2+n (where n is the number of active mods and 2 is the number of base resource directories). Thus the loading time decreases proportionally. Not n-fold, of course, but should be noticeable enough. I don't see such behavior now, strace log clearly shows that the engine, ahem, "never learns" and runs through all mods and the base directory every time. It makes no sense — no new files can appear inside any mod folder, if there's no such file there now there won't be in the future.

What's killing the performance is that the found file is being requested again and again. Let's look at the very beginning of the file:

21:40:16.686120 stat64("/home/rkfg/.config/Natural Selection 2/Workshop/m601c0a0_1408733618/materials/effects/particle_generic.material.polygons", 0xff8dfd40) = -1 ENOENT (No such file or directory)
21:40:16.686887 stat64("/home/rkfg/.config/Natural Selection 2/Workshop/m601c0a0_1408733618/materials/effects/particle_generic.material.polygons", 0xff8dfde0) = -1 ENOENT (No such file or directory)

I see no difference here, only the buffer address differs. Man says:

int stat(const char *pathname, struct stat *buf);

So the second parameter is the memory address. Still, after failing one attempt the engine does it again. And after that it tries another mod, again, twice. And what's even worse, it once again does it for the first and then for the second mod. Why? Overall, we have 8 (eight) calls for just two mods. Then it searches in /Natural Selection 2/system (twice) and then in /Natural Selection 2/ns2 (again twice). The result for the latter is different, not "-1 ENOENT (No such file or directory)" but "0", and the structure is now filled with data ({st_mode=S_IFREG|0755, st_size=234, ...}). It indicates that the file was found. Indeed, it's there, at "/Natural Selection 2/ns2/materials/effects/particle_generic.material.polygons".

This process took 3 ms. Is it over? No, not even nearly so. It's then repeated once again, the entire sequence! So the engine searches "/materials/effects/particle_generic.material.polygons" again, 8 times among the mods and 4 more calls for the core files. Totalling in 24 calls and 6 ms. Per just a single file.

But wait, there's more! Now we need "/cinematics/vfx_materials/vfx_Lite_smkLoop_02_animated.material.polygons". 12 more calls. 3 more ms. Then it needs "/cinematics/vfx_materials/vfx_fire_glow_01.material.polygons". Then "/cinematics/vfx_materials/vfx_fire_twist_animated.material.polygons". Adding smoke: "/cinematics/vfx_materials/vfx_impact_smoke_01_animated.material.polygons". Adding sparks: "/cinematics/vfx_materials/vfx_sparks_01_animated.material.polygons". Throwing away frames.

I seriously hope a hotfix will be released soon. This is a gamebreaker, no joke.

MauvaisVitrier

@IronHorse : I'll create a plog for cluster grenades too. I don't know if that has anything to do with the shotgun lag/fpsdrop/warp thingy; but I'll send you a demo anyway. Do you want them sent by PM ?

I seriously hope a hotfix will be released soon. This is a gamebreaker, no joke.

I don't know if @rkfg is talking about my issue (I haven't read everything, it's all very technical) but I agree with that statement, this is a gamebreaker. I first experienced this issue during the div 2 finals, and if I manage to deal with this in pubs, it becomes really problematic in comp play.

Anyhoo, let's get to work. I'll edit this post when I have everything you need.

Asraniel

the cause of this particular issue has been found and corrected. We are also looking into some other issues and need to be sure that other performance problems reported by people are actually related.

MauvaisVitrier

Do the plogs provide you with visual feedback ? or just info ?

@Ironhorse and maybe other CDT members interested : here are two plogs
- PLOG #1 : this one has been recorded on a local server. I used cluster grenades, lmg and shotgun in various places and shot a bunch of bots in their hive

- PLOG #2 : this one is an online game. Had a few fps drops here and there that were pretty annoying. I had an intentional 10 seconds recording session that dropped my fps even lower, towards the end of the plog.

Tell me if that's what you needed.

@Asraniel : which "particular issue" ? the one I talked about ? or rkfg's ?

Asraniel

they might be the same.

There is an issue that everytime a particle system is created, like when shooting a shotgun, there is a unnecessary check if some files exist. this check costs more if there are more mods loaded. This was introduced to be more resistant to missing files, like the previous bug where on some maps there where sudden fps drops, caused by missing files. This issue was fixed (the actual playtest of it has yet to happen).
The question is now if those other reports, of people loosing fps even in the readyroom are in any way connected.

Deck_

Asraniel wrote: »

they might be the same.

There is an issue that everytime a particle system is created, like when shooting a shotgun, there is a unnecessary check if some files exist. this check costs more if there are more mods loaded. This was introduced to be more resistant to missing files, like the previous bug where on some maps there where sudden fps drops, caused by missing files. This issue was fixed (the actual playtest of it has yet to happen).
The question is now if those other reports, of people loosing fps even in the readyroom are in any way connected.

Does it matter how many files of mods I have in my workshop folder or does it matter how many mods the server is running that causes the check to cost more?

rkfg

I believe it's the running count that matters. You can also check messages in the console, it should say "Mounting mod from ...", then the mod joins the game's virtual filesystem.

Asraniel

Yes, the amount of mods matter. As i said, this should be fixed soon.

IronHorse

Thanks everyone!
As asraniel said, the issue seems to have been found and corrected. Now its just a matter of testing this fix to ensure it doesn't create other bugs, and then getting it out to you ASAP.
While I cannot promise anything officially, there is a very good chance of a hotfix occurring in the next couple of days. Sit tight, and thank you!

MauvaisVitrier

Great!

Yojimbo

Love the collaboration between the forum community and CDT, great job on pinpointing the cause and already have a fix in the pipeline, recently took a break from NS2, came back to 273, found the performance to be lower than usual, so decided to leave it for awhile comfortable in the knowledge it will be running smooth as a jamaican bob sled team by the time I get back, nice to know people will always care for this game, as its truly one of most underrated FPS/RTS games to ever grace the PC, feel gutted for all the brain dead COD fans.

Big love.

matso

rkfg wrote: »

Good, good. The same function that I have in plog/profiler so it's not platform-specific. Looks to me like it doesn't cache the loaded particle files and reloads them again and again. As I assumed here, mods make it more visible because they add more directories to the game's virtual filesystem so it should scan them as well for every file. This makes any file search proportionally slower. I hope devs implement a smarter system one day. For example, it could scan mod directories once on startup and create a dictionary (a hash map) of files in them.

275.

Loulebe

I have the same problem.

Shotgun freeze ... I think that freeze more when i shoot alien...
When i play alien and that marine have shotgun i have crazy freeze...
When i build with welder that lag... too

i7-920 overclock to 3,2 ghz.
hd 6870
Win 7, 32-bit
4 giga ram.

Generally i have 80-100 fps

The problem is the gpu ?

Asraniel

Loulebe wrote: »

I have the same problem.

Shotgun freeze ... I think that freeze more when i shoot alien...
When i play alien and that marine have shotgun i have crazy freeze...
When i build with welder that lag... too

i7-920 overclock to 3,2 ghz.
hd 6870
Win 7, 32-bit
4 giga ram.

Generally i have 80-100 fps

The problem is the gpu ?

Your problem will be fixed with the next patch, which should come out in about a day.

rkfg

Patch is out, the issue is gone. Great work, CDT&UWE!

Yojimbo

@rkfg for new member of CDT!