I'm no C/C++ low-level developer but instead a C#/Java developer so this may be really stupid but...in C# you've a FileSystemWatcher class with which you can listen to changes done inside a specified directory. Event types would be added/removed/renamed files. Wouldn't that be sufficient? If the event fires, you know someone tinkered with the files. If it doesn't fire you're good. Add to that an initial checksum check right after you registered for the events and everything's good.
Or did I miss something important somewhere? (I most probably did, but I don't see it)
I don't get these hitches when the game starts and my load times are normal, so I don't understand this. Is this something just affecting a small group of people?
I do however get hitch/lag/stutter mid-late game, or entering rooms with a lot going on. Is this related to this fix?
IronHorseDeveloper, QA Manager, Technical Support & contributorJoin Date: 2010-05-08Member: 71669Members, Super Administrators, Forum Admins, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Blue, Subnautica Playtester, Subnautica PT Lead, Pistachionauts
@ghosthree3
You should include the .hlsl file in the ignore list, since thats the new AV file type (which allows everyone to run their own AV, like its supposed to be doing [should be fixed down the road hopefully])
I would hope they are verifying memory, not files.
It's not. Countless tests showing how often the files are getting opened (200 times in some files cases when loading or running around in game) shows that these files are getting accessed way more than they need to be. Perhaps it checks memory as well, but it definitely checks the files. How the game SHOULD work is check all the files once when the game boots and have them set to read only until the game closes. Then keep a list of all the hashes it got from that initial check and use that for comparison (also read only). Unfortunately I don't know how to do this right now, I haven't even looked, might even be binary stuff I can't touch.
@ghosthree3
You should include the .hlsl file in the ignore list, since thats the new AV file type (which allows everyone to run their own AV, like its supposed to be doing [should be fixed down the road hopefully])
There's ignores for darkvision etc. The problem is files like this exist "Cloaked.hlsl". Pretty sure that one needs to be left blocked.
EDIT: Now blocks edits to marine models, reskinning should be possible though?
With blocked viewmodels for the rifle, shotgun and pistol (we don't need to stop people turning the GL or FL invisible give me a break), also alien viewmodels (eg. skulk mouth): https://gist.github.com/anonymous/7f586c04b0a81c6a24b5
SamusDroidColoradoJoin Date: 2013-05-13Member: 185219Members, Forum Moderators, NS2 Developer, NS2 Playtester, Squad Five Gold, Subnautica Playtester, NS2 Community Developer, Pistachionauts
edited November 2013
Just to put it in perspective http://pastebin.com/ese3WhcB
That's a list of things not loaded with texture streaming on when you join the RR. https://www.dropbox.com/s/nog96h8lg5sbe1t/249 TS test.zip
Use notepad++'s compare plugin to compare TS on and off on 24, as well as 249 TS on and 260 with TS.
For some reason lua files are opened like 90 times when loading the same too...
Another big note: switching from one build to another (even 260 and 259) will result in obscene hitches. With TS OFF I get NO hitches whatsoever. NONE. I then turn TS ON (which is forced on now) and I go from 260 to 249. I get the same MASSSSIVE hitches. Like the MASSIVE team join hitch.
I think the list of checked models should also include everything that is dynamic and relevant for gameplay - for example cysts, hydras, clogs, webs, babblers, ...
The problem with not checking these is that one could edit these models' geometry to make them stick through walls - IIRC, there was a hack in TFC that added long spikes to the sentry model, which gave away their position. So even a modified clog or babbler model would give away the information that a gorge is active in that area.
Following that line of though, the models of all alien and marine buildings should be added as well. Maybe it's easier to add models/* and define exclusions?
I'm fine with removing most of the texture checks, though. Checking building textures, for example, is not worth the overhead.
Yeah I guess so, it shouldn't take long to implement, will edit in a few minutes. I'd be adding only *.model files so textures should still be changeable if you want pink cysts (lol).
With blocked viewmodels for the rifle, shotgun and pistol (we don't need to stop people turning the GL or FL invisible give me a break), also alien viewmodels (eg. skulk mouth): https://gist.github.com/anonymous/ad4bd68ec2519709d2a1
Can you SamusDroid upload that 249 build, I want to check is the texture streaming my biggest reason that DX9 give some slowdowns and while using dx11 it just give some hitches or windows 8/8.1 thing.. Ofc that build don't have dx11
Can you SamusDroid upload that 249 build, I want to check is the texture streaming my biggest reason that DX9 give some slowdowns and while using dx11 it just give some hitches or windows 8/8.1 thing.. Ofc that build don't have dx11
I already did prove it's texture streaming. No I won't upload it, because I don't have 8 GB somewhere to store it and ...well many reasons. It's forced texture streaming causing your hitches. While I do hate windows 8 I don't have a reason to blame it here. DX11 is less stable in this game than DX9. So that and TS being on probably explains your hitches.
Ok so if I am following this correct, the hitching is a possible result of the texture streaming re-loading unloaded textures (to conserve memory), which causes them to be consistency checked again resulting in hitches?
I do recall that previously I was able to smooth my game be ENABLING TS, back in like 245 or something. Then in more recent builds the hitching is back and as you said, the texture streaming option seems to not make a difference, seems like it is always on (game loads and you can actually watch textures load onto their models for a few seconds).
So is there no way to manually disable TS to see if it helps?
So it is definitely loading the same textures repeatedly? That would match some console output I've been wondering about, but I don't understand why it's done this way. IIRC, even a 32-bit app like NS2 has a 4Gb address space on a 64-bit system, which should be plenty to hold all textures once loaded, even with memory fragmentation. Of course, on 32-bit systems (especially without the /3gb flag) it needs to fall back to what it is doing now. But why force it on all systems?
I have actually seen zero hitches mid game (except for the early game first engagement shit like blood and shell casings loading for the first time) since enabling low texture quality. I guess streaming isn't unloading them as much any more for me.
Ok so if I am following this correct, the hitching is a possible result of the texture streaming re-loading unloaded textures (to conserve memory), which causes them to be consistency checked again resulting in hitches?
I think I might have a solution to that.
From tests I've done I think consistency checking does make hitches worse. But only because of the texture streaming in the first place. If you preload everything with no streaming (like 249) then you wouldn't hitch due to the consistency.
Why do I think this? Ran multiple tests and on a server with NO consistency I froze up for under half the time for first engagement related freezes.
EDIT: I've actually done more tests recently with someone helping me out and discovered the following. Load time is not only hindered by disk read speed but also communication to the server when loading.
Example: Loading a listen server on an SSD with default consistency takes about 15s or so (I can't actually remember the numbers but look at them relative to others). If you load a listen server the same way after putting the entire game on a ramdrive it will load in about 3s.
However, connecting to a server (default consistency) on an SSD is about 20s to load, and on a ram is about the same, maybe a little less so 18s.
Connecting to a server with no consistency checking on an SSD is about a 15s load and on a ramdrive is about 5s.
So if you have a standard HDD then your load time is being bottlenecked by read speed. Once you get an SSD it's being bottlenecked by your connection to the server more than anything else (sending a ton of hashes and waiting for replies maybe?).
Updated post above. Though it's not related to the original point, new findings!
EDIT: Hitches didn't disappear for me with consistency off, they just weren't as bad. Easy example of a MASSIVE hitch, joining marine/alien/spectator after first joining a server. That first time it freezes for ages, no, this isn't it loading things. It's because it's checking consistency, with none it's instant (well almost) every time first time.
This is just a sequence of many hitches one after the other resulting in one big freeze. Unlike say the shell casing which is just one thing and as such is a smaller freeze.
Well I think i can block it from unloading textures for clients, but after testing it a bit it seems to still be doing it with models also (perhaps, didnt investigate it very far).
If you feel like testing, i added it to the NSL mod (sv_nsltextures to toggle it on/off). I dont think it will completely resolve hitching on team joins, but I would be curious to see if you think it helps hitching during a round.
Comments
EDIT: Also modding alien vision in the default consistency OR this one could make them turn blue as well.
I don't understand the problem.
https://gist.github.com/anonymous/816dee7a03174e772066
EDIT: Damn it, didn't realise I double posted. Mod merge.
EDIT: Replaced ignore for all py files,
https://gist.github.com/anonymous/26f56f483e5ecfe63de5
Or did I miss something important somewhere? (I most probably did, but I don't see it)
I do however get hitch/lag/stutter mid-late game, or entering rooms with a lot going on. Is this related to this fix?
You should include the .hlsl file in the ignore list, since thats the new AV file type (which allows everyone to run their own AV, like its supposed to be doing [should be fixed down the road hopefully])
It's not. Countless tests showing how often the files are getting opened (200 times in some files cases when loading or running around in game) shows that these files are getting accessed way more than they need to be. Perhaps it checks memory as well, but it definitely checks the files. How the game SHOULD work is check all the files once when the game boots and have them set to read only until the game closes. Then keep a list of all the hashes it got from that initial check and use that for comparison (also read only). Unfortunately I don't know how to do this right now, I haven't even looked, might even be binary stuff I can't touch.
There's ignores for darkvision etc. The problem is files like this exist "Cloaked.hlsl". Pretty sure that one needs to be left blocked.
EDIT: Now blocks edits to marine models, reskinning should be possible though?
https://gist.github.com/anonymous/e177fae557447f0e7d84
EDIT2: Now blocks changes to mines so you can't make them red etc.
https://gist.github.com/anonymous/75d89b8cbeaf28fd77e0
Without blocking gun viewmodels:
https://gist.github.com/anonymous/8bc8611b7a70dfc60a88
With blocked viewmodels for the rifle, shotgun and pistol (we don't need to stop people turning the GL or FL invisible give me a break), also alien viewmodels (eg. skulk mouth):
https://gist.github.com/anonymous/7f586c04b0a81c6a24b5
http://pastebin.com/ese3WhcB
That's a list of things not loaded with texture streaming on when you join the RR.
https://www.dropbox.com/s/nog96h8lg5sbe1t/249 TS test.zip
Use notepad++'s compare plugin to compare TS on and off on 24, as well as 249 TS on and 260 with TS.
For some reason lua files are opened like 90 times when loading the same too...
Another big note: switching from one build to another (even 260 and 259) will result in obscene hitches. With TS OFF I get NO hitches whatsoever. NONE. I then turn TS ON (which is forced on now) and I go from 260 to 249. I get the same MASSSSIVE hitches. Like the MASSIVE team join hitch.
I think the list of checked models should also include everything that is dynamic and relevant for gameplay - for example cysts, hydras, clogs, webs, babblers, ...
The problem with not checking these is that one could edit these models' geometry to make them stick through walls - IIRC, there was a hack in TFC that added long spikes to the sentry model, which gave away their position. So even a modified clog or babbler model would give away the information that a gorge is active in that area.
Following that line of though, the models of all alien and marine buildings should be added as well. Maybe it's easier to add models/* and define exclusions?
I'm fine with removing most of the texture checks, though. Checking building textures, for example, is not worth the overhead.
EDIT:
Without blocking gun viewmodels:
https://gist.github.com/anonymous/97350fff834444e4e20b
With blocked viewmodels for the rifle, shotgun and pistol (we don't need to stop people turning the GL or FL invisible give me a break), also alien viewmodels (eg. skulk mouth):
https://gist.github.com/anonymous/ad4bd68ec2519709d2a1
Except you can feel the slower mouse response time as that was actually improved after reinforced
They didn't want to remove the option for some reason...
I think I might have a solution to that.
So is there no way to manually disable TS to see if it helps?
I have actually seen zero hitches mid game (except for the early game first engagement shit like blood and shell casings loading for the first time) since enabling low texture quality. I guess streaming isn't unloading them as much any more for me.
From tests I've done I think consistency checking does make hitches worse. But only because of the texture streaming in the first place. If you preload everything with no streaming (like 249) then you wouldn't hitch due to the consistency.
Why do I think this? Ran multiple tests and on a server with NO consistency I froze up for under half the time for first engagement related freezes.
EDIT: I've actually done more tests recently with someone helping me out and discovered the following. Load time is not only hindered by disk read speed but also communication to the server when loading.
Example: Loading a listen server on an SSD with default consistency takes about 15s or so (I can't actually remember the numbers but look at them relative to others). If you load a listen server the same way after putting the entire game on a ramdrive it will load in about 3s.
However, connecting to a server (default consistency) on an SSD is about 20s to load, and on a ram is about the same, maybe a little less so 18s.
Connecting to a server with no consistency checking on an SSD is about a 15s load and on a ramdrive is about 5s.
So if you have a standard HDD then your load time is being bottlenecked by read speed. Once you get an SSD it's being bottlenecked by your connection to the server more than anything else (sending a ton of hashes and waiting for replies maybe?).
EDIT: Hitches didn't disappear for me with consistency off, they just weren't as bad. Easy example of a MASSIVE hitch, joining marine/alien/spectator after first joining a server. That first time it freezes for ages, no, this isn't it loading things. It's because it's checking consistency, with none it's instant (well almost) every time first time.
This is just a sequence of many hitches one after the other resulting in one big freeze. Unlike say the shell casing which is just one thing and as such is a smaller freeze.
If you feel like testing, i added it to the NSL mod (sv_nsltextures to toggle it on/off). I dont think it will completely resolve hitching on team joins, but I would be curious to see if you think it helps hitching during a round.