On Netcode/Hitreg/Lag (-Compensation)/etc - Compliation of issues
Omega_K2
Join Date: 2011-12-25 Member: 139013Members, Reinforced - Shadow
Join Date: 2011-12-25 Member: 139013Members, Reinforced - Shadow
To summarize things related to hitreg, netcode, lag, lag compensation and other things. If someone has more to add, or videos/screens to contribute to some issue (or even to invalidate some issues) just post below.
1. General things one should know about:
a) Update rate is fixed to 20 ( i.e. once per 50ms)
b) Interp is at 100ms (can be changed by some server mods)
c) What you see is what you hit approach; marines can you shoot you if they see you on your screen, even if you aren't
d) Hitbox is close to the model (more of a simplified model; Skulk)
e) There have been some issues with hitreg in the past that have been fixed, such as the animation issue (hitreg being off when animation plays)
f) LMG has random spread (close range*, far range*)
g) Shotgun has fixed, but randomly rotating spread (8 pellets out, 8 pellets inner circle, 1 middle; close range, far range)
2. Non hitreg/netcode Issues:
a) Skulk head is too small for all shotgun pellets to hit; i.e. you can not oneshot a skulk from front unless in your face
b) LMG spread can make you miss with good aim, especially on range
c) LMG has a "double-fire" issue, it fires twice when you press your mouse once (sometimes), so it's possible to miss two (or hit two) shots with just fireing once
d) There seems to be slight input lag from "firing" and actually "shooting"; most noticeable with shotgun
3. Issues:
a) Server tickrate breaking in will affect hitreg/compensation negatively, a server not running at 30 tick, will cause some shots or pellets to miss
b) Packet loss can potentially cause players to warp around "lag", resulting in hard-to-hit players
c) Players with high ping, still can shoot what they see, itensifiying the "i've been killed around a corner"
d) Hit feedback is slow/delayed, causing you to notice damage only after several bullets have been shoot (due to high interp etc); possibly more stuff causing it to be delayed further besides netcode
e) Sometimes (tm!) shooting a player that is very close to you will cause all bullets to miss; you can replicate this with bots or players, doesn't seem to matter. I suspect this might be due to either that you're "dead" on the server already or some issue with collisions (i.e. other model glitching though yours). This is VERY bad with shotgun, as it can make a sure-kill into a 0-dmg you're-dead nightmare
f) Fixed updaterate should be acutally be modifiyable
g) Due to high interp, it's possible to go though tunnels/phase gates, but not actually appearing on the other side; you are resetting in position (i.e. still in tunnel or on the other phase gates); also includes building buildings, it's possible to see that a building is at 100% percent/done, and if you stop too quickly it will be left unbuild at 99%.
h) Tickrate Variance
The server is suffering from tickrate variance, which make ticks take inconsistent time. In theory with a tickrate of 1/30 a tick should result in a time of 33.33 ms for each tick, however this is not the case.
Server has pretty high flunctuations in the tickrate:
Personal findings (with the lua data recording plugin made by Ghoul), on a 22-slot server that has a stable 30 tickrate. Recoded over ~40 mins of summit.
1GBit/s, 32GB mem, 4770k, SSD RAID1, Build 260, linux 64bit ubuntu server with lowlatency kernel
(Large image - download suggested)
http://omegak2.net/tick.png
Legend (x = 1 tick per pixel, y = 1 ms per 10 pixels):
Green - Time taken to process the frame
Red line - "should be" processing time
Blue line - Standard deviation based of ideal value
Some other stats:
Avarage: 0.034s
STDDEV: 0.0132s
Max: 2.3769s (lagspike anyone?)
Min: 0.0035s
z) Other unaccounted for issues that make bullets miss/not register properly (visible hitsprites, but no dmg); also see below
4. Suspected issues:
a) Low FPS seems also like a possible cause warping
Might be due to lower updaterate and inconsitency in the frames. Frames below 20 should not be able to send enough updates, and bumpy fps might send updates irreguarily causing "inconsistent" behaviour in the hitreg.
b) It seems that higher server load (increased player count) causes hitreg to suffer;
Even with a stable 30 tickrate and cpu < 100%, it seems that it becomes significantly worse as the load increases on the server applicated.
It might have to do somthing with frame/tick discrepancy, which means that some frames/tick take siginificantly longer to calcuate then others; example:
Tickrate 4 (for simplicity); each frame should take 250 ms:
Tick 1: 250ms - normal
Tick 2: 450ms - tick took significantly longer then expected
Tick 3: 100ms - tick took significantly less then expected
Tick 4: 200ms - tick took sightly less then expected
This would result in inconsistent results, as some things proccessed in tick 2 for example would take longer then otherwise; I think might also cause bullets to "disappear" or behave wierd. Also other unaccounted for issues.
I'm guessing this happens because you see odd behaviour on FULL, BUSY and LARGE servers, such as 24 player game is usually much worse then a 12 game, even if the server doesn't "lag" and runs at tickrate 30; chances are that some ticks simply take much longer then others to process causing the issue.
Notes:
* The position in the lmg spread screenshots is not related to the range shot from - I've actually moved closer for a more visible result :P
1. General things one should know about:
a) Update rate is fixed to 20 ( i.e. once per 50ms)
b) Interp is at 100ms (can be changed by some server mods)
c) What you see is what you hit approach; marines can you shoot you if they see you on your screen, even if you aren't
d) Hitbox is close to the model (more of a simplified model; Skulk)
e) There have been some issues with hitreg in the past that have been fixed, such as the animation issue (hitreg being off when animation plays)
f) LMG has random spread (close range*, far range*)
g) Shotgun has fixed, but randomly rotating spread (8 pellets out, 8 pellets inner circle, 1 middle; close range, far range)
2. Non hitreg/netcode Issues:
a) Skulk head is too small for all shotgun pellets to hit; i.e. you can not oneshot a skulk from front unless in your face
b) LMG spread can make you miss with good aim, especially on range
c) LMG has a "double-fire" issue, it fires twice when you press your mouse once (sometimes), so it's possible to miss two (or hit two) shots with just fireing once
d) There seems to be slight input lag from "firing" and actually "shooting"; most noticeable with shotgun
3. Issues:
a) Server tickrate breaking in will affect hitreg/compensation negatively, a server not running at 30 tick, will cause some shots or pellets to miss
b) Packet loss can potentially cause players to warp around "lag", resulting in hard-to-hit players
c) Players with high ping, still can shoot what they see, itensifiying the "i've been killed around a corner"
d) Hit feedback is slow/delayed, causing you to notice damage only after several bullets have been shoot (due to high interp etc); possibly more stuff causing it to be delayed further besides netcode
e) Sometimes (tm!) shooting a player that is very close to you will cause all bullets to miss; you can replicate this with bots or players, doesn't seem to matter. I suspect this might be due to either that you're "dead" on the server already or some issue with collisions (i.e. other model glitching though yours). This is VERY bad with shotgun, as it can make a sure-kill into a 0-dmg you're-dead nightmare
f) Fixed updaterate should be acutally be modifiyable
g) Due to high interp, it's possible to go though tunnels/phase gates, but not actually appearing on the other side; you are resetting in position (i.e. still in tunnel or on the other phase gates); also includes building buildings, it's possible to see that a building is at 100% percent/done, and if you stop too quickly it will be left unbuild at 99%.
h) Tickrate Variance
The server is suffering from tickrate variance, which make ticks take inconsistent time. In theory with a tickrate of 1/30 a tick should result in a time of 33.33 ms for each tick, however this is not the case.
Server has pretty high flunctuations in the tickrate:
Personal findings (with the lua data recording plugin made by Ghoul), on a 22-slot server that has a stable 30 tickrate. Recoded over ~40 mins of summit.
1GBit/s, 32GB mem, 4770k, SSD RAID1, Build 260, linux 64bit ubuntu server with lowlatency kernel
(Large image - download suggested)
http://omegak2.net/tick.png
Legend (x = 1 tick per pixel, y = 1 ms per 10 pixels):
Green - Time taken to process the frame
Red line - "should be" processing time
Blue line - Standard deviation based of ideal value
Some other stats:
Avarage: 0.034s
STDDEV: 0.0132s
Max: 2.3769s (lagspike anyone?)
Min: 0.0035s
z) Other unaccounted for issues that make bullets miss/not register properly (visible hitsprites, but no dmg); also see below
4. Suspected issues:
a) Low FPS seems also like a possible cause warping
Might be due to lower updaterate and inconsitency in the frames. Frames below 20 should not be able to send enough updates, and bumpy fps might send updates irreguarily causing "inconsistent" behaviour in the hitreg.
b) It seems that higher server load (increased player count) causes hitreg to suffer;
Even with a stable 30 tickrate and cpu < 100%, it seems that it becomes significantly worse as the load increases on the server applicated.
It might have to do somthing with frame/tick discrepancy, which means that some frames/tick take siginificantly longer to calcuate then others; example:
Tickrate 4 (for simplicity); each frame should take 250 ms:
Tick 1: 250ms - normal
Tick 2: 450ms - tick took significantly longer then expected
Tick 3: 100ms - tick took significantly less then expected
Tick 4: 200ms - tick took sightly less then expected
This would result in inconsistent results, as some things proccessed in tick 2 for example would take longer then otherwise; I think might also cause bullets to "disappear" or behave wierd. Also other unaccounted for issues.
I'm guessing this happens because you see odd behaviour on FULL, BUSY and LARGE servers, such as 24 player game is usually much worse then a 12 game, even if the server doesn't "lag" and runs at tickrate 30; chances are that some ticks simply take much longer then others to process causing the issue.
Notes:
* The position in the lmg spread screenshots is not related to the range shot from - I've actually moved closer for a more visible result :P
Comments
EDIT: Gonna move my post here from the hitbox thread as this is a better thread for it.
Example: A marine circle strafes an alien that doesn't move. The marine has an unstable Internet connection with >100 ms ping. The marine player sends updates: A is pressed down, mouse moved a bit to the right (to circle the alien). Because of the Internet connection, these updates will be partially lost or delayed.
So if the server extrapolated these commands, the marine should effectively stop rotating (no mouse updates come in) and just strafe to the left (last update was A is pressed down) for a few ms, until updates come in again. That doesn't seem to be what is happening. Instead the client seems to send his new absolute position, which means that the server has to warp the player from the predicted position to the position reported by the marine. On the marine side everything looks smooth.
But each time this happens the marine will warp for the alien and all other players, which is extremely annoying, because they don't know the real absolute position of the marine. Instead they get some wrong position by the server, that can radically change.
What should be happening: the marine just sends his key/mouse events but gets sent his new absolute position by the server so he will be punished for his unstable Internet connection - not the other players.
Since the server does the prediction all players will get valid positions and updates that result in smooth movement, except if those players have unstable Internet connection as well.
If there was an "absolute" server reality that would be forced to all the clients, you would experience a lot of missed bites and bullets, which would no doubt infuriate you. Psychologically, you are often not exactly aware of how you got shot/bitten, but you are seeing exactly whether you hit or miss your target because your own attack is always in the center of your attention.
Objectively it does not matter, one version or the other will always have inherent inaccuracy due to what are in fact pronounced relativistic effects resulting from the limited speed of information. There is no way to do a perfect solution once the network latency passes certain threshold.
No, with proper lag compensation you would still hit like you do now, with the difference that you are now also able to hit laggy players, because they don't warp anymore.
Even the classic Half-Life / ns1 did it better (server doesn't trust the clients on absolute positional information).
Generally trusting the client in competitive games is a bad idea in most cases. Every game developer will tell you that.
I'm not sure why this isn't what's already happening tbh.
Here's how it's done "right": Source Multiplayer Networking
Here's how it's done "right": Source Multiplayer Networking
We know that both (or more) players won't have the same story on their screen.
I/ Centralized information to the server
The clients send their updates and receive one big update for all players connected. The point is to have regular updates for everyone connected, not loosing anything. The more you deal with errors the more you loose time and the more you spoil the experience.
The server compensate for lag issues but SHOULD not wait anyone in order to send updates regularly. I feel and think (but not sure) that the spikes are due to something is waited.
See it as a tape on which every connected player is represented by a color. This tape never stops. Even if there are blanks because update by players aren't received. So every X millisecond update are sent by server to player depending on their respective negotiated rates.
The same goes for client. Message sent should be sent regularly and not depending on rendering engine or something else as it is suspected.
Basically : Send little message often, send bigger messages a little less. Don't worry we're on a millisecond unit... it's tight, right!
then comes...
II/ Lag compensation
The lag is compensated by 2 main ways.
A) Testing and giving scores to every connection. Example: Actually if you connect through a VPN you have a list of server that are a little different from your usual location. Simply because you look like you're closer to them than in reality. The pings in the scoreboard aren't what it should be in reality (player-VPN/VPN-Server).
So having a synchronization tool like NTP (Network time protocol) that can give an average of the real lag will be the first brick to start with.
*3 parameters should be implemented
-The average lag (in milliseconds) evaluated regularly during the game (every 10 secs give or take).
-A statistic about stability. Messages arrive sooner or later compare to the average. The stat gives the amplitude of variation.
-A statistic about lost message or unusable message.
Then the server and the client can tune the connection in order to make the most of it (helped by the score). It's a standard negotiation. Ex: A unstable connection will not be good if the update rates are to high. It doesn't solve any problem.
So we have 3 possible cases
-Steady connection: Good ping, stable (no variation range), no loss.
-Average connection : Good ping, deviance can occur frequently, no loss.
-Bad connection : ping above 100, high variation, loss.
Note: everybody try to connect to low ping server isn't it?
The only real case is the second which should be the most frequently seen.
**Sometime the network route to the server isn't the same as the route from the server to the player. You can't do anything about that unless you are an AS owner. In that case you don't need to worry about your connection isn't it ?
**Player suffer from interference on his line (ADSL) that disrupt the bandwidth or force to resynchronize from time to time.
**Network are bottlenecked. Not common. Depend on many factors like the size of link, time of day, etc. The higher the bandwidth the higher the probabilities of micro lags.
Lag compensation is a set of rule that goes from prediction (position, move etc) and rules for averaging /lowering damage of any "hit" (which is done by server).
The worst a connection is :
-the more the hit registration will suffer. Will be uncertain.
-the more the compensation will have to average things out.
But (and it's big) every player have a stable connection to the server as it has been tuned based on the scores and negotiations. So it is a stable connection. The lag will seem less problematic as it is today in this kind of games.
It's will not be perfect as lag is unpredictable but far better than the tick methods. If you know about the bullet warping from the left of the head to the right and not hiting (the head), you know what i'm talking about.
With that kind of tool it would be possible to draw line for bullets (instead of the warping bullet tick method) and count damage in a fair way.
Then maybe...
III/Anti-cheat things
The server can test a lot of things in the game like :
Do player see each other ?
Do player are distant by more than X ?
So the server can send correct update with true X,Y,Z depending on the results of these tests. Else it give some 0,0,0 or a defined location by map file.
Etc..
Put simply there is little that can be done about people with bad connections warping. The same things happen in Source also, its just usually less noticeable (because of higher update rates). A few dropped updates here and there accounts for much less at 60/s over 20.
Clients run prediction in NS2 just like source, but the server is still absolute. The position you see of someone else on your client is always 'correct', as it is the position the server told you. However that does not mean that his position will not be corrected due to extrapolation errors, which is what you perceive as 'warping'. You can still shoot and hit that player so long as you aim directly at him at any time, however obviously the warping can be quite offputting to smooth tracking. Generally speaking playing with lost/delayed client updates is much worse for the person with the connection problems (trust me I know, i have terrible internet). Players with bad connections used to warp and be almost unkillable in NS1 also, the problem is not exclusive to NS2 (remember nagadasts fade?)
The larger problem I see with NS2 is not so much individual players warping, its when everyone starts warping because of choke. Most of the warping I see happening is because of choke issues, which are amplified when there are more players around, and/or more things happening at once.
Source uses a high tick rate because the server does not trust the position of the client. The clients just send input (W, A, S, D, etc.) so to get accurate and snappy movement the rate has to be high.
No, I haven't seen unhittable players in NS1 with a ping just above 100 ms, or any other HL/Source game for that matter. I have never seen players warping in those games when they had a ping just above 100ms.
The ns2 server does NOT seem to be master regarding player position.
Let me clarify again:
ns2: Player A circles player B. A sends his updates (including absolute position) to server. Some of these updates get delayed. Player A keeps moving locally in a circle (he doesn't notice anything and can do damage to B normally). New updates finally arrive just after the old delayed updates - the server sees the huge difference in the position but doesn't care, he just pushes the new position to B. Warping occurs. B has a hard time hitting A.
In Source the server simulates where the player is moving according to the inputs he sends. Warping as described above cannot occur. The player with the fluctuating ping has the disadvantage.
What's so special about ns2's amount of network data? Please explain.
The stuff producing the most (amount AND frequency) traffic is movement/shooting input, no?
I don't believe you. Unhittable players with 50ms ping? Guess it was just your aim..
Maybe this thread will turn out more amusing the the guy claiming to play in CEVO TF2.
Still assuming that that is true, I find it dumb when games give laggy players the advantage (COD and Warframe)
I like it when games are built in a way where laggy players are given a disadvantage.
Post a video showing the other players ping, the warping and your netgraph. Until then as I said I don't believe you.
That is exactly what is happening. Client-side input prediction is obviously crucial for a smooth experience, but those things are not mutually exclusive. Client-side prediction is an addon to this.
No. Try it yourself if you don't believe me:
Connect to a Half-Life server with a high ping. Run around and try to shoot people. It's a mess.
Connect to a ns2 server with a high ping. You can run around, shoot and hit people as if the server is running locally. For the other players you are a warping mess.
Yes, when the server chokes that's another big problem. Tried a 24 slot ns2 server yesterday - unplayable. Whenever I entered a room with ~3 other players choke and warping started.
You lament about people providing bad information, yet you do so yourself. First of all, what the server "tells" you is correct only the instant it generates the packet, when it arrives and is interpreted by the client, it's no longer "correct", because a certain amount of time has passed, and the gameworld has moved on.
Moreover, the frames you see in your client is only partially based on what the server tells you, because there is an update only each 50ms or less, in fact, what you see most of the time is the result of client-side run prediction (extrapolation). Most of the time, if the lag is small enough, the error is small enough not to matter, otherwise a correction perceived as banding or warping must be made to maintain consistency.
Look at the night sky, you will see stars that no longer exist, but it may take hundreds of years for the information to arrive to us, so that we may see the supernova that killed them. There is no "correct" version of "now" in a relativistic world where information travels with finite speed. Does the star exist? That question has no sense until you specify: relative to what observer?
Other players reported I was warping badly, but I was running around smoothly as if everything was perfect...
edit: Not at all. What I expect is that players with unstable Internet connection do not warp for all other players.
And yes anyone with a good connection will generally not warp in either game, ns2 or ns1 or any goldsource game.. thats hardly a useful comparison.
NS2 is more lenient with its client side prediction, but clients are by no means dictating their position to the server.
Now NS2 may be back processing delayed inputs from the client, but that is not something anyone here would be able to say definitively, thats something Max or Dushan would need to answer.
If a client sends his updates to the server but drops a few packets, then there's a gap.
The server doesn't seem to extrapolate (keep the client moving in the direction he was moving before until packets come in again) nor does he tell the client: "hey, last time I saw you, you were 5m away from the new position you just reported, please go back".
Instead the server seems to accept the new client position, even if it is a few meters away, and sends that to other players - they see warping.