Resolved Report: [TF2] 2Fort servers experiencing lag spikes

[herrshaun]

Summary [TF2] 2Fort servers experiencing lag spikes

Detailed Explanation
For about a week or more, I have experienced small to large lag spikes during gameplay. Sometimes it may be as simple as my movement jerks for a second, but other times it may pause gameplay for almost an entire second. Every time this happens, I ask the chat and they're experiencing a similar issue.

I ran MTR for 1000 pings on MENTLE and these are the results:
|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.0.1 - 0 | 1405 | 1405 | 0 | 1 | 6 | 1 |
| HIDDEN - 0 | 1404 | 1404 | 4 | 20 | 94 | 16 |
| HIDDEN - 1 | 1401 | 1400 | 5 | 20 | 91 | 16 |
| HIDDEN - 0 | 1404 | 1404 | 5 | 20 | 95 | 13 |
| HIDDEN - 0 | 1404 | 1404 | 7 | 22 | 105 | 22 |
| 209.148.233.106 - 0 | 1404 | 1404 | 23 | 37 | 108 | 37 |
| No response from host - 100 | 282 | 0 | 0 | 0 | 0 | 0 |
| be40.br03.nyc06.pccwbtn.net - 43 | 522 | 300 | 28 | 41 | 82 | 52 |
| No response from host - 100 | 282 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 282 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 282 | 0 | 0 | 0 | 0 | 0 |
| 91.216.250.55 - 1 | 1393 | 1390 | 29 | 45 | 1940 | 42 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

NOTE: I hid the IPs of my local area for my privacy since they didn't show any issues. If you want the missing IPs/hosts and you are an admin, PM me and I'll provide them.

This is what I have observed simply from playing:

• This affects all 2fort Skial servers, both 24 and 32, including THE MENTLE INSTITUTION.
• The lag spikes are inconsistent but occur at least every 3 minutes.
• Ping tests to the server IP indicate inconsistent ping, but the highest I observed was 128 ms during a big spike.
• Running net_graph 1 doesn't show any increase in ping when the lag spikes occur.
• I hate to say it, but I played on a UGC server for an hour and did not experience any of these issues.

If you need any further data let me know.

 

[herrshaun]

From 2FORT+ | US 2:

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.0.1 - 0 | 3287 | 3287 | 0 | 1 | 7 | 1 |
| HIDDEN - 0 | 3287 | 3287 | 6 | 22 | 140 | 73 |
| HIDDEN - 0 | 3287 | 3287 | 5 | 20 | 98 | 24 |
| HIDDEN - 0 | 3287 | 3287 | 5 | 21 | 117 | 27 |
| HIDDEN - 0 | 3287 | 3287 | 8 | 24 | 127 | 42 |
| 209.148.233.106 - 0 | 3287 | 3287 | 23 | 38 | 139 | 37 |
| No response from host - 100 | 658 | 0 | 0 | 0 | 0 | 0 |
| be40.br03.nyc06.pccwbtn.net - 24 | 1710 | 1315 | 28 | 44 | 129 | 44 |
| No response from host - 100 | 658 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 658 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 658 | 0 | 0 | 0 | 0 | 0 |
| 91.216.250.31 - 1 | 3250 | 3240 | 53 | 69 | 670 | 69 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.0.1 - 0 | 1188 | 1188 | 0 | 1 | 6 | 1 |
| HIDDEN - 0 | 1188 | 1188 | 6 | 23 | 203 | 15 |
| HIDDEN - 0 | 1188 | 1188 | 7 | 22 | 148 | 18 |
| HIDDEN - 0 | 1187 | 1187 | 6 | 21 | 124 | 88 |
| HIDDEN - 0 | 1188 | 1188 | 10 | 25 | 93 | 15 |
| 209.148.233.106 - 0 | 1188 | 1188 | 23 | 40 | 160 | 30 |
| No response from host - 100 | 238 | 0 | 0 | 0 | 0 | 0 |
| be40.br03.nyc06.pccwbtn.net - 19 | 682 | 555 | 29 | 45 | 141 | 45 |
| No response from host - 100 | 238 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 238 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 238 | 0 | 0 | 0 | 0 | 0 |
| 91.216.250.31 - 1 | 1176 | 1173 | 54 | 70 | 961 | 60 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

 

[herrshaun]

Whatever server "be40.br03.nyc06.pccwbtn.net" is seems to be the problem child. This is from MENTLE today:

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.0.1 - 0 | 5814 | 5814 | 0 | 1 | 48 | 3 |
| HIDDEN - 0 | 5814 | 5814 | 5 | 20 | 69 | 21 |
| HIDDEN - 0 | 5814 | 5814 | 6 | 19 | 68 | 14 |
| HIDDEN - 1 | 5810 | 5809 | 5 | 20 | 81 | 20 |
| HIDDEN - 1 | 5807 | 5805 | 7 | 22 | 82 | 18 |
| 209.148.233.106 - 0 | 5814 | 5814 | 22 | 37 | 93 | 29 |
| No response from host - 100 | 1174 | 0 | 0 | 0 | 0 | 0 |
| be40.br03.nyc06.pccwbtn.net - 39 | 2294 | 1400 | 28 | 41 | 82 | 38 |
| No response from host - 100 | 1174 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 1174 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 1174 | 0 | 0 | 0 | 0 | 0 |
| 91.216.250.55 - 1 | 5774 | 5763 | 29 | 43 | 183 | 38 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

 

[herrshaun]

Interestingly, this time the one host was fine, but...

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.0.1 - 0 | 1334 | 1334 | 0 | 1 | 6 | 2 |
| HIDDEN - 0 | 1333 | 1333 | 6 | 19 | 73 | 19 |
| HIDDEN - 0 | 1333 | 1333 | 6 | 18 | 53 | 16 |
| HIDDEN - 0 | 1333 | 1333 | 6 | 18 | 42 | 22 |
| HIDDEN - 0 | 1333 | 1333 | 8 | 20 | 53 | 22 |
| 209.148.233.106 - 0 | 1333 | 1333 | 23 | 35 | 80 | 29 |
| No response from host - 100 | 267 | 0 | 0 | 0 | 0 | 0 |
| be40.br03.nyc06.pccwbtn.net - 1 | 1329 | 1328 | 28 | 41 | 65 | 47 |
| No response from host - 100 | 267 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 267 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 267 | 0 | 0 | 0 | 0 | 0 |
| 91.216.250.55 - 1 | 1314 | 1309 | 29 | 44 | 1732 | 38 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

Any knowledge of what's going on @Bottiger ?

 

[Bottiger]

There was a problem with our ddos protection provider in chicago which you omitted in your MTR. They said they fixed it on the 13th but I noticed there were still issues on the 14th. I don't see any problems with it now.

 

[herrshaun]

@Bottiger The problem has diminished but is not solved. I am still getting second-long lag spikes on all NA servers. This is from MENTLE:

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.0.1 - 0 | 1646 | 1646 | 0 | 1 | 84 | 0 |
| HIDDEN - 0 | 1646 | 1646 | 7 | 21 | 106 | 27 |
| HIDDEN - 0 | 1647 | 1647 | 7 | 20 | 67 | 15 |
| HIDDEN - 0 | 1647 | 1647 | 6 | 20 | 114 | 15 |
| HIDDEN - 0 | 1647 | 1647 | 8 | 24 | 76 | 20 |
| 209.148.233.106 - 0 | 1646 | 1646 | 23 | 38 | 148 | 40 |
| No response from host - 100 | 330 | 0 | 0 | 0 | 0 | 0 |
| be40.br03.nyc06.pccwbtn.net - 3 | 1498 | 1460 | 28 | 42 | 80 | 39 |
| No response from host - 100 | 330 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 330 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 330 | 0 | 0 | 0 | 0 | 0 |
| 91.216.250.55 - 1 | 1630 | 1626 | 30 | 46 | 2122 | 41 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.0.1 - 0 | 3383 | 3383 | 0 | 1 | 14 | 3 |
| HIDDEN - 0 | 3383 | 3383 | 6 | 20 | 122 | 27 |
| HIDDEN - 0 | 3383 | 3383 | 5 | 19 | 131 | 13 |
| HIDDEN - 1 | 3379 | 3378 | 5 | 19 | 96 | 31 |
| HIDDEN - 0 | 3383 | 3383 | 9 | 22 | 117 | 14 |
| 209.148.233.106 - 0 | 3383 | 3383 | 23 | 36 | 126 | 33 |
| No response from host - 100 | 678 | 0 | 0 | 0 | 0 | 0 |
| be40.br03.nyc06.pccwbtn.net - 2 | 3215 | 3172 | 28 | 42 | 124 | 41 |
| No response from host - 100 | 678 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 678 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 678 | 0 | 0 | 0 | 0 | 0 |
| 91.216.250.55 - 1 | 3359 | 3353 | 29 | 44 | 2253 | 37 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

Also I have only hidden hostnames local to my hometown to protect my location. I have not omitted any servers.

 

[herrshaun]

Some testimonials from other users from today coinciding with the lag spikes (I was in these servers at the time of):

2FORT | NY █████	02/16/2024 - 12:34:33 PM	🔗Cooper	AT THE WORST TIME
2FORT | NY █████	02/16/2024 - 12:34:30 PM	🔗Cooper	LAG

THE MENTLE INSTITUTION	02/16/2024 - 03:51:44 PM	🔗84est	wtf
THE MENTLE INSTITUTION	02/16/2024 - 03:51:42 PM	🔗84est	this lag is fucking insane

THE MENTLE INSTITUTION

02/16/2024 - 04:02:38 PM

🔗KingGamer617

SHIT LAG

THE MENTLE INSTITUTION

02/16/2024 - 04:05:31 PM

🔗Hyper37XD

Damn that was a good lag

THE MENTLE INSTITUTION

02/16/2024 - 04:08:24 PM

🔗martin80py

the lag is lobotomysing me

THE MENTLE INSTITUTION

02/16/2024 - 04:12:59 PM

🔗pillow

holy lag i cant

THE MENTLE INSTITUTION

02/16/2024 - 04:14:56 PM

🔗Yikes Incorporated™

THE LAG

THE MENTLE INSTITUTION

02/16/2024 - 04:29:01 PM

🔗System32.exe

lags

THE MENTLE INSTITUTION

02/16/2024 - 04:44:48 PM

🔗RAT LUNG (HAWAII LETTUCE)

wtf. lag here also.

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.0.1 - 0 | 1837 | 1837 | 0 | 2 | 29 | 1 |
| HIDDEN - 0 | 1837 | 1837 | 7 | 21 | 91 | 29 |
| HIDDEN - 0 | 1837 | 1837 | 6 | 21 | 108 | 25 |
| HIDDEN - 0 | 1837 | 1837 | 7 | 21 | 97 | 27 |
| HIDDEN - 0 | 1837 | 1837 | 9 | 24 | 112 | 14 |
| 209.148.233.106 - 0 | 1837 | 1837 | 24 | 38 | 120 | 29 |
| No response from host - 100 | 371 | 0 | 0 | 0 | 0 | 0 |
| be40.br03.nyc06.pccwbtn.net - 18 | 1094 | 903 | 28 | 43 | 139 | 64 |
| No response from host - 100 | 371 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 371 | 0 | 0 | 0 | 0 | 0 |
| No response from host - 100 | 371 | 0 | 0 | 0 | 0 | 0 |
| 91.216.250.55 - 1 | 1822 | 1818 | 29 | 45 | 545 | 46 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

@Bottiger Can you or another admin please remove the "Resolved" tag until this is fixed?

 

[Bottiger]

I assume you are only experience this issue on NY servers now?

 

[herrshaun]

I assume you are only experience this issue on NY servers now?

Is that where Mentle Institution is hosted? That's where I experienced the prime of all the issues. I'm on 2FORT+ NY right now and haven't noticed a lot of lag.
EDIT: NY+ may have just experienced a netsplit because a bunch of people dropped all at once.

 

[Bottiger]

Is it still happening?

 

[herrshaun]

Is it still happening?

I don't seem to notice it anymore. Other people have complained about lag so I have encouraged them to post their own WinMTR traces. Thanks for looking into it.