Bug 222074 - TCP stall with rfc1323 enabled
Summary: TCP stall with rfc1323 enabled
Status: Closed Works As Intended
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.1-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-09-05 16:44 UTC by Aragon Gouveia
Modified: 2017-09-08 16:47 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Aragon Gouveia 2017-09-05 16:44:11 UTC
I've stumbled upon some strange behaviour when RFC1323 is enabled in conjunction with multiple layers of MPD interfaces.  TCP transfers stall, and with net.inet.tcp.log_debug enabled I see one kernel message get logged when the stall occurs:

Sep  5 18:12:28 <kern.debug> roo kernel: TCP: [213.138.116.73]:80 to [192.168.104.0]:48841 tcpflags 0x12<SYN,ACK>; tcp_do_segment: Timestamp not expected, no action

My network setup looks like this:

192.168.104.0 [ng1] <-> MPD L2TP <-> 197.215.183.131 [ng0] <-> MPD PPPoE <-> Internet <-> 213.138.116.73

So in words, I'm running an L2TP tunnel over a PPPoE tunnel to my ISP, and all these layers operate on the same host that's experiencing the RFC1323 related TCP issue.

213.138.116.73 above is pkg.FreeBSD.org, and 192.168.104.0 is the local tunnel address on the affected host.  When I do an HTTP GET, eg.

curl -o /dev/null http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/All/bird-1.6.3_3.txz

The transfer stalls after a few KB, and a tcpdump of ng1 shows this:

18:28:13.643389 IP 192.168.104.0.35277 > 213.138.116.73.80: Flags [S], seq 353584016, win 65535, options [mss 1360,nop,wscale 6,sackOK,TS val 764211736 ecr 0], length 0
18:28:13.835812 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [S.], seq 4183396526, ack 353584017, win 65535, options [mss 1310,nop,wscale 11,sackOK,TS val 3582066608 ecr 764211736], length 0
18:28:13.835882 IP 192.168.104.0.35277 > 213.138.116.73.80: Flags [.], ack 1, win 1043, options [nop,nop,TS val 764211928 ecr 3582066608], length 0
18:28:13.835970 IP 192.168.104.0.35277 > 213.138.116.73.80: Flags [P.], seq 1:127, ack 1, win 1043, options [nop,nop,TS val 764211928 ecr 3582066608], length 126: HTTP: GET /FreeBSD:11:amd64/quarterly/All/bird-1.6.3_3.txz HTTP/1.1
18:28:14.028199 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], ack 127, win 512, options [nop,nop,TS val 3582066800 ecr 764211928], length 0
18:28:14.028317 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 1:1299, ack 127, win 512, options [nop,nop,TS val 3582066800 ecr 764211928], length 1298: HTTP: HTTP/1.1 200 OK
18:28:14.028482 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 1299:2597, ack 127, win 512, options [nop,nop,TS val 3582066800 ecr 764211928], length 1298: HTTP
18:28:14.028498 IP 192.168.104.0.35277 > 213.138.116.73.80: Flags [.], ack 2597, win 1023, options [nop,nop,TS val 764212121 ecr 3582066800], length 0
18:28:14.028524 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 2597:3895, ack 127, win 512, options [nop,nop,TS val 3582066800 ecr 764211928], length 1298: HTTP
18:28:14.028532 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 3895:5193, ack 127, win 512, options [nop,nop,TS val 3582066800 ecr 764211928], length 1298: HTTP
18:28:14.028744 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 5193:6491, ack 127, win 512, options [nop,nop,TS val 3582066800 ecr 764211928], length 1298: HTTP
18:28:14.028755 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 6491:7789, ack 127, win 512, options [nop,nop,TS val 3582066800 ecr 764211928], length 1298: HTTP
18:28:14.028828 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 7789:9087, ack 127, win 512, options [nop,nop,TS val 3582066800 ecr 764211928], length 1298: HTTP
18:28:14.028944 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 9087:10385, ack 127, win 512, options [nop,nop,TS val 3582066800 ecr 764211928], length 1298: HTTP
18:28:14.029071 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 10385:11683, ack 127, win 512, options [nop,nop,TS val 3582066800 ecr 764211928], length 1298: HTTP
18:28:14.220094 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 11683:12981, ack 127, win 512, options [nop,nop,TS val 3582066992 ecr 764212121], length 1298: HTTP
18:28:14.220112 IP 192.168.104.0.35277 > 213.138.116.73.80: Flags [.], ack 2597, win 1043, options [nop,nop,TS val 764212313 ecr 3582066800,nop,nop,sack 1 {11683:12981}], length 0
18:28:14.220136 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 12981:14279, ack 127, win 512, options [nop,nop,TS val 3582066992 ecr 764212121], length 1298: HTTP
18:28:14.220159 IP 192.168.104.0.35277 > 213.138.116.73.80: Flags [.], ack 2597, win 1043, options [nop,nop,TS val 764212313 ecr 3582066800,nop,nop,sack 1 {11683:14279}], length 0
18:28:14.412476 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 14279:15577, ack 127, win 512, options [nop,nop,TS val 3582067184 ecr 764212313], length 1298: HTTP
18:28:14.412489 IP 192.168.104.0.35277 > 213.138.116.73.80: Flags [.], ack 2597, win 1043, options [nop,nop,TS val 764212505 ecr 3582066800,nop,nop,sack 1 {11683:15577}], length 0
18:28:14.604109 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 2597:3895, ack 127, win 512, options [nop,nop,TS val 3582067376 ecr 764212505], length 1298: HTTP
18:28:14.604127 IP 192.168.104.0.35277 > 213.138.116.73.80: Flags [.], ack 3895, win 1023, options [nop,nop,TS val 764212697 ecr 3582067376,nop,nop,sack 1 {11683:15577}], length 0
18:28:14.604236 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 3895:5193, ack 127, win 512, options [nop,nop,TS val 3582067376 ecr 764212505], length 1298: HTTP
18:28:14.604257 IP 192.168.104.0.35277 > 213.138.116.73.80: Flags [.], ack 5193, win 1023, options [nop,nop,TS val 764212697 ecr 3582067376,nop,nop,sack 1 {11683:15577}], length 0
18:28:14.604287 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 5193:6491, ack 127, win 512, options [nop,nop,TS val 3582067376 ecr 764212505], length 1298: HTTP
18:28:14.604298 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 6491:7789, ack 127, win 512, options [nop,nop,TS val 3582067376 ecr 764212505], length 1298: HTTP
18:28:14.604357 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 7789:7837, ack 127, win 512, options [nop,nop,TS val 3582067376 ecr 764212505], length 48: HTTP
18:28:14.604367 IP 192.168.104.0.35277 > 213.138.116.73.80: Flags [.], ack 5193, win 1043, options [nop,nop,TS val 764212697 ecr 3582067376,nop,nop,sack 2 {7789:7837}{11683:15577}], length 0
18:28:14.796498 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 7837:9135, ack 127, win 512, options [nop,nop,TS val 3582067568 ecr 764212697], length 1298: HTTP
18:28:14.796513 IP 192.168.104.0.35277 > 213.138.116.73.80: Flags [.], ack 5193, win 1043, options [nop,nop,TS val 764212889 ecr 3582067376,nop,nop,sack 2 {7789:9135}{11683:15577}], length 0
18:28:14.796533 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 9135:10433, ack 127, win 512, options [nop,nop,TS val 3582067568 ecr 764212697], length 1298: HTTP
18:28:14.796544 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 10433:10481, ack 127, win 512, options [nop,nop,TS val 3582067568 ecr 764212697], length 48: HTTP
18:28:14.988262 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 10481:11683, ack 127, win 512, options [nop,nop,TS val 3582067760 ecr 764212889], length 1202: HTTP
18:28:14.988275 IP 192.168.104.0.35277 > 213.138.116.73.80: Flags [.], ack 5193, win 1043, options [nop,nop,TS val 764213081 ecr 3582067376,nop,nop,sack 2 {10481:15577}{7789:9135}], length 0
18:28:15.204013 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 5193:6491, ack 127, win 512, options [nop,nop,TS val 3582067976 ecr 764213081], length 1298: HTTP
18:28:15.204030 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 6491:6503, ack 127, win 512, options [nop,nop,TS val 3582067976 ecr 764213081], length 12: HTTP
18:28:15.820052 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 5193:6491, ack 127, win 512, options [nop,nop,TS val 3582068592 ecr 764213081], length 1298: HTTP
18:28:15.820066 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 6491:6503, ack 127, win 512, options [nop,nop,TS val 3582068592 ecr 764213081], length 12: HTTP
18:28:16.852097 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 5193:6491, ack 127, win 512, options [nop,nop,TS val 3582069624 ecr 764213081], length 1298: HTTP
18:28:16.852110 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 6491:6503, ack 127, win 512, options [nop,nop,TS val 3582069624 ecr 764213081], length 12: HTTP
18:28:18.716112 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 5193:6491, ack 127, win 512, options [nop,nop,TS val 3582071488 ecr 764213081], length 1298: HTTP
18:28:18.716125 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 6491:6503, ack 127, win 512, options [nop,nop,TS val 3582071488 ecr 764213081], length 12: HTTP
18:28:22.243953 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 5193:6491, ack 127, win 512, options [nop,nop,TS val 3582075016 ecr 764213081], length 1298: HTTP
18:28:22.243965 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 6491:6503, ack 127, win 512, options [nop,nop,TS val 3582075016 ecr 764213081], length 12: HTTP
18:28:29.099858 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 5193:6491, ack 127, win 512, options [nop,nop,TS val 3582081872 ecr 764213081], length 1298: HTTP
18:28:29.099872 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 6491:6503, ack 127, win 512, options [nop,nop,TS val 3582081872 ecr 764213081], length 12: HTTP
18:28:42.614902 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 5193:6491, ack 127, win 512, options [nop,nop,TS val 3582095387 ecr 764213081], length 1298: HTTP
18:28:42.614915 IP 213.138.116.73.80 > 192.168.104.0.35277: Flags [.], seq 6491:6503, ack 127, win 512, options [nop,nop,TS val 3582095387 ecr 764213081], length 12: HTTP

If I disable net.inet.tcp.rfc1323, the transfer succeeds with no kernel messages.

If I enable net.inet.tcp.rfc1323, but I route IP traffic via the PPPoE interface (one less NG/MPD layer), the transfer succeeds with 1 kernel message:

Sep  5 18:39:45 <kern.debug> roo kernel: TCP: [213.138.116.73]:80 to [197.215.183.131]:59766 tcpflags 0x12<SYN,ACK>; tcp_do_segment: Timestamp not expected, no action
Comment 1 Aragon Gouveia 2017-09-05 21:37:28 UTC
Please close this bug report.

After a lot of experimenting, the problem turned out to be caused by receive checksum offloading on a virtio(4) interface, presumably resulting in TCP packets getting dropped.  Although TCP transfers worked with rfc1323 disabled, I later noticed they were running much too slowly.
Comment 2 Mark Linimon freebsd_committer freebsd_triage 2017-09-08 16:47:38 UTC
Closed at submitter's request.