Bug 210860 - [pf] [ip6] Incorrect TCP checksums after rdr
Summary: [pf] [ip6] Incorrect TCP checksums after rdr
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.3-STABLE
Hardware: Any Any
: --- Affects Many People
Assignee: freebsd-pf mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-07-05 22:54 UTC by Joao Vieira
Modified: 2016-11-01 20:22 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Joao Vieira 2016-07-05 22:54:39 UTC
PF seems to incorrectly computes the checksum of IPv6 TCP packets that it rewrites as part of a simple rdr rule.

E.g., with the following rules:

table <forward_subnets> {2001:16d8:ee03::cafe:d00d/128,!::1}
rdr pass log on lo0 inet6 proto tcp to <forward_subnets> -> ::1 port 12300
pass out log route-to lo0 inet6 proto tcp to <forward_subnets> keep state

and while running nc -l ::1 12300 trying nc -vz 2001:16d8:ee03::cafe:d00d 80 to see if any packets get to nc listening on 12300 shows that nothing is received although tcpdump -n -e -ttt -i pflog0 -vvv shows the following incorrect checksum in rdr:

00:00:00.000000 rule 0..16777216/0(match): pass out on vtnet0: (flowlabel 0x481b8, hlim 64, next-header TCP (6) payload length: 40) 2a03:b0c0:1:d0::425:1001.29559 > 2001:16d8:ee03::cafe:d00d.80: Flags [S], cksum 0xafd2 (incorrect -> 0x7083), seq 417394668, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 240470 ecr 0], length 0

00:00:00.000049 rule 0..16777216/0(match): rdr in on lo0: (flowlabel 0x481b8, hlim 64, next-header TCP (6) payload length: 40) 2a03:b0c0:1:d0::425:1001.29559 > ::1.12300: Flags [S], cksum 0x3fff (incorrect -> 0x00b0), seq 417394668, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 240470 ecr 0], length 0


tcpdump'ing lo0 repeatedly shows "cksum 0xafd2 (incorrect -> 0x...)":

22:51:58.407819 IP6 (flowlabel 0x4766f, hlim 64, next-header TCP (6) payload length: 40) fbsd.yubo.be.10904 > 2001:16d8:ee03::cafe:d00d.http: Flags [S], cksum 0xafd2 (incorrect -> 0x12d1), seq 1237643503, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 401406 ecr 0], length 0

22:52:01.438778 IP6 (flowlabel 0x4766f, hlim 64, next-header TCP (6) payload length: 40) fbsd.yubo.be.10904 > 2001:16d8:ee03::cafe:d00d.http: Flags [S], cksum 0xafd2 (incorrect -> 0x06fb), seq 1237643503, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 404436 ecr 0], length 0


Note that the inet version of the same rules do work as expected, e.g.
table <forward_subnets> {188.113.88.193/32,!127.0.0.1/8}
rdr pass on lo0 inet proto tcp to <forward_subnets> -> 127.0.0.1 port 12300
pass out route-to lo0 inet proto tcp to <forward_subnets> keep state
Comment 1 Kristof Provost freebsd_committer 2016-07-06 10:24:10 UTC
It'd be slightly surprising to have a checksum issue specifically for IPv6, because the IPv6 header doesn't have a checksum, and I'd expect the TCP checksum handling to be the same between v4 and v6.

There is an issue with rdr to ::1 though. There's a discussion of it in #172648.
I wonder if that's not what you're running into. It's worth trying your rdr rule again, but with '-> <GUA>' instead of '-> ::1'.
Comment 2 Joao Vieira 2016-07-06 22:11:29 UTC
Hello Kristof,

Thanks for replying. As you suggested, I tried with the following rules (replacing ::1 with GUA):

table <forward_subnets> {2001:16d8:ee03::cafe:d00d/128,!::1}
rdr pass log on lo0 inet6 proto tcp to <forward_subnets> -> 2a03:b0c0:1:d0::42e:f001 port 12300
pass out log route-to lo0 inet6 proto tcp to <forward_subnets> keep state

but the problem persists. I tried ifconfig lo0 -lro -tso and ifconfig vtnet0 -lro -tso and after that I saw less checksum errors but still saw one on pflog0. Nothing got to the application (nc -l 2a03:b0c0:1:d0::42e:f001 12300).

# tcpdump -n -e -ttt -i pflog0 -vvv
00:00:00.000000 rule 0..16777216/0(match): pass out on vtnet0: (flowlabel 0xc4cbb, hlim 64, next-header TCP (6) payload length: 40) 2a03:b0c0:1:d0::42e:f001.39668 > 2001:16d8:ee03::cafe:d00d.80: Flags [S], cksum 0x8fdc (incorrect -> 0x78fc), seq 2180195351, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 10681721 ecr 0], length 0
00:00:00.000093 rule 0..16777216/0(match): rdr in on lo0: (flowlabel 0xc4cbb, hlim 64, next-header TCP (6) payload length: 40) 2a03:b0c0:1:d0::42e:f001.39668 > 2a03:b0c0:1:d0::42e:f001.12300: Flags [S], cksum 0x3965 (correct), seq 2180195351, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 10681721 ecr 0], length 0

# tcpdump -n -e -ttt -i lo0 -vvv
00:00:00.000000 AF IPv6 (28), length 84: (flowlabel 0xd1846, hlim 64, next-header TCP (6) payload length: 40) 2a03:b0c0:1:d0::42e:f001.59007 > 2001:16d8:ee03::cafe:d00d.80: Flags [S], cksum 0x18ea (correct), seq 927681064, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 10630808 ecr 0], length 0
00:00:00.000125 AF IPv6 (28), length 84: (class 0x01, flowlabel 0x3450f, hlim 64, next-header TCP (6) payload length: 40) 2001:16d8:ee03::cafe:d00d.80 > 2a03:b0c0:1:d0::42e:f001.59007: Flags [S.], cksum 0xb883 (correct), seq 3088270607, ack 927681065, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 3250889069 ecr 10630808], length 0
00:00:03.022264 AF IPv6 (28), length 84: (class 0x0b, flowlabel 0x3450f, hlim 64, next-header TCP (6) payload length: 40) 2001:16d8:ee03::cafe:d00d.80 > 2a03:b0c0:1:d0::42e:f001.59007: Flags [S.], cksum 0xb883 (correct), seq 3088270607, ack 927681065, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 3250889069 ecr 10630808], length 0
00:00:00.000108 AF IPv6 (28), length 64: (hlim 64, next-header TCP (6) payload length: 20) 2a03:b0c0:1:d0::42e:f001.59007 > 2001:16d8:ee03::cafe:d00d.80: Flags [R], cksum 0xb7ee (correct), seq 927681065, win 0, length 0
Comment 3 Andrey V. Elsukov freebsd_committer 2016-07-11 13:19:06 UTC
(In reply to Kristof Provost from comment #1)
> It'd be slightly surprising to have a checksum issue specifically for IPv6,
> because the IPv6 header doesn't have a checksum, and I'd expect the TCP
> checksum handling to be the same between v4 and v6.

I didn't looked into the code, but despite the fact that IPv6 header doesn't have a checksum field, ULP checksums are handled differently in IPv4 and IPv6 due to different pseudo headers.
Comment 4 Joao Vieira 2016-11-01 18:46:49 UTC
Still present on 11.0-RELEASE-p1.

Were you able to reproduce it?
Comment 5 Kristof Provost freebsd_committer 2016-11-01 20:22:32 UTC
I've not had the time to look at this yet. It's on my list, but I can't promise when I'll have the time.