PF incorrectly computes the checksum of IPv6 TCP packets that it rewrites as part of a simple rdr rule that just changes the port number. Maybe it just does not update the checksum... The following IPv4 rule works properly: rdr pass inet proto tcp from any to 188.165.36.95 port 443 -> 188.165.36.95 port 8443 The following IPv6 rule does not work: rdr pass inet6 proto tcp from any to 2001:41d0:8:8822::1 port 443 -> 2001:41d0:8:8822::1 port 8443 Ports redirection happens properly, but on their way out, packets do not have the proper checksum and are eventually dropped by the client. Since this happens during TCP connection handshake, the only rewritten packets on their way out are SYN, ACK. This happens with and without scrub rules. The problem, or a similar problem, is discussed on mailing lists. Yet no PR seems to have been filed, probably because it was insufficiently documented. In both threads, it is linked with jails as jails are one of the use-case of rdr/nat with IPv6 : http://lists.freebsd.org/pipermail/freebsd-stable/2012-July/068987.html http://lists.freebsd.org/pipermail/freebsd-pf/2012-August/006710.html and http://lists.freebsd.org/pipermail/freebsd-pf/2012-December/006916.html Also, this bug might or might not be related to kern/172648 or kern/171733 However, unlike kern/172648, this bug has no known workaround. Indeed, the problem here is to be able to rewrite incoming packets on a privileged port and forward them to a local server, then get the response from the local server and forward it to the client after rewriting the source port. How-To-Repeat: Create a simple rdr rule on IPv6 and let everything else pass. rdr pass inet6 proto tcp from any to [IPv6 External IP] port 443 -> [IPv6 External IP] port 8443 Start a server on this port, for example with nc. Try to connect from outside on port 8443. It should work. Restart nc. Try to connect from outside port 443. The client will eventually timeout. Eventually, use tcpdump to record the packets and notice the checksums are incorrect on received SYN, ACK packets from port 443.
Responsible Changed From-To: freebsd-bugs->freebsd-pf Over to maintainer(s).
Hello FreeBSD team, hello Paul, I would like to confirm this. The original PR says it all. I have also reproduced it on FreeBSD 9.2R. FreeBSD 9.2-RELEASE-p2 #3 r258725 on amd64 The checksum is not updated, tcpdump says it clearly. The packet is being quietly dropped and never arrives at the service listening socket. Here the comparison for two rules for an intercepting HTTP proxy: # works (IPv4) rdr on $if_int inet proto tcp \ from any to !$net_int port www -> 127.0.0.1 port 8118 # incorrect checksum (IPv6) rdr on $if_int inet6 proto tcp \ from any to !$net_int port www -> ::1 port 8118 In my opinion, this is quite important. It costed me a day to find out what is going on and I have come to the same conclusion as Paul (independently). Yours Martin Sugioarto
I've replicated this issue as well, on 10.0-RELEASE, amd64. With jails running on a cloned lo1, outbound IPv6 works fine, but pf redirect traffic gets results in incorrect checksums and traffic being dropped. Loopback interfaces no longer seem to support the -txcsum6 or -rxcsum6 flags. Would love to have a fix for this, as it kind of breaks the "service jail" model for IPv6. Cheers, David
I was hit by this problem also, on 9.2-RELEASE. I think it might be caused by the workaround introduced in kern/170070. Ari S.
We also have encountered this issue. PF not working properly with IPv6 seems like a very serious problem. What needs to happen in order to advance this issue? How can we help? Thanks!
Hi all, I also have this bug on FreeBSD 10.0. I have nothing to add, everything have been said in Paul's post and the above comments.
(In reply to j.david.lists from comment #5) > We also have encountered this issue. > > PF not working properly with IPv6 seems like a very serious problem. > > What needs to happen in order to advance this issue? How can we help? > > Thanks! Can you look at a similar PR, 172648 and the suggested patch, and test it and report back ?
As this is the only PR with rdr and TCP checksum errors I guess I'm seeing the same problem with IPv4 on a pair of fresh 10.1-amd64 VMs (using KVM on Fedora 20). I make a request to VM1:$ext_if and can see the redirected SYNs arriving on VM2:$dmz_if in tcpdump. Redirects to a port on the same VM work, from one VMs the other VM's $dmz_if is reachable and I've changed the interface type from rtl8139 (showing up as re0) to e1000 (em0) to virtio (vtnet1) to make sure it's not a problem with the emulated NICs and the problem persists (I'm also NATing the redirected requests so the replys will be send to the redirecting VM instead of the default gateway but I see bad checksums with and without NAT). Relevant parts of my pf.conf: [...] table <webpool> persist { 10.0.0.11, 10.0.0.12 } [...] nat on $dmz_if inet proto tcp from !$dmz_if to <webpool> port $webports \ -> $dmz_if rdr on $ext_if proto tcp from !$dmz_if:network to $webserver port $webports \ -> <webpool> block all pass in proto tcp to port ssh # works for own and other <webpool> IPs: pass quick proto tcp to <webpool> port $webports [...] Maybe the same problem and more info required or a different problem and I should open a new PR? Regards, Florian
A commit references this bug: Author: eri Date: Wed Nov 19 13:31:09 UTC 2014 New revision: 274709 URL: https://svnweb.freebsd.org/changeset/base/274709 Log: pf(4) needs to have a correct checksum during its processing. Calculate checksums for the IPv6 path when needed before delving into pf(4) code as required. PR: 172648, 179392 Reviewed by: glebius@ Approved by: gnn@ Obtained from: pfSense MFC after: 1 week Sponsored by: Netgate Changes: head/sys/netpfil/pf/pf_ioctl.c
(In reply to commit-hook from comment #9) > A commit references this bug: > > Author: eri > Date: Wed Nov 19 13:31:09 UTC 2014 > New revision: 274709 > URL: https://svnweb.freebsd.org/changeset/base/274709 > > [...] I've upgraded one of my test-VMs from 10.1-RELEASE to this revision of 11-CURRENT and now redirected packets have correct checksums. Thanks eri!
Is there an easy way to backport this fix?
I second this:
It'd be great to get this merged to stable/10 at the very least, but stable/9 too would be nice.
A commit references this bug: Author: glebius Date: Fri Jan 23 18:15:16 UTC 2015 New revision: 277581 URL: https://svnweb.freebsd.org/changeset/base/277581 Log: Merge r274709 by eri@: deal with IPv6 same way as we IPv4 and calculate the checksum before entering pf_test6(). PR: 172648, 179392 Changes: _U stable/10/ stable/10/sys/netpfil/pf/pf_ioctl.c
Merged to stable/10.
I think this problem (or some variation of it) is still present on FreeBSD 10.2-RELEASE-p9. E.g., with the following rules: table <forward_subnets> {2001:16d8:ee03::cafe:d00d/128,!::1} rdr pass log on lo0 proto tcp to <forward_subnets> -> ::1 port 12300 pass out log route-to lo0 inet6 proto tcp to <forward_subnets> keep state and while running nc -l ::1 12300 trying nc -vz 2001:16d8:ee03::cafe:d00d 80 to see if any packets get to app listening on 12300 shows that nothing is received although tcpdump'ing pflog0 shows that the rules are applied: 00:00:00.000000 rule 0.sshuttle.0/0(match): pass out on vtnet0: 2a03:b0c0:1:d0::142:f001.12911 > 2001:16d8:ee03::cafe:d00d.80: Flags [S], seq 3860904544, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 629750 ecr 0], length 0 00:00:00.000074 rule 0..16777216/0(match): rdr in on lo0: 2a03:b0c0:1:d0::142:f001.12911 > ::1.12300: Flags [S], seq 3860904544, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 629750 ecr 0], length 0 tcpdump'ing lo0 repeatedly shows "cksum 0x8cf0 (incorrect -> 0x...)": 00:00:00.000000 AF IPv6 (28), length 84: (flowlabel 0x49699, hlim 64, next-header TCP (6) payload length: 40) 2a03:b0c0:1:d0::142:f001.53043 > 2001:16d8:ee03::cafe:d00d.80: Flags [S], cksum 0x8cf0 (incorrect -> 0x29da), seq 4029399333, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 1577100 ecr 0], length 0 00:00:03.008841 AF IPv6 (28), length 84: (flowlabel 0x49699, hlim 64, next-header TCP (6) payload length: 40) 2a03:b0c0:1:d0::142:f001.53043 > 2001:16d8:ee03::cafe:d00d.80: Flags [S], cksum 0x8cf0 (incorrect -> 0x1e19), seq 4029399333, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 1580109 ecr 0], length 0