If checksum offloading is enabled on fxp0 pf/nat does not work properly. Therefore the network in my jails does not work. With tcpdump I see SYN packets going out and SYN+ACK packets coming in on fxp0, but connections time out (no ACKs). pflog doesn't show any blocked packets. I can reproduce this issue on different computers with fxp. With ipfw/natd it's working. pf/nat and different nic (e.g: em0) are OK too. After disabling checksum offloading (ifconfig fxp0 -rxcsum) pf is working. (Thanks to yongari@ for the hint). How-To-Repeat: # ifconfig lo1 create # ifconfig lo1 10.0.0.10 netmask 0xffffff00 # pf -e -f /root/pf.conf [1] # nc -s 10.0.0.10 173.194.70.108 25 Connection fails. [1] My pf.conf: nat log on fxp0 inet from 10.0.0.0/8 to any -> fxp0 pass in all pass out all
Responsible Changed From-To: freebsd-bugs->freebsd-net Over to maintainer(s).
Hi! Getting the exact same problem. No network traffic to my jails. # uname -a FreeBSD nexus 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 09:23:10 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 Here's part of my pf.cf file -- snip-- ext_if="fxp0" int_if="bge0" jail_if="lo1" ext_ip="192.168.1.2" # behind a firewall int_ip="192.168.0.1" jail_net="10.0.0.0/24" scrub in all nat on $ext_if from !($ext_if) -> $ext_ip --snip -- I removed rxcsum from fxp0 using: # ifconfig fxp0 -rxcsum Now everything works great and jails get network traffic. Let me know if you need more detailed information. Thanks Johan
I had the same problem. uname -rs FreeBSD 10.2-RELEASE jails and nat (pf) on fxp0 does not work. After disabling rx checksum ( ifconfig fxp0 -rxcsum ) it's working perfectly.
The fix for this issue went in in r289703 (for stable/10). See also 154428, 193579, 198868. The commit message, for reference: pf: Fix TSO issues In certain configurations (mostly but not exclusively as a VM on Xen) pf produced packets with an invalid TCP checksum. The problem was that pf could only handle packets with a full checksum. The FreeBSD IP stack produces TCP packets with a pseudo-header checksum (only addresses, length and protocol). Certain network interfaces expect to see the pseudo-header checksum, so they end up producing packets with invalid checksums. To fix this stop calculating the full checksum and teach pf to only update TCP checksums if TSO is disabled or the change affects the pseudo-header checksum.
I am now running FreeBSD 10.2-STABLE (r289837) but I still have this issue.
(In reply to h.skuhra from comment #5) Can you confirm the problem still goes away after disabling checksum offloading? If so, can you grab a quick tcpdump as well?
Yes, network starts working immediately after running 'ifconfig fxp0 -rxcsum'. I'll send you a tcpdump in a moment.
It seems that there's still an issue with (inbound) checksums with the fxp interface. From Herbert: While running 'nc <ip> 25' in the jail I see the number of 'discarded for bad checksums' in the output of 'netstat -s -p tcp' grow. While running 'drill -x <ip>' I see the number of 'with bad checksum' in the output of 'netstat -s -p udp' grow. No more bad checksum when I disable rxcsum on fxp0. No problem if I use a) ipfw/natd or b) put alias ip address (192.168.77.x) for jail on fxp0 and disable pf and nat.
Due to H/W limitation, fxp(4), gem(4), hme(4) and sk(4) controllers have no pseudo header checksum capability in RX. Those drivers set CSUM_DATA_VALID and pseudo header calculation is up to upper layer. I looked over pf's checksum validation code and noticed it changed a lot since I lastly checked it. It seems pf_check_proto_cksum() is not called for UDP/ICMP at all. For TCP, pf_check_proto_cksum() is called for specific cases. I recall one of the first thing pf did in pf_test()/pf6_test() was pf_check_proto_cksum() to verify checksum validation but it seems it was changed. I vaguely guess checksum fixup code in pf does not handle partial RX checksummed packet(i.e. no pseudo header checksum) and CSUM_DATA_VALID bit is ignored in the checksum fixup.
(In reply to Pyun YongHyeon from comment #9) Interesting... The pseudo-header checksum thing is what was changed in r289703, but that only applied to outbound packets (we check for CSUM_DELAY_DATA in csum_flags). Incoming packets are assumed to have full checksums. The fxp driver does appear to set csum_flags, but it never sets DELAY_DATA. As far as I can see it also doesn't touch the checksum in the packets, so I don't really see how that would break things.
Given that this issue had been there for a long time, I don't think r289703 is triggering the issue. As I said in earlier, some H/Ws can't compute full checksum in RX path so drivers set a partial checksum and pass it to upper stack. It's responsibility of upper stack to compute full checksum with the partial checksum(See tcp_input() or udp_input()). DELAY_DATA is not used in RX path at all. CSUM_DATA_VALID is used to let upper stack know the existence of partial checksum of TCP/UDP packet in csum_data field of mbuf and upper stack should compute pseudo header checksum with the supplied partial checksum. If the H/W can compute full checksum, driver will set both CSUM_DATA_VALID and CSUM_PSEUDO_HDR for TCP/UDP packets.
Can anyone affected by this test the patch in bug #198868 please? Unfortunately I don't have the hardware to test this myself. Briefly, the patch works around this problem by treating partial rx checksums as if there is no checksum. That *should* fix the problem here.
With your patch i still have the problem (FreeBSD 10.2-STABLE r293422). NIC is Intel 82801DB (ICH4) Pro/100 Ethernet (fxp0)
(In reply to Herbert J. Skuhra from comment #13) Just to be clear: you tested with https://bugs.freebsd.org/bugzilla/attachment.cgi?id=165123 (Test-patch) right? Not just the commits referenced in that bug? If so, I'm completely lost about what could be causing this.
Yes, I took the patch from Bug 198868 (pf_fxp_csum.patch) and I applied it to stable/10 (r293422). Rebuilt/installed kernel/world (NO_CLEAN) and rebooted. I have to trash this PC in the near future anyway. Thanks for your efforts. Let me know if I should test other patches.