If checksum offloading is enabled on fxp0 pf/nat does not work properly.
Therefore the network in my jails does not work.
With tcpdump I see SYN packets going out and SYN+ACK packets coming in on fxp0, but connections time out (no ACKs). pflog doesn't show any blocked packets.
I can reproduce this issue on different computers with fxp.
With ipfw/natd it's working.
pf/nat and different nic (e.g: em0) are OK too.
After disabling checksum offloading (ifconfig fxp0 -rxcsum) pf is working. (Thanks to yongari@ for the hint).
How-To-Repeat: # ifconfig lo1 create
# ifconfig lo1 10.0.0.10 netmask 0xffffff00
# pf -e -f /root/pf.conf 
# nc -s 10.0.0.10 18.104.22.168 25
 My pf.conf:
nat log on fxp0 inet from 10.0.0.0/8 to any -> fxp0
pass in all
pass out all
Over to maintainer(s).
Getting the exact same problem. No network traffic to my jails.
# uname -a
FreeBSD nexus 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4
09:23:10 UTC 2012
Here's part of my pf.cf file
ext_ip="192.168.1.2" # behind a firewall
scrub in all
nat on $ext_if from !($ext_if) -> $ext_ip
I removed rxcsum from fxp0 using:
# ifconfig fxp0 -rxcsum
Now everything works great and jails get network traffic.
Let me know if you need more detailed information.
I had the same problem.
jails and nat (pf) on fxp0 does not work. After disabling rx checksum ( ifconfig fxp0 -rxcsum ) it's working perfectly.
The fix for this issue went in in r289703 (for stable/10). See also 154428, 193579, 198868.
The commit message, for reference:
pf: Fix TSO issues
In certain configurations (mostly but not exclusively as a VM on Xen) pf
produced packets with an invalid TCP checksum.
The problem was that pf could only handle packets with a full checksum. The
FreeBSD IP stack produces TCP packets with a pseudo-header checksum (only
addresses, length and protocol).
Certain network interfaces expect to see the pseudo-header checksum, so they
end up producing packets with invalid checksums.
To fix this stop calculating the full checksum and teach pf to only update TCP
checksums if TSO is disabled or the change affects the pseudo-header checksum.
I am now running FreeBSD 10.2-STABLE (r289837) but I still have this issue.
(In reply to h.skuhra from comment #5)
Can you confirm the problem still goes away after disabling checksum offloading?
If so, can you grab a quick tcpdump as well?
Yes, network starts working immediately after running 'ifconfig fxp0 -rxcsum'. I'll send you a tcpdump in a moment.
It seems that there's still an issue with (inbound) checksums with the fxp interface.
While running 'nc <ip> 25' in the jail I see the number of 'discarded
for bad checksums' in the output of 'netstat -s -p tcp' grow.
While running 'drill -x <ip>' I see the number of 'with bad checksum'
in the output of 'netstat -s -p udp' grow.
No more bad checksum when I disable rxcsum on fxp0.
No problem if I use a) ipfw/natd or b) put alias ip address
(192.168.77.x) for jail on fxp0 and disable pf and nat.
Due to H/W limitation, fxp(4), gem(4), hme(4) and sk(4) controllers
have no pseudo header checksum capability in RX. Those drivers
set CSUM_DATA_VALID and pseudo header calculation is up to upper
I looked over pf's checksum validation code and noticed it changed
a lot since I lastly checked it. It seems pf_check_proto_cksum()
is not called for UDP/ICMP at all. For TCP, pf_check_proto_cksum()
is called for specific cases. I recall one of the first thing pf
did in pf_test()/pf6_test() was pf_check_proto_cksum() to verify
checksum validation but it seems it was changed.
I vaguely guess checksum fixup code in pf does not handle partial
RX checksummed packet(i.e. no pseudo header checksum) and
CSUM_DATA_VALID bit is ignored in the checksum fixup.
(In reply to Pyun YongHyeon from comment #9)
The pseudo-header checksum thing is what was changed in r289703, but that only applied to outbound packets (we check for CSUM_DELAY_DATA in csum_flags). Incoming packets are assumed to have full checksums.
The fxp driver does appear to set csum_flags, but it never sets DELAY_DATA. As far as I can see it also doesn't touch the checksum in the packets, so I don't really see how that would break things.
Given that this issue had been there for a long time, I don't think
r289703 is triggering the issue.
As I said in earlier, some H/Ws can't compute full checksum in RX
path so drivers set a partial checksum and pass it to upper stack.
It's responsibility of upper stack to compute full checksum with
the partial checksum(See tcp_input() or udp_input()).
DELAY_DATA is not used in RX path at all. CSUM_DATA_VALID is used
to let upper stack know the existence of partial checksum of TCP/UDP
packet in csum_data field of mbuf and upper stack should compute
pseudo header checksum with the supplied partial checksum. If the
H/W can compute full checksum, driver will set both CSUM_DATA_VALID
and CSUM_PSEUDO_HDR for TCP/UDP packets.
Can anyone affected by this test the patch in bug #198868 please?
Unfortunately I don't have the hardware to test this myself.
Briefly, the patch works around this problem by treating partial rx checksums as if there is no checksum. That *should* fix the problem here.
With your patch i still have the problem (FreeBSD 10.2-STABLE r293422).
NIC is Intel 82801DB (ICH4) Pro/100 Ethernet (fxp0)
(In reply to Herbert J. Skuhra from comment #13)
Just to be clear: you tested with https://bugs.freebsd.org/bugzilla/attachment.cgi?id=165123 (Test-patch) right? Not just the commits referenced in that bug?
If so, I'm completely lost about what could be causing this.
Yes, I took the patch from Bug 198868 (pf_fxp_csum.patch) and I applied it to stable/10 (r293422). Rebuilt/installed kernel/world (NO_CLEAN) and rebooted.
I have to trash this PC in the near future anyway. Thanks for your efforts. Let me know if I should test other patches.