Bug 170081 - [fxp] pf/nat/jails not working if checksum offloading is enabled on fxp0
Summary: [fxp] pf/nat/jails not working if checksum offloading is enabled on fxp0
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 9.1-PRERELEASE
Hardware: Any Any
: Normal Affects Only Me
Assignee: Kristof Provost
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-07-23 09:40 UTC by Herbert J. Skuhra
Modified: 2019-06-04 11:19 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Herbert J. Skuhra 2012-07-23 09:40:11 UTC
If checksum offloading is enabled on fxp0 pf/nat does not work properly.
Therefore the network in my jails does not work.

With tcpdump I see SYN packets going out and SYN+ACK packets coming in on fxp0, but connections time out (no ACKs). pflog doesn't show any blocked packets.

I can reproduce this issue on different computers with fxp.
With ipfw/natd it's working.

pf/nat and different nic (e.g: em0) are OK too.

After disabling checksum offloading (ifconfig fxp0 -rxcsum) pf is working. (Thanks to yongari@ for the hint).

How-To-Repeat: # ifconfig lo1 create
# ifconfig lo1 10.0.0.10 netmask 0xffffff00
# pf -e -f /root/pf.conf [1]
# nc -s 10.0.0.10 173.194.70.108 25
Connection fails.

[1] My pf.conf:
nat log on fxp0 inet from 10.0.0.0/8 to any -> fxp0
pass in all
pass out all
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2012-07-23 11:10:30 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-net

Over to maintainer(s).
Comment 2 je.broman 2013-04-12 09:30:50 UTC
Hi!

Getting the exact same problem. No network traffic to my jails.

# uname -a

FreeBSD nexus 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec  4
09:23:10 UTC 2012
root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
amd64

Here's part of my pf.cf file

-- snip--

ext_if="fxp0"
int_if="bge0"
jail_if="lo1"
ext_ip="192.168.1.2" # behind a firewall
int_ip="192.168.0.1"
jail_net="10.0.0.0/24"


scrub in all

nat on $ext_if from !($ext_if) -> $ext_ip

--snip --

I removed rxcsum from fxp0 using:

# ifconfig fxp0 -rxcsum

Now everything works great and jails get network traffic.

Let me know if you need more detailed information.

Thanks
Johan
Comment 3 alucard.ui 2015-10-24 10:20:34 UTC
I had the same problem.
uname -rs
FreeBSD 10.2-RELEASE

jails and nat (pf) on fxp0 does not work. After disabling rx checksum ( ifconfig fxp0 -rxcsum ) it's working perfectly.
Comment 4 Kristof Provost freebsd_committer 2015-10-24 17:46:55 UTC
The fix for this issue went in in r289703 (for stable/10). See also 154428, 193579, 198868.

The commit message, for reference:
pf: Fix TSO issues

In certain configurations (mostly but not exclusively as a VM on Xen) pf
produced packets with an invalid TCP checksum.

The problem was that pf could only handle packets with a full checksum. The
FreeBSD IP stack produces TCP packets with a pseudo-header checksum (only
addresses, length and protocol).
Certain network interfaces expect to see the pseudo-header checksum, so they
end up producing packets with invalid checksums.

To fix this stop calculating the full checksum and teach pf to only update TCP
checksums if TSO is disabled or the change affects the pseudo-header checksum.
Comment 5 Herbert J. Skuhra 2015-10-24 21:33:37 UTC
I am now running FreeBSD 10.2-STABLE (r289837) but I still have this issue.
Comment 6 Kristof Provost freebsd_committer 2015-10-24 21:36:12 UTC
(In reply to h.skuhra from comment #5)
Can you confirm the problem still goes away after disabling checksum offloading?

If so, can you grab a quick tcpdump as well?
Comment 7 Herbert J. Skuhra 2015-10-24 22:10:50 UTC
Yes, network starts working immediately after running 'ifconfig fxp0 -rxcsum'. I'll send you a tcpdump in a moment.
Comment 8 Kristof Provost freebsd_committer 2015-10-25 20:58:49 UTC
It seems that there's still an issue with (inbound) checksums with the fxp interface.


From Herbert:
While running 'nc <ip> 25' in the jail I see the number of 'discarded
for bad checksums' in the output of 'netstat -s -p tcp' grow.

While running 'drill -x <ip>' I see the number of 'with bad checksum'
in the output of 'netstat -s -p udp' grow.

No more bad checksum when I disable rxcsum on fxp0.

No problem if I use a) ipfw/natd or b) put alias ip address
(192.168.77.x) for jail on fxp0 and disable pf and nat.
Comment 9 Pyun YongHyeon freebsd_committer 2015-10-26 04:35:39 UTC
Due to H/W limitation, fxp(4), gem(4), hme(4) and sk(4) controllers
have no pseudo header checksum capability in RX.  Those drivers
set CSUM_DATA_VALID and pseudo header calculation is up to upper
layer.
I looked over pf's checksum validation code and noticed it changed
a lot since I lastly checked it.  It seems pf_check_proto_cksum()
is not called for UDP/ICMP at all. For TCP, pf_check_proto_cksum()
is called for specific cases.  I recall one of the first thing pf
did in pf_test()/pf6_test() was pf_check_proto_cksum() to verify
checksum validation but it seems it was changed.
I vaguely guess checksum fixup code in pf does not handle partial
RX checksummed packet(i.e. no pseudo header checksum) and
CSUM_DATA_VALID bit is ignored in the checksum fixup.
Comment 10 Kristof Provost freebsd_committer 2015-10-26 22:38:29 UTC
(In reply to Pyun YongHyeon from comment #9)
Interesting...

The pseudo-header checksum thing is what was changed in r289703, but that only applied to outbound packets (we check for CSUM_DELAY_DATA in csum_flags). Incoming packets are assumed to have full checksums.

The fxp driver does appear to set csum_flags, but it never sets DELAY_DATA. As far as I can see it also doesn't touch the checksum in the packets, so I don't really see how that would break things.
Comment 11 Pyun YongHyeon freebsd_committer 2015-10-27 01:03:03 UTC
Given that this issue had been there for a long time, I don't think
r289703 is triggering the issue.

As I said in earlier, some H/Ws can't compute full checksum in RX
path so drivers set a partial checksum and pass it to upper stack.
It's responsibility of upper stack to compute full checksum with
the partial checksum(See tcp_input() or udp_input()).

DELAY_DATA is not used in RX path at all.  CSUM_DATA_VALID is used 
to let upper stack know the existence of partial checksum of TCP/UDP
packet in csum_data field of mbuf and upper stack should compute
pseudo header checksum with the supplied partial checksum.  If the
H/W can compute full checksum, driver will set both CSUM_DATA_VALID
and CSUM_PSEUDO_HDR for TCP/UDP packets.
Comment 12 Kristof Provost freebsd_committer 2016-01-08 20:07:36 UTC
Can anyone affected by this test the patch in bug #198868 please?
Unfortunately I don't have the hardware to test this myself.

Briefly, the patch works around this problem by treating partial rx checksums as if there is no checksum. That *should* fix the problem here.
Comment 13 Herbert J. Skuhra 2016-01-08 23:02:58 UTC
With your patch i still have the problem (FreeBSD 10.2-STABLE r293422).
NIC is Intel 82801DB (ICH4) Pro/100 Ethernet (fxp0)
Comment 14 Kristof Provost freebsd_committer 2016-01-08 23:12:11 UTC
(In reply to Herbert J. Skuhra from comment #13)
Just to be clear: you tested with https://bugs.freebsd.org/bugzilla/attachment.cgi?id=165123 (Test-patch) right? Not just the commits referenced in that bug?

If so, I'm completely lost about what could be causing this.
Comment 15 Herbert J. Skuhra 2016-01-09 10:14:01 UTC
Yes, I took the patch from Bug 198868 (pf_fxp_csum.patch) and I applied it to stable/10 (r293422). Rebuilt/installed kernel/world (NO_CLEAN) and rebooted.
I have to trash this PC in the near future anyway. Thanks for your efforts. Let me know if I should test other patches.