When pf is enabled on ue0 interface, tcp connection to this interface is not possible. tcpdump shows incorrect tcp checksum dmseg: axe0: <vendor 0x0b95 product 0x7e2b, rev 2.00/0.01, addr 2> on usbus3 miibus1: <MII bus> on axe0 ukphy0: <Generic IEEE 802.3u media interface> PHY 16 on miibus1 ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow ue0: <USB Ethernet> on axe0 ue0: Ethernet address: 9c:eb:e8:06:c4:a7 pf.conf: pass log quick on lo0 pass log quick on ue0 tcpdump output: 16:42:03.742071 IP (tos 0x0, ttl 64, id 759, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.85.ssh > 192.168.0.77.57379: Flags [F.], cksum 0x8219 (incorrect -> 0x3432), seq 2656523414, ack 3256801674, win 1040, options [nop,nop,TS val 2997057078 ecr 124263796], length 0
The problem is not reproducible: With 9.2 on the same hardware Witn 10.1 but different USB ethernet card or non usb-ethernet card.
Any updates?
A commit references this bug: Author: kp Date: Wed Oct 14 16:21:42 UTC 2015 New revision: 289316 URL: https://svnweb.freebsd.org/changeset/base/289316 Log: pf: Fix TSO issues In certain configurations (mostly but not exclusively as a VM on Xen) pf produced packets with an invalid TCP checksum. The problem was that pf could only handle packets with a full checksum. The FreeBSD IP stack produces TCP packets with a pseudo-header checksum (only addresses, length and protocol). Certain network interfaces expect to see the pseudo-header checksum, so they end up producing packets with invalid checksums. To fix this stop calculating the full checksum and teach pf to only update TCP checksums if TSO is disabled or the change affects the pseudo-header checksum. PR: 154428, 193579, 198868 Reviewed by: sbruno MFC after: 1 week Relnotes: yes Sponsored by: RootBSD Differential Revision: https://reviews.freebsd.org/D3779 Changes: head/sys/net/pfvar.h head/sys/netpfil/pf/pf.c head/sys/netpfil/pf/pf_ioctl.c head/sys/netpfil/pf/pf_norm.c
A commit references this bug: Author: kp Date: Wed Oct 21 15:32:21 UTC 2015 New revision: 289703 URL: https://svnweb.freebsd.org/changeset/base/289703 Log: MFC r289316: pf: Fix TSO issues In certain configurations (mostly but not exclusively as a VM on Xen) pf produced packets with an invalid TCP checksum. The problem was that pf could only handle packets with a full checksum. The FreeBSD IP stack produces TCP packets with a pseudo-header checksum (only addresses, length and protocol). Certain network interfaces expect to see the pseudo-header checksum, so they end up producing packets with invalid checksums. To fix this stop calculating the full checksum and teach pf to only update TCP checksums if TSO is disabled or the change affects the pseudo-header checksum. PR: 154428, 193579, 198868 Relnotes: yes Sponsored by: RootBSD Changes: _U stable/10/ stable/10/sys/net/pfvar.h stable/10/sys/netpfil/pf/pf.c stable/10/sys/netpfil/pf/pf_ioctl.c stable/10/sys/netpfil/pf/pf_norm.c
A commit references this bug: Author: kp Date: Fri Dec 25 15:12:12 UTC 2015 New revision: 292731 URL: https://svnweb.freebsd.org/changeset/base/292731 Log: pf: Fix TSO issues In certain configurations (mostly but not exclusively as a VM on Xen) pf produced packets with an invalid TCP checksum. The problem was that pf could only handle packets with a full checksum. The FreeBSD IP stack produces TCP packets with a pseudo-header checksum (only addresses, length and protocol). Certain network interfaces expect to see the pseudo-header checksum, so they end up producing packets with invalid checksums. To fix this stop calculating the full checksum and teach pf to only update TCP checksums if TSO is disabled or the change affects the pseudo-header checksum. PR: 154428, 193579, 198868 Sponsored by: RootBSD Changes: stable/9/sys/contrib/pf/net/pf.c stable/9/sys/contrib/pf/net/pf_ioctl.c stable/9/sys/contrib/pf/net/pf_norm.c stable/9/sys/contrib/pf/net/pfvar.h
Assign to committer that's taking care of (resolving) this issue
Can anyone who's affected by this (or PR170081) list the device they're using? Unfortunately I do not appear to have any affected hardware, which makes debugging this a little tricky.
Created attachment 165123 [details] Test-patch Here's a pretty quick patch which simply disables handling partial checksum offloading in pf. I.e. we treat a partially offloaded checksum as if there's no checksum offloading at all. The downside is that we'll spend CPU time calculating something the interface itself already calculated, but it should fix this problem. As far as I can tell this would only affect a handful of older cards anyway, and as I don't have any affected hardware this is likely the best I can do.
(In reply to Kristof Provost from comment #8) Hi Kristof, I've just inserted two, recently purchased new, usb interface cards from J5Create into a 10.2Stable box. They are recognised as follows: # Card 1: This is a J5Create JUE125 USB2.0 Ethernet adapter (that I bought new a few weeks ago) ugen0.4: <vendor 0x0b95> at usbus0 axe0: <vendor 0x0b95 product 0x772b, rev 2.00/0.02, addr 3> on usbus0 miibus0: <MII bus> on axe0 ukphy0: <Generic IEEE 802.3u media interface> PHY 16 on miibus0 ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow ue0: <USB Ethernet> on axe0 ue0: Ethernet address: 00:05:1b:a4:9f:1c ue0: link state changed to DOWN ue0: link state changed to UP ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE> ether 00:05:1b:a4:9f:1c nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (100baseTX <full-duplex>) status: active # Card 2: J5Create JUE130 USB3.0 Gigabit Ethernet adapter (as above) ugen0.5: <ASIX Elec.> at usbus0 axge0: <NetworkInterface> on usbus0 miibus1: <MII bus> on axge0 rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 3 on miibus1 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow ue1: <USB Ethernet> on axge0 ue1: Ethernet address: 00:05:1b:a1:79:76 ue1: link state changed to DOWN ue1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE> ether 00:05:1b:a1:79:76 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (none) status: no carrier The following demonstrates the difference between em1: <Intel(R) PRO/1000 Network Connection 7.4.2> and ue0 that is axe0: <vendor 0x0b95 product 0x772b, rev 2.00/0.02, addr 3> The source box running pf and 10.2Stable (FreeBSD hathor 10.2-STABLE FreeBSD 10.2-STABLE #0 r293123M: Mon Jan 4 17:36:12 AEDT 2016 root@hathor:/usr/obj/prod/100201/D/K8/pd3/src/sys/hqdev-amd64-smp-vga amd64 1002505 1002505) Source IP: 10.0.7.91 (and jail 10.0.7.92) Destination: 10.0.7.6 (Running FreeBSD92 and ipfw) Using a working interface (em1) to a box one hop away and using incorrect ssh key, I've listed the first six lines from tcpdump for a negotation between A) Using em1 interface between a base system (10.0.7.91) and a remote node B) Using em1 interface from within a jail (10.0.7.92) C) Using ue0 between base r10.0.7.91 D) Using ue0 between jail 10.0.7.92 A) # tcpdump -s1518 -vni em1 host 10.0.7.6 and port 22 tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size 1518 bytes 09:19:25.943098 IP (tos 0x0, ttl 48, id 26320, offset 0, flags [DF], proto TCP (6), length 60) 10.0.7.91.52418 > 10.0.7.6.22: Flags [S], cksum 0x228f (incorrect -> 0xef75), seq 2076369872, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 90899540 ecr 0], length 0 09:19:25.943317 IP (tos 0x0, ttl 64, id 45550, offset 0, flags [DF], proto TCP (6), length 60) 10.0.7.6.22 > 10.0.7.91.52418: Flags [S.], cksum 0xef96 (correct), seq 1751036782, ack 2076369873, win 65535, options [mss 1460,nop,wscale 5,sackOK,TS val 324653224 ecr 90899540], length 0 09:19:25.943354 IP (tos 0x0, ttl 48, id 48994, offset 0, flags [DF], proto TCP (6), length 52) 10.0.7.91.52418 > 10.0.7.6.22: Flags [.], cksum 0x2287 (incorrect -> 0x1653), ack 1, win 2058, options [nop,nop,TS val 90899544 ecr 324653224], length 0 09:19:25.943574 IP (tos 0x0, ttl 64, id 45551, offset 0, flags [DF], proto TCP (6), length 52) 10.0.7.6.22 > 10.0.7.91.52418: Flags [.], cksum 0x0e48 (correct), ack 1, win 4117, options [nop,nop,TS val 324653224 ecr 90899544], length 0 09:19:25.943862 IP (tos 0x0, ttl 48, id 25543, offset 0, flags [DF], proto TCP (6), length 81) 10.0.7.91.52418 > 10.0.7.6.22: Flags [P.], cksum 0x22a4 (incorrect -> 0x1540), seq 1:30, ack 1, win 2058, options [nop,nop,TS val 90899544 ecr 324653224], length 29 09:19:26.043316 IP (tos 0x0, ttl 64, id 45552, offset 0, flags [DF], proto TCP (6), length 52) 10.0.7.6.22 > 10.0.7.91.52418: Flags [.], cksum 0x0dc8 (correct), ack 30, win 4116, options [nop,nop,TS val 324653324 ecr 90899544], length 0 And from a jail using the same interface to a jail B) # tcpdump -s1518 -vni em1 host 10.0.7.6 and port 22 tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size 1518 bytes 09:24:12.271961 IP (tos 0x0, ttl 48, id 24663, offset 0, flags [DF], proto TCP (6), length 60) 10.0.7.92.11369 > 10.0.7.6.22: Flags [S], cksum 0x2290 (incorrect -> 0x81ba), seq 3957334854, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 91185872 ecr 0], length 0 09:24:12.272467 IP (tos 0x0, ttl 64, id 45639, offset 0, flags [DF], proto TCP (6), length 60) 10.0.7.6.22 > 10.0.7.92.11369: Flags [S.], cksum 0x1a60 (correct), seq 2844259951, ack 3957334855, win 65535, options [mss 1460,nop,wscale 5,sackOK,TS val 3406822466 ecr 91185872], length 0 09:24:12.272488 IP (tos 0x0, ttl 48, id 39982, offset 0, flags [DF], proto TCP (6), length 52) 10.0.7.92.11369 > 10.0.7.6.22: Flags [.], cksum 0x2288 (incorrect -> 0x4120), ack 1, win 2058, options [nop,nop,TS val 91185872 ecr 3406822466], length 0 09:24:12.272710 IP (tos 0x0, ttl 64, id 45640, offset 0, flags [DF], proto TCP (6), length 52) 10.0.7.6.22 > 10.0.7.92.11369: Flags [.], cksum 0x3915 (correct), ack 1, win 4117, options [nop,nop,TS val 3406822466 ecr 91185872], length 0 09:24:12.273335 IP (tos 0x0, ttl 48, id 54149, offset 0, flags [DF], proto TCP (6), length 81) 10.0.7.92.11369 > 10.0.7.6.22: Flags [P.], cksum 0x22a5 (incorrect -> 0x400d), seq 1:30, ack 1, win 2058, options [nop,nop,TS val 91185872 ecr 3406822466], length 29 09:24:12.373101 IP (tos 0x0, ttl 64, id 45641, offset 0, flags [DF], proto TCP (6), length 52) 10.0.7.6.22 > 10.0.7.92.11369: Flags [.], cksum 0x3894 (correct), ack 30, win 4116, options [nop,nop,TS val 3406822567 ecr 91185872], length 0 Disconnecting the cable from that interface and inserting into ue0, changing the external interface in pf and restarting, ue0 has this C) # tcpdump -s1518 -vni ue0 host 10.0.7.6 and port 22 tcpdump: listening on ue0, link-type EN10MB (Ethernet), capture size 1518 bytes 09:33:13.585464 IP (tos 0x0, ttl 48, id 641, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->25db)!) 10.0.7.91.29122 > 10.0.7.6.22: Flags [S], cksum 0x228f (incorrect -> 0xdaf0), seq 1199433362, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 91727184 ecr 0], length 0 09:33:13.587153 IP (tos 0x0, ttl 64, id 45967, offset 0, flags [DF], proto TCP (6), length 60) 10.0.7.6.22 > 10.0.7.91.29122: Flags [S.], cksum 0xe549 (correct), seq 3082007530, ack 1199433363, win 65535, options [mss 1460,nop,wscale 5,sackOK,TS val 4112225500 ecr 91727184], length 0 09:33:13.587175 IP (tos 0x0, ttl 48, id 5810, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->11b2)!) 10.0.7.91.29122 > 10.0.7.6.22: Flags [.], cksum 0x2287 (incorrect -> 0x0c06), ack 1, win 2058, options [nop,nop,TS val 91727188 ecr 4112225500], length 0 09:33:13.587504 IP (tos 0x0, ttl 48, id 39472, offset 0, flags [DF], proto TCP (6), length 81, bad cksum 0 (->8e16)!) 10.0.7.91.29122 > 10.0.7.6.22: Flags [P.], cksum 0x22a4 (incorrect -> 0x0af3), seq 1:30, ack 1, win 2058, options [nop,nop,TS val 91727188 ecr 4112225500], length 29 09:33:13.588820 IP (tos 0x0, ttl 64, id 45968, offset 0, flags [DF], proto TCP (6), length 52) 10.0.7.6.22 > 10.0.7.91.29122: Flags [.], cksum 0x03f9 (correct), ack 1, win 4117, options [nop,nop,TS val 4112225502 ecr 91727188], length 0 09:33:13.688305 IP (tos 0x0, ttl 64, id 45969, offset 0, flags [DF], proto TCP (6), length 52) 10.0.7.6.22 > 10.0.7.91.29122: Flags [.], cksum 0x0379 (correct), ack 30, win 4116, options [nop,nop,TS val 4112225602 ecr 91727188], length 0 And from a jail D) # tcpdump -s1518 -vni ue0 host 10.0.7.6 and port 22 tcpdump: listening on ue0, link-type EN10MB (Ethernet), capture size 1518 bytes 09:34:25.701515 IP (tos 0x0, ttl 48, id 7850, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->9b1)!) 10.0.7.92.58725 > 10.0.7.6.22: Flags [S], cksum 0x2290 (incorrect -> 0x9b31), seq 2136629531, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 91799300 ecr 0], length 0 09:34:25.703204 IP (tos 0x0, ttl 64, id 46014, offset 0, flags [DF], proto TCP (6), length 60) 10.0.7.6.22 > 10.0.7.92.58725: Flags [S.], cksum 0x47ef (correct), seq 2091274476, ack 2136629532, win 65535, options [mss 1460,nop,wscale 5,sackOK,TS val 269251987 ecr 91799300], length 0 09:34:25.703224 IP (tos 0x0, ttl 48, id 4000, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->18c3)!) 10.0.7.92.58725 > 10.0.7.6.22: Flags [.], cksum 0x2288 (incorrect -> 0x6eab), ack 1, win 2058, options [nop,nop,TS val 91799304 ecr 269251987], length 0 09:34:25.703539 IP (tos 0x0, ttl 48, id 41112, offset 0, flags [DF], proto TCP (6), length 81, bad cksum 0 (->87ad)!) 10.0.7.92.58725 > 10.0.7.6.22: Flags [P.], cksum 0x22a5 (incorrect -> 0x6d98), seq 1:30, ack 1, win 2058, options [nop,nop,TS val 91799304 ecr 269251987], length 29 09:34:25.704871 IP (tos 0x0, ttl 64, id 46015, offset 0, flags [DF], proto TCP (6), length 52) 10.0.7.6.22 > 10.0.7.92.58725: Flags [.], cksum 0x669e (correct), ack 1, win 4117, options [nop,nop,TS val 269251989 ecr 91799304], length 0 09:34:25.804852 IP (tos 0x0, ttl 64, id 46016, offset 0, flags [DF], proto TCP (6), length 52) 10.0.7.6.22 > 10.0.7.92.58725: Flags [.], cksum 0x661e (correct), ack 30, win 4116, options [nop,nop,TS val 269252089 ecr 91799304], length 0 These dumps are from a failed ssh connection between two nodes so we can see the handshaking experience. I've compressed the content (attached) and provide an abridged content for the PR. I hope that this provides some clarification that this doesn't apply to old interfaces, nor complex vm setups ;) Happy to assist further, as I was about to change our firewall from ipfw to pf but we do use ue (devices)... Regards, Dewayne.
(In reply to dewayne from comment #9) Thanks for the report! It's useful to know what hardware is affected. I haven't yet found a place that'll sell one to me though. Can you give the patch a try? It'd be useful to confirm that this really does fix the problem. If nothing else, it's a good temporary fix, because things will at least start working. We can worry about actually using the partial checksum offloading later, when I get my hands on hardware.
(In reply to Kristof Provost from comment #10) My apologies Kristof, I'm unable to add any value here. I'm using product ID=0x772b (ASIX Electronics Corporation, an AX88772B 10/100) while Dmitry is using 0x7e2b (we're using the same vendor 0x0b95), and both devices are regarded as AX88772B. Unfortunately I was responding to the interface ue0 as being the problem. Regardless, we aren't comparing apples to oranges - as I am getting successful connections albeit with bad cksums; which I continue to get after disabling pf (via pfctl -d) References: Note from /usr/src/sys/dev/usb/net/if_axereg.h * This driver does not enable partial checksum feature which will * compute 16bit checksum from 14th byte to the end of the frame. If * this feature is enabled, computed checksum value is embedded into * RX header which in turn means it uses different RX header format. Linux drivers are here: http://www.asix.com.tw/download.php?sub=driverdetail&PItemID=86 Documentation: http://www.asix.com.tw/download.php?sub=briefdetail&PItemID=105
(In reply to dewayne from comment #11) Oh, right, I should have caught that too. It's pretty normal to see incorrect checkums on outbound packets. The checksum is done by the hardware, so at the point that tcpdump captures them it's still not calculated. The bug here (at least to the best of my understanding) is about inbound packets
I got my hands on a couple of AX88772 based USB ethernet devices: > ugen1.2: <vendor 0x0fe6> at usbus1 > udav0: <vendor 0x0fe6 USB 2.0 10100M Ethernet Adaptor, class 0/0, rev 1.10/1.01, add 1> on usbus1 > ue0: <USB Ethernet> on udav0 > ue0: Ethernet address: 00:e0:4c:53:44:58 and: > ugen1.2: <vendor 0x0b95> at usbus1 > axe0: <vendor 0x0b95 product 0x772b, rev 2.00/0.01, addr 1> on usbus1 > miibus0: <MII bus> on axe0 > ukphy0: <Generic IEEE 802.3u media interface> PHY 16 on miibus0 > ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow > ue0: <USB Ethernet> on axe0 > ue0: Ethernet address: 00:00:00:00:14:7e Unfortunately I'm still unable to reproduce this problem. I'm afraid I don't know what else to do at the moment.
Agreed upon by kp@.