This issue appeared for me when updating my RockPro64 from FreeBSD 14.3-RELEASE to 15.0-RELEASE. The initial symptoms appeared to be a lack of access to the external network from VNET jails, which on further inspection was limited to outgoing TCP traffic from the jails. Running tcpdump revealed that the TCP checksums of packets originating from VNET jails was incorrect, here is an example of trying to connect to the jail with SSH: ``` root@central-rock:~ # tcpdump -vvv -i epair111a tcpdump: listening on epair111a, link-type EN10MB (Ethernet), snapshot length 262144 bytes 17:36:18.943373 IP (tos 0x10, ttl 64, id 3737, offset 0, flags [DF], proto TCP (6), length 60) 192.168.100.101.34048 > irc.central-rock.home.ssh: Flags [S], cksum 0x5f54 (correct), seq 644269107, win 64240, options [mss 1460,sackOK,TS val 2246290942 ecr 0,nop,wscale 10], length 0 17:36:18.943555 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) irc.central-rock.home.ssh > 192.168.100.101.34048: Flags [S.], cksum 0x4a54 (incorrect -> 0x735d), seq 4098743306, ack 644269108, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 3678485310 ecr 2246290942], length 0 ``` ifconfig on the host: ``` root@central-rock:~ # ifconfig -a dwc0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE> ether 82:16:d0:7b:f5:0b media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=10<VLAN_HWTAGGING> ether 58:9c:fc:10:77:67 inet 192.168.100.100 netmask 0xffffff00 broadcast 192.168.100.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 bridge flags=0<> member: epair111a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> port 9 priority 128 path cost 2000 vlan protocol 802.1q member: epair110a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> port 5 priority 128 path cost 2000 vlan protocol 802.1q member: dwc0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> port 1 priority 128 path cost 55 vlan protocol 802.1q groups: bridge nd6 options=9<PERFORMNUD,IFDISABLED> pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152 options=0 groups: pflog epair110a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: jail:www options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6> ether 58:9c:fc:10:24:0b groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> epair111a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: jail:irc options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6> ether 58:9c:fc:10:bf:1d groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> ``` ifconfig in the jail: ``` root@central-rock:~ # jexec -u root irc ifconfig -a lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> pflog0: flags=0 metric 0 mtu 33152 options=0 groups: pflog epair111b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6> ether 58:9c:fc:10:98:82 inet 192.168.100.111 netmask 0xffffff00 broadcast 192.168.100.255 groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> ``` The bad checksum likely isn't an issue in in of itself, as it's documented in the epair manpage: ``` The epair has RXCSUM and RXCSUM6 enabled because it may receive a packet where the checksum has already been validated by a physical interface. The epair supports TXCSUM and TXCSUM6 for TCP and UDP, but only by forwarding the order to compute the checksum. Thus, when using an epair interface, a TCP or UDP sender can offload checksum computation to a physical interface. Note that, in case the packet does not leave the host, the checksum is unnecessary and will be ignored if offloaded. Such packets contain an incorrect checksum, since it is not computed yet. ``` And indeed, disabling TXCSUM by using the following prestart options in the jail works around the issue (with a presumed performance penalty): ``` exec.prestart = "ifconfig ${epair} create up"; exec.prestart += "ifconfig ${epair}a up descr jail:${name} -txcsum"; exec.prestart += "ifconfig ${bridge} addm ${epair}a up"; ``` Full jail configuration without the above workaround: ``` irc { # STARTUP/LOGGING exec.consolelog = "/var/log/jail_console_${name}.log"; # PERMISSIONS allow.raw_sockets; exec.clean; mount.devfs; devfs_ruleset = 5; # PATH/HOSTNAME path = "/usr/local/jails/containers/${name}"; host.hostname = "${name}.central-rock.home"; # VNET/VIMAGE vnet; vnet.interface = "${epair}b"; # NETWORKS/INTERFACES $id = "111"; $ip = "192.168.100.${id}/24"; $gateway = "192.168.100.1"; $bridge = "bridge0"; $epair = "epair${id}"; # ADD TO bridge INTERFACE exec.prestart = "ifconfig ${epair} create up"; exec.prestart += "ifconfig ${epair}a up descr jail:${name}"; exec.prestart += "ifconfig ${bridge} addm ${epair}a up"; exec.start = "ifconfig ${epair}b ${ip} up"; exec.start += "route add default ${gateway}"; exec.start += "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.poststop = "ifconfig ${bridge} deletem ${epair}a"; exec.poststop += "ifconfig ${epair}a destroy"; } ``` And the corresponding network rules from /etc/rc.conf: ``` hostname="central-rock.home" defaultrouter="192.168.100.1" cloned_interfaces="bridge0" ifconfig_bridge0="inet 192.168.100.100/24 addm dwc0 up" ifconfig_dwc0="up" ``` It's quite likely that this issue is related to bug #235607, the symptoms appear similar, but in the case of this bug, it doesn't appear to be caused by the firewall, as I'm not using it for NAT, I'm using a bridge / virtual switch for level 2 switching. I've tried disabling pf (my firewall) both in the host and guest machines, and it didn't change the behavior. I've tried reproducing the issue in a virtual machine, both in qemu-system-x86_64 and qemu-system-aarch64, and I was unable to reproduce it, network traffic for VNET jails with TCP checksum offloading worked correctly. The checksum of the packets coming from the epair was initially bad, and it was presumably corrected by the NIC. Here's an example of what tcpdump shows when connecting to FreeBSD.org from the jail in a QEMU VM: ``` tcpdump: listening on epair16a, link-type EN10MB (Ethernet), snapshot length 262144 bytes 19:13:59.056947 IP (tos 0x0, ttl 64, id 63837, offset 0, flags [none], proto UDP (17), length 61) 10.0.2.16.37700 > 10.0.2.3.domain: [bad udp cksum 0x184d -> 0x8e11!] 32678+ A? www.freebsd.org. (33) 19:13:59.209415 IP (tos 0x0, ttl 64, id 30, offset 0, flags [none], proto UDP (17), length 110) 10.0.2.3.domain > 10.0.2.16.37700: [udp sum ok] 32678 q: A? www.freebsd.org. 2/0/0 www.freebsd.org. [10s] CNAME web.geo.freebsd.org., web.geo.freebsd.org. [2m30s] A 85.30.190.141 (82) 19:13:59.210656 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.0.2.16.10688 > wfe0.sjb.freebsd.org.https: Flags [S], cksum 0x1fea (incorrect -> 0xc108), seq 4277776097, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 3808209639 ecr 0], length 0 19:13:59.248539 IP (tos 0x0, ttl 64, id 31, offset 0, flags [none], proto TCP (6), length 44) wfe0.sjb.freebsd.org.https > 10.0.2.16.10688: Flags [S.], cksum 0x34ea (correct), seq 1600001, ack 4277776098, win 65535, options [mss 1460], length 0 ``` This suggests to me that FreeBSD 15 included changes to the dwc driver that broke the checksum offloading support on the RockPro64 SBC. Taking a quick glance at the code, it appears to have been re-factored, split up into multiple code files from one, so it's possible that something slipped during that process.
Just double checking: Can you ssh into the host (not the jail) from external nodes (not a local jail) or vice versa?
Thanks for reporting your problem. If I understand your text correctly, your setup looks like this: Ext. Host Host Host-Jail bridge0 ?------------dwc0 epair111a---epair111b .101 .100 ? 192.168.100.0/24 Since epair111b has txcsum enabled, the jail uses checksum offloading and sends TCP segments with an incorrect checksum (i.e., the checksum for the pseudo header). epair does not compute it, so it's OK to see the incorrect checksum on the host, as your tcpdump shows. When the segments leave the host over dwc0, the network card should compute the correct checksum and insert it in the TCP header field. Does this happen? If you can start tcpdump on the external host you will find it out by looking at the checksum of the segments from the jail. They should by correct there. If they are still incorrect, the external host will drop the segments. If that's the case, do you see the same problem with TCP segments coming from the host itself (not the jail) to the external host?
That is correct, I'm able to SSH into the host machine and access the internet there without any issues, it's the jails that have problem with external connectivity. Network traffic between the jails themselves and the jails and the host machine works correctly, but any time the traffic leaves through the dwc NIC, those packets appear to be dropped. The topology drawing created by Timo is correct. And as for the packets on the external host, when I run tcpdump and try to SSH into the jails from the external host (my workstation), I do not see reply packets, they must've already been dropped earlier in the process. I only see the reply packets when I run tcpdump on the RockPro64 with FreeBSD. When I disable TXCSUM offloading on the epair, the checksums of packets coming from the epair into the bridge are correct, so that's where the similarity to #235607 comes in, the packet forwarding works when the checksum is already correct.
(In reply to Marek Benc from comment #3) What does tcpdump show, if you capture on the dwc0 device and try to ssh into the jail? Are you running any firewall or nat on your RockPro64?
Yes, I do see the reply packets when doing a tcpdump on dwc0, here's what I see when trying to access the jail (irc.central-rock.home, 192.168.100.111) from my workstation (192.168.100.101) with TXCSUM enabled on epair111: ``` root@central-rock:~ # tcpdump -i dwc0 -vvv | grep irc tcpdump: listening on dwc0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 192.168.100.101.41467 > central-rock.home.domain: [udp sum ok] 26274+ A? irc.central-rock.home. (39) 192.168.100.101.41467 > central-rock.home.domain: [udp sum ok] 30372+ AAAA? irc.central-rock.home. (39) central-rock.home.domain > 192.168.100.101.41467: [udp sum ok] 26274* q: A? irc.central-rock.home. 1/0/0 irc.central-rock.home. [1w] A 192.168.100.111 (55) central-rock.home.domain > 192.168.100.101.41467: [udp sum ok] 30372* q: AAAA? irc.central-rock.home. 0/1/0 ns: home. [1h] SOA ns1.home. root.central-rock.home. 2025051800 86400 7200 3600000 3600 (88) 192.168.100.101.51108 > irc.central-rock.home.ssh: Flags [S], cksum 0x7d86 (correct), seq 3676237699, win 64240, options [mss 1460,sackOK,TS val 2378327360 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > 192.168.100.101.51108: Flags [S.], cksum 0x4a54 (incorrect -> 0xedc4), seq 3074001737, ack 3676237700, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2417932289 ecr 2378327360], length 0 central-rock.home.domain > 192.168.100.101.42128: [udp sum ok] 38881* q: PTR? 111.100.168.192.in-addr.arpa. 1/0/0 111.100.168.192.IN-ADDR.ARPA. [1w] PTR irc.central-rock.home. (109) 192.168.100.101.51108 > irc.central-rock.home.ssh: Flags [S], cksum 0x799d (correct), seq 3676237699, win 64240, options [mss 1460,sackOK,TS val 2378328361 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > 192.168.100.101.51108: Flags [S.], cksum 0x4a54 (incorrect -> 0xe5f1), seq 3074001737, ack 3676237700, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2417933291 ecr 2378328361], length 0 192.168.100.101.51108 > irc.central-rock.home.ssh: Flags [S], cksum 0x759d (correct), seq 3676237699, win 64240, options [mss 1460,sackOK,TS val 2378329385 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > 192.168.100.101.51108: Flags [S.], cksum 0x4a54 (incorrect -> 0xddf1), seq 3074001737, ack 3676237700, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2417934315 ecr 2378329385], length 0 192.168.100.101.51108 > irc.central-rock.home.ssh: Flags [S], cksum 0x719d (correct), seq 3676237699, win 64240, options [mss 1460,sackOK,TS val 2378330409 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > 192.168.100.101.51108: Flags [S.], cksum 0x4a54 (incorrect -> 0xd5f1), seq 3074001737, ack 3676237700, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2417935339 ecr 2378330409], length 0 192.168.100.101.51108 > irc.central-rock.home.ssh: Flags [S], cksum 0x6d9d (correct), seq 3676237699, win 64240, options [mss 1460,sackOK,TS val 2378331433 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > 192.168.100.101.51108: Flags [S.], cksum 0x4a54 (incorrect -> 0xcdf1), seq 3074001737, ack 3676237700, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2417936363 ecr 2378331433], length 0 ``` As you can see, the SSH server in the jail is trying to send a reply to the packets from my workstation, but the ssh client doesn't appear to receive the reply. And now that I've tested it again on the workstation, I do see that the packets do actually end up arriving, with the wrong checksums though: ``` # tcpdump -vvv | grep -E '192.168.100.111|irc' tcpdump: listening on enp9s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes sapphire-scalpel.home.41467 > central-rock.home.domain: [udp sum ok] 26274+ A? irc.central-rock.home. (39) sapphire-scalpel.home.41467 > central-rock.home.domain: [udp sum ok] 30372+ AAAA? irc.central-rock.home. (39) central-rock.home.domain > sapphire-scalpel.home.41467: [udp sum ok] 26274* q: A? irc.central-rock.home. 1/0/0 irc.central-rock.home. [1w] A 192.168.100.111 (55) central-rock.home.domain > sapphire-scalpel.home.41467: [udp sum ok] 30372* q: AAAA? irc.central-rock.home. 0/1/0 ns: home. [1h] SOA ns1.home. root.central-rock.home. 2025051800 86400 7200 3600000 3600 (88) sapphire-scalpel.home.51108 > irc.central-rock.home.ssh: Flags [S], cksum 0x7d86 (correct), seq 3676237699, win 64240, options [mss 1460,sackOK,TS val 2378327360 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > sapphire-scalpel.home.51108: Flags [S.], cksum 0x4a54 (incorrect -> 0xedc4), seq 3074001737, ack 3676237700, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2417932289 ecr 2378327360], length 0 central-rock.home.domain > sapphire-scalpel.home.42128: [udp sum ok] 38881* q: PTR? 111.100.168.192.in-addr.arpa. 1/0/0 111.100.168.192.IN-ADDR.ARPA. [1w] PTR irc.central-rock.home. (109) sapphire-scalpel.home.51108 > irc.central-rock.home.ssh: Flags [S], cksum 0x799d (correct), seq 3676237699, win 64240, options [mss 1460,sackOK,TS val 2378328361 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > sapphire-scalpel.home.51108: Flags [S.], cksum 0x4a54 (incorrect -> 0xe5f1), seq 3074001737, ack 3676237700, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2417933291 ecr 2378328361], length 0 sapphire-scalpel.home.51108 > irc.central-rock.home.ssh: Flags [S], cksum 0x759d (correct), seq 3676237699, win 64240, options [mss 1460,sackOK,TS val 2378329385 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > sapphire-scalpel.home.51108: Flags [S.], cksum 0x4a54 (incorrect -> 0xddf1), seq 3074001737, ack 3676237700, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2417934315 ecr 2378329385], length 0 sapphire-scalpel.home.51108 > irc.central-rock.home.ssh: Flags [S], cksum 0x719d (correct), seq 3676237699, win 64240, options [mss 1460,sackOK,TS val 2378330409 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > sapphire-scalpel.home.51108: Flags [S.], cksum 0x4a54 (incorrect -> 0xd5f1), seq 3074001737, ack 3676237700, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2417935339 ecr 2378330409], length 0 sapphire-scalpel.home.51108 > irc.central-rock.home.ssh: Flags [S], cksum 0x6d9d (correct), seq 3676237699, win 64240, options [mss 1460,sackOK,TS val 2378331433 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > sapphire-scalpel.home.51108: Flags [S.], cksum 0x4a54 (incorrect -> 0xcdf1), seq 3074001737, ack 3676237700, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2417936363 ecr 2378331433], length 0 irc.central-rock.home.ssh > sapphire-scalpel.home.51108: Flags [S.], cksum 0x4a54 (incorrect -> 0xca07), seq 3074001737, ack 3676237700, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2417937365 ecr 2378331433], length 0 ``` So apologies for my earlier statements about them not arriving, turns out I had tested it wrong previously, they do arrive, but with a bad checksum, which prevents further processing. It's not limited to just accessing my workstation, but also the internet at large, e.g. I was unable to use pkg to update the jails with the -j option, I had to use the chroot option -c. As for a firewall, yes, I do have a pf firewall, but not a NAT, here is my pf.conf: ``` scrub in ext_if="bridge0" anchor "blocklistd/*" in on $ext_if block all pass proto tcp from 127.0.0.1 to 127.0.0.1 port { rndc } keep state pass out proto tcp to any port { ssh, domain, kerberos-sec, nfsd, sunrpc, http, https } keep state pass in proto tcp to any port { ssh, domain, kerberos-sec, nfsd, sunrpc } keep state pass proto udp to any port { domain, ntp, kerberos-sec, nfsd, sunrpc } keep state block all user disconnected_guy ``` I tried disabling and enabling the firewall, but it had no effect.
(In reply to Marek Benc from comment #5) Could you disable pf by unloading the module (like running kldunload pf) or by even not loading the module at all? Does the problem persist when no pf module is loaded at all?
I've disabled pf, blacklisting the module and disabling it in rc.conf, but I'm still getting the same behavior: ``` root@central-rock:~ # kldstat Id Refs Address Size Name 1 44 0xffff000000000000 140b840 kernel 2 1 0xffff00000140d000 256e0 umodem.ko 3 2 0xffff000001433000 28c58 ucom.ko 4 1 0xffff0000d6e00000 33000 geom_eli.ko 5 1 0xffff0000d7400000 3e9000 zfs.ko 6 1 0xffff0000d6e33000 24000 mx25l.ko 7 1 0xffff0000d6e57000 22000 fdt_slicer.ko 8 1 0xffff0000d6e79000 2e000 if_bridge.ko 9 1 0xffff0000d6ea7000 27000 bridgestp.ko 10 1 0xffff0000d6ece000 22000 mac_ntpd.ko 11 1 0xffff0000d6ef0000 28000 nullfs.ko 12 1 0xffff0000d6f18000 28000 kgssapi_krb5.ko 13 1 0xffff0000d6f40000 34000 kgssapi.ko 14 1 0xffff0000d6f74000 21000 rc4.ko 15 1 0xffff0000d6f95000 27000 if_epair.ko root@central-rock:~ # tcpdump -i dwc0 -vvv | grep irc tcpdump: listening on dwc0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 192.168.100.101.45558 > central-rock.home.domain: [udp sum ok] 12906+ A? irc.central-rock.home. (39) 192.168.100.101.45558 > central-rock.home.domain: [udp sum ok] 29543+ AAAA? irc.central-rock.home. (39) central-rock.home.domain > 192.168.100.101.45558: [udp sum ok] 12906* q: A? irc.central-rock.home. 1/0/0 irc.central-rock.home. [1w] A 192.168.100.111 (55) central-rock.home.domain > 192.168.100.101.45558: [udp sum ok] 29543* q: AAAA? irc.central-rock.home. 0/1/0 ns: home. [1h] SOA ns1.home. root.central-rock.home. 2025051800 86400 7200 3600000 3600 (88) 192.168.100.101.60970 > irc.central-rock.home.ssh: Flags [S], cksum 0x948e (correct), seq 2738041900, win 64240, options [mss 1460,sackOK,TS val 2393643531 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > 192.168.100.101.60970: Flags [S.], cksum 0x4a54 (incorrect -> 0x7fec), seq 1129113533, ack 2738041901, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 1801943314 ecr 2393643531], length 0 192.168.100.101.60970 > irc.central-rock.home.ssh: Flags [S], cksum 0x9090 (correct), seq 2738041900, win 64240, options [mss 1460,sackOK,TS val 2393644553 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > 192.168.100.101.60970: Flags [S.], cksum 0x4a54 (incorrect -> 0x77f1), seq 1129113533, ack 2738041901, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 1801944335 ecr 2393644553], length 0 192.168.100.101.60970 > irc.central-rock.home.ssh: Flags [S], cksum 0x8c90 (correct), seq 2738041900, win 64240, options [mss 1460,sackOK,TS val 2393645577 ecr 0,nop,wscale 10], length 0 ```
Thanks for checking...
Can you test the following setup: Don't create a bridge interface and assign the address, which was assigned to the bridge interface, to the dwc interface. Try ssh'ing into the host or from the host to some node in your network. Does that work? We want to make sure that checksum offloading does work on the dwc interface. Unfortunately, we don't have a dwc interface for testing and we were not able to reproduce your issue here with a different NIC.
(In reply to Michael Tuexen from comment #9) Sure! I just gave it a try, and it seems to work fine, I'm able to access the RockPro64 and access the internet and other machines on the network just fine: ``` root@central-rock:~ # ifconfig dwc0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE> ether 82:16:d0:7b:f5:0b inet 192.168.100.100 netmask 0xffffff00 broadcast 192.168.100.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152 options=0 groups: pflog root@central-rock:~ # ssh 192.168.100.101 The authenticity of host '192.168.100.101 (192.168.100.101)' can't be established. ED25519 key fingerprint is SHA256:hSJajRMbzpf1fF8t980K7IEgqgBYOn5xP+lQcmnHo2A. This host key is known by the following other names/addresses: ~/.ssh/known_hosts:4: sapphire-scalpel.home Are you sure you want to continue connecting (yes/no/[fingerprint])? ^C root@central-rock:~ # fetch https://www.freebsd.org/ fetch.out 15 kB 436 kBps 00s root@central-rock:~ # file fetch.out fetch.out: HTML document, Unicode text, UTF-8 text ``` As you can see, TXCSUM is listed in the options in ifconfig for dwc0, so it seems to handle offloading the checksum calculations for normal packets, but appears to ignore packets coming from an epipe and leaves their invalid checksums intact, as seen from that tcpdump I showed earlier where I ran it on my workstation.
(In reply to Marek Benc from comment #10) Thanks for testing. Right now we are running out of ideas what can cause the problem you are experiencing. The packet dwc receives when you are using it directly should be the same as when receiving the packets via the bridge interface when coming into the bridge interface via an epair. And the setup is working for us when using a different interface. So either there is something special related to your setup we are failing to see or there is a problem we have not yet figured out. Is it possible for you to compile a kernel on the RockPro if we send you a patch or should we focus on trying dtrace to gather more information. I guess we need to figure out if the dwc interface sees the flags which should tell it to compute the checksums.
(In reply to Michael Tuexen from comment #11) Sure, I'm willing to assist in testing patches! Though just a heads-up, I'll be gone for a week starting on next next Monday (22nd), returning back on Tuesday 30th, so there will be a gap in my availability there. And yes, I do think looking at the flags and looking for where the request gets lost is a good approach!
(In reply to Marek Benc from comment #12) Thanks for testing the case without a bridge. I dug a little deeper into the code of the dwc driver. The code around this line https://cgit.freebsd.org/src/tree/sys/dev/dwc/dwc1000_dma.c?h=releng/15.0#n280 reads like dwc does checksum offloading for TCP/UDP only if it does checksum offloading for IP (the IPv4 header checksum). The issue with this is that, currently, epair offers checksum offloading only for TCP/UDP, not for IP. Hence, packets from epair may have CSUM_TCP or CSUM_UDP set, but never CSUM_IP. I created a patch against the FreeBSD 15.0-RELEASE code that changes the epair interface to offer checksum offloading for IP as well and attached it (epair_csum_ip.patch). It changes only one line in if_epair.c. Can you apply the patch, build and install the patched kernel, and run the test in your epair setup again?
Created attachment 266365 [details] epair_csum_ip.patch
(In reply to Timo Voelker from comment #13) Thanks! That fixed up the problem with the external checksums, and I'm able to connect to the jail just fine now: ``` root@central-rock:~ # ifconfig -a dwc0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE> ether 82:16:d0:7b:f5:0b media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=10<VLAN_HWTAGGING> ether 58:9c:fc:10:77:67 inet 192.168.100.100 netmask 0xffffff00 broadcast 192.168.100.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 bridge flags=0<> member: epair111a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> port 9 priority 128 path cost 2000 vlan protocol 802.1q member: epair110a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> port 5 priority 128 path cost 2000 vlan protocol 802.1q member: dwc0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> port 1 priority 128 path cost 55 vlan protocol 802.1q groups: bridge nd6 options=9<PERFORMNUD,IFDISABLED> pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152 options=0 groups: pflog epair110a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: jail:www options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6> ether 58:9c:fc:10:24:0b groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> epair111a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: jail:irc options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6> ether 58:9c:fc:10:bf:1d groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@central-rock:~ # tcpdump -vvvv -i epair111a tcpdump: listening on epair111a, link-type EN10MB (Ethernet), snapshot length 262144 bytes 11:04:29.155847 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has irc.central-rock.home tell 192.168.100.101, length 46 11:04:29.155931 ARP, Ethernet (len 6), IPv4 (len 4), Reply irc.central-rock.home is-at 58:9c:fc:10:98:82 (oui Unknown), length 28 11:04:29.156447 IP (tos 0x10, ttl 64, id 36020, offset 0, flags [DF], proto TCP (6), length 60) 192.168.100.101.34006 > irc.central-rock.home.ssh: Flags [S], cksum 0x083d (correct), seq 330289282, win 64240, options [mss 1460,sackOK,TS val 1740581324 ecr 0,nop,wscale 10], length 0 11:04:29.156559 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->f096)!) irc.central-rock.home.ssh > 192.168.100.101.34006: Flags [S.], cksum 0x4a54 (incorrect -> 0x8390), seq 1118561469, ack 330289283, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2990843360 ecr 1740581324], length 0 11:04:29.156683 IP (tos 0x10, ttl 64, id 36021, offset 0, flags [DF], proto TCP (6), length 52) 192.168.100.101.34006 > irc.central-rock.home.ssh: Flags [.], cksum 0xb21d (correct), seq 1, ack 1, win 63, options [nop,nop,TS val 1740581325 ecr 2990843360], length 0 11:04:29.156863 IP (tos 0x10, ttl 64, id 36022, offset 0, flags [DF], proto TCP (6), length 85) 192.168.100.101.34006 > irc.central-rock.home.ssh: Flags [P.], cksum 0xf5b1 (correct), seq 1:34, ack 1, win 63, options [nop,nop,TS val 1740581325 ecr 2990843360], length 33: SSH: SSH-2.0-OpenSSH_10.0p2 Debian-7 11:04:29.199126 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->f09e)!) irc.central-rock.home.ssh > 192.168.100.101.34006: Flags [.], cksum 0x4a4c (incorrect -> 0xb110), seq 1, ack 34, win 257, options [nop,nop,TS val 2990843402 ecr 1740581325], length 0 11:04:29.225799 IP (tos 0x0, ttl 64, id 12357, offset 0, flags [none], proto UDP (17), length 74, bad cksum 0 (->9d)!) irc.central-rock.home.39084 > 192.168.100.1.domain: [bad udp cksum 0x4a09 -> 0x3540!] 63169+ PTR? 101.100.168.192.in-addr.arpa. (46) 11:04:29.226778 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has irc.central-rock.home tell 192.168.100.1, length 46 11:04:29.226846 ARP, Ethernet (len 6), IPv4 (len 4), Reply irc.central-rock.home is-at 58:9c:fc:10:98:82 (oui Unknown), length 28 11:04:29.227122 IP (tos 0x0, ttl 64, id 18754, offset 0, flags [DF], proto UDP (17), length 74) 192.168.100.1.domain > irc.central-rock.home.39084: [udp sum ok] 63169 NXDomain q: PTR? 101.100.168.192.in-addr.arpa. 0/0/0 (46) 11:04:29.227626 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 91, bad cksum 0 (->f077)!) irc.central-rock.home.ssh > 192.168.100.101.34006: Flags [P.], cksum 0x4a73 (incorrect -> 0xb1f2), seq 1:40, ack 34, win 257, options [nop,nop,TS val 2990843431 ecr 1740581325], length 39: SSH: SSH-2.0-OpenSSH_10.0 FreeBSD-20250801 11:04:29.227755 IP (tos 0x10, ttl 64, id 36023, offset 0, flags [DF], proto TCP (6), length 52) 192.168.100.101.34006 > irc.central-rock.home.ssh: Flags [.], cksum 0xb147 (correct), seq 34, ack 40, win 63, options [nop,nop,TS val 1740581396 ecr 2990843431], length 0 11:04:29.227972 IP (tos 0x10, ttl 64, id 36024, offset 0, flags [DF], proto TCP (6), length 1500) 192.168.100.101.34006 > irc.central-rock.home.ssh: Flags [.], cksum 0x5d5d (correct), seq 34:1482, ack 40, win 63, options [nop,nop,TS val 1740581396 ecr 2990843431], length 1448 11:04:29.227992 IP (tos 0x10, ttl 64, id 36025, offset 0, flags [DF], proto TCP (6), length 172) 192.168.100.101.34006 > irc.central-rock.home.ssh: Flags [P.], cksum 0xe52f (correct), seq 1482:1602, ack 40, win 63, options [nop,nop,TS val 1740581396 ecr 2990843431], length 120 11:04:29.228060 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->f09e)!) irc.central-rock.home.ssh > 192.168.100.101.34006: Flags [.], cksum 0x4a4c (incorrect -> 0xaa6b), seq 40, ack 1602, win 251, options [nop,nop,TS val 2990843431 ecr 1740581396], length 0 11:04:29.267085 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1092, bad cksum 0 (->ec8e)!) irc.central-rock.home.ssh > 192.168.100.101.34006: Flags [P.], cksum 0x4e5c (incorrect -> 0x144c), seq 40:1080, ack 1602, win 251, options [nop,nop,TS val 2990843470 ecr 1740581396], length 1040 11:04:29.268488 IP (tos 0x10, ttl 64, id 36026, offset 0, flags [DF], proto TCP (6), length 1284) 192.168.100.101.34006 > irc.central-rock.home.ssh: Flags [P.], cksum 0x28e3 (correct), seq 1602:2834, ack 1080, win 66, options [nop,nop,TS val 1740581437 ecr 2990843470], length 1232 11:04:29.299433 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500, bad cksum 0 (->eaf6)!) irc.central-rock.home.ssh > 192.168.100.101.34006: Flags [.], cksum 0x4ff4 (incorrect -> 0x97ff), seq 1080:2528, ack 2834, win 257, options [nop,nop,TS val 2990843502 ecr 1740581437], length 1448 11:04:29.299469 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 184, bad cksum 0 (->f01a)!) irc.central-rock.home.ssh > 192.168.100.101.34006: Flags [P.], cksum 0x4ad0 (incorrect -> 0x9193), seq 2528:2660, ack 2834, win 257, options [nop,nop,TS val 2990843502 ecr 1740581437], length 132 11:04:29.299641 IP (tos 0x10, ttl 64, id 36027, offset 0, flags [DF], proto TCP (6), length 52) 192.168.100.101.34006 > irc.central-rock.home.ssh: Flags [.], cksum 0x9b8a (correct), seq 2834, ack 2660, win 65, options [nop,nop,TS val 1740581468 ecr 2990843502], length 0 11:04:29.302178 IP (tos 0x10, ttl 64, id 36028, offset 0, flags [DF], proto TCP (6), length 136) 192.168.100.101.34006 > irc.central-rock.home.ssh: Flags [P.], cksum 0x9d64 (correct), seq 2834:2918, ack 2660, win 65, options [nop,nop,TS val 1740581470 ecr 2990843502], length 84 ``` tcpdump from the workstation: ``` # tcpdump -vvvv | grep -E 'irc|192.168.100.111' tcpdump: listening on enp9s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes sapphire-scalpel.home.34257 > central-rock.home.domain: [udp sum ok] 50646+ A? irc.central-rock.home. (39) sapphire-scalpel.home.34257 > central-rock.home.domain: [udp sum ok] 41424+ AAAA? irc.central-rock.home. (39) central-rock.home.domain > sapphire-scalpel.home.34257: [udp sum ok] 50646* q: A? irc.central-rock.home. 1/0/0 irc.central-rock.home. [1w] A 192.168.100.111 (55) central-rock.home.domain > sapphire-scalpel.home.34257: [udp sum ok] 41424* q: AAAA? irc.central-rock.home. 0/1/0 ns: home. [1h] SOA ns1.home. root.central-rock.home. 2025051800 86400 7200 3600000 3600 (88) sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [S], cksum 0x34b1 (correct), seq 330289282, win 64240, options [mss 1460,sackOK,TS val 1740569944 ecr 0,nop,wscale 10], length 0 central-rock.home.domain > sapphire-scalpel.home.35849: [udp sum ok] 57286* q: PTR? 111.100.168.192.in-addr.arpa. 1/0/0 111.100.168.192.IN-ADDR.ARPA. [1w] PTR irc.central-rock.home. (109) sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [S], cksum 0x30bd (correct), seq 330289282, win 64240, options [mss 1460,sackOK,TS val 1740570956 ecr 0,nop,wscale 10], length 0 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [S], cksum 0x2cbd (correct), seq 330289282, win 64240, options [mss 1460,sackOK,TS val 1740571980 ecr 0,nop,wscale 10], length 0 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [S], cksum 0x28bd (correct), seq 330289282, win 64240, options [mss 1460,sackOK,TS val 1740573004 ecr 0,nop,wscale 10], length 0 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [S], cksum 0x24bd (correct), seq 330289282, win 64240, options [mss 1460,sackOK,TS val 1740574028 ecr 0,nop,wscale 10], length 0 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [S], cksum 0x20bd (correct), seq 330289282, win 64240, options [mss 1460,sackOK,TS val 1740575052 ecr 0,nop,wscale 10], length 0 11:04:23.010726 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has irc.central-rock.home tell sapphire-scalpel.home, length 28 11:04:24.035044 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has irc.central-rock.home tell sapphire-scalpel.home, length 28 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [S], cksum 0x18dd (correct), seq 330289282, win 64240, options [mss 1460,sackOK,TS val 1740577068 ecr 0,nop,wscale 10], length 0 11:04:25.059044 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has irc.central-rock.home tell sapphire-scalpel.home, length 28 11:04:29.155117 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has irc.central-rock.home tell sapphire-scalpel.home, length 28 11:04:29.155743 ARP, Ethernet (len 6), IPv4 (len 4), Reply irc.central-rock.home is-at 58:9c:fc:10:98:82 (oui Unknown), length 46 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [S], cksum 0x083d (correct), seq 330289282, win 64240, options [mss 1460,sackOK,TS val 1740581324 ecr 0,nop,wscale 10], length 0 irc.central-rock.home.ssh > sapphire-scalpel.home.34006: Flags [S.], cksum 0x8390 (correct), seq 1118561469, ack 330289283, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2990843360 ecr 1740581324], length 0 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [.], cksum 0xb21d (correct), seq 1, ack 1, win 63, options [nop,nop,TS val 1740581325 ecr 2990843360], length 0 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [P.], cksum 0xf5b1 (correct), seq 1:34, ack 1, win 63, options [nop,nop,TS val 1740581325 ecr 2990843360], length 33: SSH: SSH-2.0-OpenSSH_10.0p2 Debian-7 irc.central-rock.home.ssh > sapphire-scalpel.home.34006: Flags [.], cksum 0xb110 (correct), seq 1, ack 34, win 257, options [nop,nop,TS val 2990843402 ecr 1740581325], length 0 11:04:29.226136 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has irc.central-rock.home tell _gateway, length 46 irc.central-rock.home.ssh > sapphire-scalpel.home.34006: Flags [P.], cksum 0xb1f2 (correct), seq 1:40, ack 34, win 257, options [nop,nop,TS val 2990843431 ecr 1740581325], length 39: SSH: SSH-2.0-OpenSSH_10.0 FreeBSD-20250801 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [.], cksum 0xb147 (correct), seq 34, ack 40, win 63, options [nop,nop,TS val 1740581396 ecr 2990843431], length 0 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [.], cksum 0x5d5d (correct), seq 34:1482, ack 40, win 63, options [nop,nop,TS val 1740581396 ecr 2990843431], length 1448 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [P.], cksum 0xe52f (correct), seq 1482:1602, ack 40, win 63, options [nop,nop,TS val 1740581396 ecr 2990843431], length 120 irc.central-rock.home.ssh > sapphire-scalpel.home.34006: Flags [.], cksum 0xaa6b (correct), seq 40, ack 1602, win 251, options [nop,nop,TS val 2990843431 ecr 1740581396], length 0 irc.central-rock.home.ssh > sapphire-scalpel.home.34006: Flags [P.], cksum 0x144c (correct), seq 40:1080, ack 1602, win 251, options [nop,nop,TS val 2990843470 ecr 1740581396], length 1040 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [P.], cksum 0x28e3 (correct), seq 1602:2834, ack 1080, win 66, options [nop,nop,TS val 1740581437 ecr 2990843470], length 1232 irc.central-rock.home.ssh > sapphire-scalpel.home.34006: Flags [P.], cksum 0x5078 (incorrect -> 0x8e25), seq 1080:2660, ack 2834, win 257, options [nop,nop,TS val 2990843502 ecr 1740581437], length 1580 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [.], cksum 0x9b8a (correct), seq 2834, ack 2660, win 65, options [nop,nop,TS val 1740581468 ecr 2990843502], length 0 sapphire-scalpel.home.34006 > irc.central-rock.home.ssh: Flags [P.], cksum 0x9d64 (correct), seq 2834:2918, ack 2660, win 65, options [nop,nop,TS val 1740581470 ecr 2990843502], length 84 ``` Though I'm seeing two issues. One is less severe, and is probably unrelated to the patch, but for packets that end up being merged, like in the following case, I'm getting a bad checksum: on epair: ``` 11:04:29.299433 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500, bad cksum 0 (->eaf6)!) irc.central-rock.home.ssh > 192.168.100.101.34006: Flags [.], cksum 0x4ff4 (incorrect -> 0x97ff), seq 1080:2528, ack 2834, win 257, options [nop,nop,TS val 2990843502 ecr 1740581437], length 1448 11:04:29.299469 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 184, bad cksum 0 (->f01a)!) irc.central-rock.home.ssh > 192.168.100.101.34006: Flags [P.], cksum 0x4ad0 (incorrect -> 0x9193), seq 2528:2660, ack 2834, win 257, options [nop,nop,TS val 2990843502 ecr 1740581437], length 132 ``` on the destination: ``` irc.central-rock.home.ssh > sapphire-scalpel.home.34006: Flags [P.], cksum 0x5078 (incorrect -> 0x8e25), seq 1080:2660, ack 2834, win 257, options [nop,nop,TS val 2990843502 ecr 1740581437], length 1580 ``` The second, more severe issue, that you should probably be ale to reproduce as well with the patch, is that I'm now unable to access the other jails, or even the host machine, from within a jail, only external connectivity works, here's an example: ``` Terminal 1: root@irc:~ $ fetch http://192.168.100.110 <hang> Terminal 2: root@central-rock:~ # tcpdump -vvvv -i epair111a tcpdump: listening on epair111a, link-type EN10MB (Ethernet), snapshot length 262144 bytes 11:25:17.352867 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->f08d)!) irc.central-rock.home.36989 > www.central-rock.home.http: Flags [S], cksum 0x4a5d (incorrect -> 0xa34e), seq 1878053123, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2272264528 ecr 0], length 0 11:25:18.373013 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->f08d)!) irc.central-rock.home.36989 > www.central-rock.home.http: Flags [S], cksum 0x4a5d (incorrect -> 0x9f52), seq 1878053123, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2272265548 ecr 0], length 0 11:25:20.598786 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->f08d)!) irc.central-rock.home.36989 > www.central-rock.home.http: Flags [S], cksum 0x4a5d (incorrect -> 0x96a0), seq 1878053123, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2272267774 ecr 0], length 0 ^C 3 packets captured 3 packets received by filter 0 packets dropped by kernel Terminal 3: root@central-rock:~ # tcpdump -vvvv -i epair110a tcpdump: listening on epair110a, link-type EN10MB (Ethernet), snapshot length 262144 bytes 11:25:17.352892 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->f08d)!) irc.central-rock.home.36989 > www.central-rock.home.http: Flags [S], cksum 0x4a5d (incorrect -> 0xa34e), seq 1878053123, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2272264528 ecr 0], length 0 11:25:18.373032 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->f08d)!) irc.central-rock.home.36989 > www.central-rock.home.http: Flags [S], cksum 0x4a5d (incorrect -> 0x9f52), seq 1878053123, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2272265548 ecr 0], length 0 11:25:20.598814 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->f08d)!) irc.central-rock.home.36989 > www.central-rock.home.http: Flags [S], cksum 0x4a5d (incorrect -> 0x96a0), seq 1878053123, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 2272267774 ecr 0], length 0 ^C 3 packets captured 3 packets received by filter 0 packets dropped by kernel ``` We see that the packets are actually sent to the neighboring jail, but for some reason they aren't handled properly So it seems that adding the CSUM_IP flag in sys/net/if_epair.c prevents the packets on the receiving end of the epair to be handled correctly by the FreeBSD system itself. This error manifested to me as not being able to use the DNS server I have set up on the host machine within the jails, on 192.168.100.100. Upon using my router's DNS server in the jails, "everything seems to work", except for accessing services on the other jails or on the host machine itself. After disabling txcsum on both epairs, the connectivity works again: ``` Terminal 2: root@central-rock:~ # ifconfig epair111a -txcsum root@central-rock:~ # ifconfig epair110a -txcsum Terminal 1: root@central-rock:~ # jexec -u root irc fetch http://192.168.100.110 192.168.100.110 362 B 807 kBps 00s Terminal 2: root@central-rock:~ # tcpdump -vvvv -i epair111a tcpdump: listening on epair111a, link-type EN10MB (Ethernet), snapshot length 262144 bytes 11:33:57.964570 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) irc.central-rock.home.45688 > www.central-rock.home.http: Flags [S], cksum 0x69ba (correct), seq 2254953706, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 3014511181 ecr 0], length 0 11:33:57.964782 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) www.central-rock.home.http > irc.central-rock.home.45688: Flags [S.], cksum 0xf72e (correct), seq 1955010186, ack 2254953707, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 4171689665 ecr 3014511181], length 0 11:33:57.964859 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) irc.central-rock.home.45688 > www.central-rock.home.http: Flags [.], cksum 0x24fb (correct), seq 1, ack 1, win 257, options [nop,nop,TS val 3014511181 ecr 4171689665], length 0 11:33:57.965658 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 157) irc.central-rock.home.45688 > www.central-rock.home.http: Flags [P.], cksum 0x6b81 (correct), seq 1:106, ack 1, win 257, options [nop,nop,TS val 3014511182 ecr 4171689665], length 105: HTTP, length: 105 GET / HTTP/1.1 Host: 192.168.100.110 Accept: */* User-Agent: fetch libfetch/2.0 Connection: close ... Terminal 3: tcpdump: listening on epair110a, link-type EN10MB (Ethernet), snapshot length 262144 bytes 11:33:57.964591 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) irc.central-rock.home.45688 > www.central-rock.home.http: Flags [S], cksum 0x69ba (correct), seq 2254953706, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 3014511181 ecr 0], length 0 11:33:57.964773 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) www.central-rock.home.http > irc.central-rock.home.45688: Flags [S.], cksum 0xf72e (correct), seq 1955010186, ack 2254953707, win 65535, options [mss 1460,nop,wscale 8,sackOK,TS val 4171689665 ecr 3014511181], length 0 11:33:57.964866 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) irc.central-rock.home.45688 > www.central-rock.home.http: Flags [.], cksum 0x24fb (correct), seq 1, ack 1, win 257, options [nop,nop,TS val 3014511181 ecr 4171689665], length 0 11:33:57.965668 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 157) irc.central-rock.home.45688 > www.central-rock.home.http: Flags [P.], cksum 0x6b81 (correct), seq 1:106, ack 1, win 257, options [nop,nop,TS val 3014511182 ecr 4171689665], length 105: HTTP, length: 105 GET / HTTP/1.1 Host: 192.168.100.110 Accept: */* User-Agent: fetch libfetch/2.0 Connection: close ... ```
(In reply to Marek Benc from comment #15) Thanks for testing. The result makes it pretty clear that the reason relates to the dwc driver doing TCP/UDP checksum offloading only if IP checksum offloading is requested as well. I find it strange to couple these two checksum offloading features. Issue one: I agree that it is probably unrelated to the patch. I don't understand what happens. Who merges the packets? Can you give more details of your setup where packets get merged. Issue two: I expected that jail to jail or jail to host TCP connection won't work with the patch. The patch was just for testing a TCP connection to the external host. Sorry for not making that clear. I don't see a huge benefit in adding IP checksum offloading to epair. It seems better to first understand why the dwc driver was implemented like that. I'll see how far I get there.
Created attachment 266418 [details] Proposed patch that changes the conditions for the offloading flags I've also taken a look at this driver and some other drivers, and it seems that DWC has two modes of calculating the checksums, either "full" for IP + TCP/UDP, or "header" for just the IP. Changing around the condition for when the checksum calculation flags are set, so that any of CSUM_TCP | CSUM_UDP trigger the full checksum calculation, CSUM_IP alone triggers the header calculation, and none leaves the flags unset, seems to fix the problem for me. Granted, there might be some other ramifications of this change that I'm not aware of, it would probably be best to consult this with the maintainer of the driver. And as for the packet merging, I think it has to do with packet fragmentation. When I do a tcpdump on my workstation and establish an SSH connection, whenever the packet length is reported longer than 1500 in tcpdump, where 1500 is the MTU, the TCP checksum is wrong. I'm not sure what's causing this / what's defragmenting the packets, I'm going to have to play around with this some more.
(In reply to Timo Voelker from comment #16) Yeah, the problem with the packet merging is unrelated, it seems that Linux is doing some packet re-assembly on my workstation and doesn't bother to calculate a correct checksum for the re-assembled packets. I've run a tcpdump on the machine and saw a bunch of those packets coming from the external internet, so I don't think there's anything wrong with the dwc in this regard.
Regarding your first issue: Does the TCP connection work without retransmissions? If that is the case then you have hardware LRO not fixing the checksum but just marking that the packets have a valid checksum. Regarding the second issue: We need to handle the IP header checksum the same way as other checksums on the inout path. Then we can offload the header checksum in epair. I checked the drivers. I found two that require IP Header checksum offload for TCP or UDP checksum offloadL (1) https://cgit.freebsd.org/src/tree/sys/dev/dwc/dwc1000_dma.c#n279 (2) https://cgit.freebsd.org/src/tree/sys/arm/allwinner/if_awg.c#n652 I looked at the spec for the first one and found: CIC: Checksum Insertion Control These bits control the insertion of checksums in Ethernet frames that encapsulate TCP, UDP, or ICMP over IPv4 or IPv6 as described below. * 0: Do nothing. Checksum Engine is bypassed * 1: Insert IPv4 header checksum. Use this value to insert IPv4 header checksum when the frame encapsulates an IPv4 datagram. * 2: Insert TCP/UDP/ICMP checksum. The checksum is calculated over the TCP, UDP, or ICMP segment only and the TCP, UDP, or ICMP pseudo-header checksum is] assumed to be present in the corresponding input frame’s Checksum field. An IPv4 header checksum is also inserted if the encapsulated datagram conforms to IPv4. * 3: Insert a TCP/UDP/ICMP checksum that is fully calculated in this engine. In other words, the TCP, UDP, or ICMP pseudo-header is included in the checksum calculation, and the input frame’s corresponding Checksum field has an all-zero value. An IPv4 Header checksum is also inserted if the encapsulated datagram conforms to IPv4. The Checksum engine detects whether the TCP, UDP, or ICMP segment is encapsulated in IPv4 or IPv6 and processes its data accordingly. My first observation is that IPv6 seems to be supported whereas the driver does not implement this. My second observation is that the code: if ((m->m_pkthdr.csum_flags & CSUM_IP) != 0) { if ((m->m_pkthdr.csum_flags & (CSUM_TCP|CSUM_UDP)) != 0) { if (!sc->dma_ext_desc) flags = NTDESC1_CIC_FULL; else flags = ETDESC0_CIC_FULL; } else { if (!sc->dma_ext_desc) flags = NTDESC1_CIC_HDR; else flags = ETDESC0_CIC_HDR; } } shouldn't work at all, since FreeBSD provides the pseudo header checksum and not zero. Could you test what happens if you use: if ((m->m_pkthdr.csum_flags & CSUM_IP) != 0) { if ((m->m_pkthdr.csum_flags & (CSUM_TCP|CSUM_UDP)) != 0) { if (!sc->dma_ext_desc) flags = NTDESC1_CIC_SEG; else flags = ETDESC0_CIC_SEG; } else { if (!sc->dma_ext_desc) flags = NTDESC1_CIC_HDR; else flags = ETDESC0_CIC_HDR; } } Does this allow external communication? If that works what does happen if you use: if ((m->m_pkthdr.csum_flags & (CSUM_TCP|CSUM_UDP)) != 0) { if (!sc->dma_ext_desc) flags = NTDESC1_CIC_SEG; else flags = ETDESC0_CIC_SEG; } else if ((m->m_pkthdr.csum_flags & CSUM_IP) != 0) { if (!sc->dma_ext_desc) flags = NTDESC1_CIC_HDR; else flags = ETDESC0_CIC_HDR; } I would be interested if this works with the patched and with the unpatched epair interface. Do you also have IPv6 support?
Just saw your proposed patch, which incorporates the same idea as I wanted you to test (except the *_FULL vs. *_SEG change). Just made the code a bit more compact in the alternative patch. It would be great if you could test it (once you are back).
Created attachment 266425 [details] Alternate patch for txcsum offload handling
(In reply to Michael Tuexen from comment #20) Well, I'm still at home, and had a bit of free time at my hands (thankfully incremental builds are fast), so here's the test results: dwc: patch 1 (FULL -> SEG) epair: unpatched External connectivity to host: Yes External connectivity to jails: No Internal connectivity between jails: Yes dwc: patch 1 (FULL -> SEG) epair: patched External connectivity to host: Yes External connectivity to jails: Yes Internal connectivity between jails: No dwc: patch 2 (independent SEG + HDR flags) epair: unpatched External connectivity to host: Yes External connectivity to jails: Yes Internal connectivity between jails: Yes dwc: patch 2 (independent SEG + HDR flags) epair: patched External connectivity to host: Yes External connectivity to jails: Yes Internal connectivity between jails: No dwc: patch 3 (dwc.patch) epair: unpatched External connectivity to host: Yes External connectivity to jails: Yes Internal connectivity between jails: Yes dwc: patch 3 (dwc.patch) epair: patched External connectivity to host: Yes External connectivity to jails: Yes Internal connectivity between jails: No I'm not sure I fully understand this though: > My second observation is that the code: > > if ((m->m_pkthdr.csum_flags & CSUM_IP) != 0) { > if ((m->m_pkthdr.csum_flags & (CSUM_TCP|CSUM_UDP)) != 0) { > if (!sc->dma_ext_desc) > flags = NTDESC1_CIC_FULL; > else > flags = ETDESC0_CIC_FULL; > } else { > if (!sc->dma_ext_desc) > flags = NTDESC1_CIC_HDR; > else > flags = ETDESC0_CIC_HDR; > } > } > > shouldn't work at all, since FreeBSD provides the pseudo header checksum and not zero. Could you test what happens if you use: Does this mean that the hardware is expected to fail computing the IP pseudo-header checksum if it already contains a non-zero value? Like, if it contained the correct value or garbage instead of zero? It doesn't appear to fail in my testing, changing out the _FULL here for _SEG results in the same exact behavior, as seen in the table above. Shouldn't we therefore "or in" the request to calculate the header as well when CSUM_IP is requested? The ETDESC0_CIC_FULL macro appears to be defined as NTDESC1_CIC_HDR | NTDESC1_CIC_SEG (1 | 2 = 3 shifted by the same amount), that's currently something that's missing in the latest patch (Alternate patch for txcsum offload handling, dwc.patch). In any case, I really appreciate that the two of you are taking your time to thoroughly investigate this issue and fix it up, it's not often that a bug report of mine gets a response like this, so thank you!
(In reply to Michael Tuexen from comment #19) And unfortunately, I haven't gotten to setting up IPv6 yet, I still have to sit down and figure out how it all works and how to configure my network for it. Perhaps one day :)
(In reply to Marek Benc from comment #23) No problem. That is anyway a feature improvement and not a bugfix.
(In reply to Marek Benc from comment #22) Thanks for testing! When you compute the IP header checksum, you put 0 in the field and do the computation over the header. Then you insert the result. When you compute the TCP checksum, you put 0 in it and perform the computation. The result is put into the field. The TCP checksum also covers some part of the IP header. Most hardware checksum offloading functions expect the pseudo header checksum in the TCP checksum field instead of 0 and then only perform the computation over the TCP packet. The dwc spec says for *_CIC_FULL: "In other words, the TCP, UDP, or ICMP pseudo-header is included in the checksum calculation, and the input frame’s corresponding Checksum field has an all-zero value." The last part of the sentence means to me that the IP header checksum and TCP checksum field must be zero. So if you assume the TCP checksum field is zero, and you just compute the checksum over the relevant data, you get an incorrect result. So the description in the spec does not match the behavior. Since FreeBSD fills in the pseudo header checksum, *_CIC_SEG fits. So I would prefer to do that. I guess a way forwards is: * Patch dwc to make UDP and TCP checksum offloading independent from IP header checksum offloading. This fixes this bug. * Improve the header checksum offloading support in FreeBSD and then the epair interface. That is a feature improvement. * Add support for IPv6 transport checksum offloading to dwc. This is also a feature improvement. We are happy to help and would also like to say thank you for the very cooperative testing to find the root cause of the problem. This also does not happen always...
(In reply to Michael Tuexen from comment #25) Alright, sounds great, thanks again! I'm heading off to visit family for the holiday period now, so I'll be wishing you Happy Holidays, Merry Christmas, and whatever other festivities you might happen to be involved in!
(In reply to Marek Benc from comment #26) Thank you. Enjoy the time with your family. Merry Christmas and a Happy New Year!
A patch is under review D54395.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f8ddf74175c8013268e65b18750e247306fa088a commit f8ddf74175c8013268e65b18750e247306fa088a Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2026-01-16 11:02:53 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2026-01-16 11:02:53 +0000 dwc: improve IPv4 transmit checksum offloading This patch provides two improvements for TCP/IPv4 and UDP/IPv4 transmit checksum offloading: (1) Use *CIC_SEG instead of *CIC_FULL, since FreeBSD always provides a pseudo header checksum. (2) Don't make transmit IPv4 header checksum offloading a prerequisite for TCP/IPv4 or UDP/IPv4 transmit checksum offloading. This is the root cause of PR 291696, since right now the epair interface does not support transmit IPv4 header checksum offloading, but TCP/IPv4 and UDP/IPv4 transmit checksum offloading. PR: 291696 Reviewed by: Timo Voelker Tested by: Marek Benc MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D54395 sys/dev/dwc/dwc1000_dma.c | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=a83f6427c15d0ae24c89c583e72abab9a4669d8f commit a83f6427c15d0ae24c89c583e72abab9a4669d8f Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2026-01-16 11:02:53 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2026-01-21 14:04:21 +0000 dwc: improve IPv4 transmit checksum offloading This patch provides two improvements for TCP/IPv4 and UDP/IPv4 transmit checksum offloading: (1) Use *CIC_SEG instead of *CIC_FULL, since FreeBSD always provides a pseudo header checksum. (2) Don't make transmit IPv4 header checksum offloading a prerequisite for TCP/IPv4 or UDP/IPv4 transmit checksum offloading. This is the root cause of PR 291696, since right now the epair interface does not support transmit IPv4 header checksum offloading, but TCP/IPv4 and UDP/IPv4 transmit checksum offloading. PR: 291696 Reviewed by: Timo Voelker Tested by: Marek Benc Differential Revision: https://reviews.freebsd.org/D54395 (cherry picked from commit f8ddf74175c8013268e65b18750e247306fa088a) sys/dev/dwc/if_dwc.c | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-)
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=decd5ea5d248db46b4cdf46baccc738b28f0713e commit decd5ea5d248db46b4cdf46baccc738b28f0713e Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2026-01-16 11:02:53 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2026-01-21 19:50:29 +0000 dwc: improve IPv4 transmit checksum offloading This patch provides two improvements for TCP/IPv4 and UDP/IPv4 transmit checksum offloading: (1) Use *CIC_SEG instead of *CIC_FULL, since FreeBSD always provides a pseudo header checksum. (2) Don't make transmit IPv4 header checksum offloading a prerequisite for TCP/IPv4 or UDP/IPv4 transmit checksum offloading. This is the root cause of PR 291696, since right now the epair interface does not support transmit IPv4 header checksum offloading, but TCP/IPv4 and UDP/IPv4 transmit checksum offloading. PR: 291696 Reviewed by: Timo Voelker Tested by: Marek Benc Differential Revision: https://reviews.freebsd.org/D54395 (cherry picked from commit f8ddf74175c8013268e65b18750e247306fa088a) sys/dev/dwc/dwc1000_dma.c | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-)