In debugging an issue with an nginx reverse proxy, I found an issue with sendfile, MEXTPG, and one of more of bridge, tap, and bhyve. I am on FreeBSD 14-CURRENT-p6. To replicate this issue, I used the following Python test harness: import socket HOST = 'target.host' PORT = 50007 with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s, open("blah.txt", "rb") as f: s.connect((HOST, PORT)) s.sendfile(f) In this case, target.host is a Debian bhyve VM running on the same host. On the target, something like 'nc -l -p 50007' can be run to receive the data. On the networking side, I've got: # ifconfig em0 em0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,HWSTATS> ether 6c:4b:90:1f:e9:a8 inet 192.168.11.15 netmask 0xffffff00 broadcast 192.168.11.255 inet6 fe80::6e4b:90ff:fe1f:e9a8%em0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> # ifconfig vm-public vm-public: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=0 ether 3a:47:10:77:5b:4d id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 7 priority 128 path cost 2000000 member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 6 priority 128 path cost 2000000 member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 5 priority 128 path cost 2000000 member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 4 priority 128 path cost 2000000 member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 1 priority 128 path cost 20000 groups: bridge vm-switch viid-4c918@ nd6 options=9<PERFORMNUD,IFDISABLED> When MEXTPG is enabled on the em0 interface, the test fails when sending a file larger than 224 bytes. When MEXTPG is enabled the test is successful. The test is successful when sending to a VM on a different host, so the bhyve VM is a requirement to replicate the issue. https://www.reddit.com/r/freebsd/comments/1bxebaz/more_vm_networking_weirdness/ has a lot more detail and some narrowing down of the problem.
Sorry it took some time for this to get attention. if_bridge should mask MEXTPG if some members don't support it: https://reviews.freebsd.org/D47294 This is basically equivalent to your workaround. I also have a patch which enables MEXTPG for tap interfaces: https://reviews.freebsd.org/D47295
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=2bbfbf80d3bb828ac782c2d990a1fba0eb51e45a commit 2bbfbf80d3bb828ac782c2d990a1fba0eb51e45a Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-10-28 13:52:07 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-10-28 15:14:21 +0000 if_bridge: Mask MEXTPG if some members don't support it Similar to how the network stack needs to use mb_unmapped_to_ext() to convert mbufs before passing them to an unsupported driver, if_bridge needs to avoid passing M_EXTPG mbufs to interfaces that don't support them. Thus, clear IFCAP_MEXTPG on the bridge if any member interfaces don't handle unmapped mbufs. PR: 278245 Reviewed by: jhb, gallatin MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D47294 sys/net/if_bridge.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=01a3c17d18bafbf4e76e9d77e8862525b50602e9 commit 01a3c17d18bafbf4e76e9d77e8862525b50602e9 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-10-28 13:52:07 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-11-11 14:02:17 +0000 if_bridge: Mask MEXTPG if some members don't support it Similar to how the network stack needs to use mb_unmapped_to_ext() to convert mbufs before passing them to an unsupported driver, if_bridge needs to avoid passing M_EXTPG mbufs to interfaces that don't support them. Thus, clear IFCAP_MEXTPG on the bridge if any member interfaces don't handle unmapped mbufs. PR: 278245 Reviewed by: jhb, gallatin MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D47294 (cherry picked from commit 2bbfbf80d3bb828ac782c2d990a1fba0eb51e45a) sys/net/if_bridge.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3b7acf9a3b2428f86291e48298ee9bae8f1276c7 commit 3b7acf9a3b2428f86291e48298ee9bae8f1276c7 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-10-28 13:52:07 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-11-11 14:02:21 +0000 if_bridge: Mask MEXTPG if some members don't support it Similar to how the network stack needs to use mb_unmapped_to_ext() to convert mbufs before passing them to an unsupported driver, if_bridge needs to avoid passing M_EXTPG mbufs to interfaces that don't support them. Thus, clear IFCAP_MEXTPG on the bridge if any member interfaces don't handle unmapped mbufs. PR: 278245 Reviewed by: jhb, gallatin MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D47294 (cherry picked from commit 2bbfbf80d3bb828ac782c2d990a1fba0eb51e45a) sys/net/if_bridge.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
We should consider the patch for inclusion in 14.2, it's quite low-risk.
A commit in branch releng/14.2 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=d86dcfc6486f7b5b9fcbc5e5ad457b0d22e1c5f6 commit d86dcfc6486f7b5b9fcbc5e5ad457b0d22e1c5f6 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-10-28 13:52:07 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-11-11 18:46:06 +0000 if_bridge: Mask MEXTPG if some members don't support it Similar to how the network stack needs to use mb_unmapped_to_ext() to convert mbufs before passing them to an unsupported driver, if_bridge needs to avoid passing M_EXTPG mbufs to interfaces that don't support them. Thus, clear IFCAP_MEXTPG on the bridge if any member interfaces don't handle unmapped mbufs. Approved by: re (kib) PR: 278245 Reviewed by: jhb, gallatin MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D47294 (cherry picked from commit 2bbfbf80d3bb828ac782c2d990a1fba0eb51e45a) (cherry picked from commit 01a3c17d18bafbf4e76e9d77e8862525b50602e9) sys/net/if_bridge.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Thank you for the report.