Bug 278245 - sendfile to bhyve VM on same host fails with MEXTPG
Summary: sendfile to bhyve VM on same host fails with MEXTPG
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 14.0-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-04-08 00:35 UTC by djk-freebsdbugs
Modified: 2024-04-08 17:34 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description djk-freebsdbugs 2024-04-08 00:35:27 UTC
In debugging an issue with an nginx reverse proxy, I found an issue with sendfile, MEXTPG, and one of more of bridge, tap, and bhyve.  I am on FreeBSD 14-CURRENT-p6.

To replicate this issue, I used the following Python test harness:

import socket    
HOST = 'target.host'
PORT = 50007    
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s, open("blah.txt", "rb") as f:
    s.connect((HOST, PORT))
    s.sendfile(f)

In this case, target.host is a Debian bhyve VM running on the same host.  On the target, something like 'nc -l -p 50007' can be run to receive the data.

On the networking side, I've got:

# ifconfig em0
em0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,HWSTATS>
        ether 6c:4b:90:1f:e9:a8
        inet 192.168.11.15 netmask 0xffffff00 broadcast 192.168.11.255
        inet6 fe80::6e4b:90ff:fe1f:e9a8%em0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
# ifconfig vm-public
vm-public: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 3a:47:10:77:5b:4d
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000000
        member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000000
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge vm-switch viid-4c918@
        nd6 options=9<PERFORMNUD,IFDISABLED>


When MEXTPG is enabled on the em0 interface, the test fails when sending a file larger than 224 bytes.  When MEXTPG is enabled the test is successful.  The test is successful when sending to a VM on a different host, so the bhyve VM is a requirement to replicate the issue.

https://www.reddit.com/r/freebsd/comments/1bxebaz/more_vm_networking_weirdness/ has a lot more detail and some narrowing down of the problem.