This is a more specific report that follows PR 273372 [1] and a forum post [2]. Simultaneous use of vnet and byhve will cause networking networking to fail if both are used on PCI hardware devices. Setup: 2 SR-IOV VFs created from a capable card (the vendor doesn't matter, I've tried Intel, Chelsio, and Mellanox). Fail Scenario: 1 passed through to a bhyve guest (OS doesn't matter). 1 setup with vnet for a jail. About 90 seconds after both are running, networking dies. Only console access is possible. Non-Fail Scenario: 2 byhve guests setup. 1 VF passed to each. One of the guests is FreeBSD, and in that guest give the network device to a jail and set it up with vnet. No problems. Also Non-Fail: Any number of bhyve passthrough devices OR any number of vnet jails using VFs. Alone they work just fine. It's only when together there is an issue. There appears to be a conflict between byhve and vnet when both are in use "on the same level," but if you separate them with a layer virtualization, no issues arise. 1. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273372 2. https://forums.freebsd.org/threads/pci-passthrough-of-vfs-crash-host-card-if-jails-bhyve-in-use.92409/
I listened the the bhyve production user call [1] and noticed people still speculating that bhyve+vnet SR-IOV issues might be PCI card specific. To clarify, I can reproduce this issue 100% of the time on Chelsio, Intel, and Mellanox NICs. I have reproduced it on two Supermicro motherboards (X12STH-F, X11SSM-F) with 3 different Intel CPUs (Xeon E-2388G, Xeon E-2324G, Xeon E3-1275 v6). So if hardware is a consideration, the only common thread for me is Supermicro motherboards and Intel Xeon CPUs. It's worth noting again, if bhyve alone is used for passthrough, I have no issues at all. It's only when the on-metal FreeBSD host passes VFs to bhyve AND vnet jails that I observe issues. I currently have 8 network VFs and 1 iGPU passed to 4 bhyve instances with no problems. My workaround is to use vnet+epair for any jail networking in the host (instead of vnet+VF). 1. Bhyve Production User Call 2024-04-25 https://www.youtube.com/watch?v=gGzpLLJTHS0