Bug 262936

Summary:	bhyve: win10 host fails to restart
Product:	Base System	Reporter:	bergerkos
Component:	bhyve	Assignee:	freebsd-virtualization (Nobody) <virtualization>
Status:	New ---
Severity:	Affects Only Me	CC:	bergerkos, swills
Priority:	---
Version:	13.1-RELEASE
Hardware:	amd64
OS:	Any

Description bergerkos 2022-03-30 10:44:42 UTC

THe system is 13.0-STABLE FreeBSD 13.0-STABLE #0 stable/13-n249491-be8e341777c
(installed from one of the snapshots)

My Windows 10 VM randomly fails to boot (bhyve console hangs dead), after which it cannot be started again without the machine reboot.
I have passthrough network and USB cards, so may be related, maybe not.

What happens in detail:
SOMETIMES boot console hangs dead at UEFI booting stage, after which `bhyvectl --vm $my_vm --force-reset / --force-poweroff` commands don't work at all. Boot console itself hangs and so does the corresponding 'bhyve' process. The `bhyve --vm $myvm --destroy` command DOES something: ls /dev/vmm is empty, VM destroyed, but the `bhyve` process is still hanging there. It disappears upon user logout/login (desktop). 

But when, after these manipulations, I try to start the VM anew it gives off this error:
bhyve: Failed to map MSI-X table BAR on 12/0/0: Device busy
bhyve: failed to initialize BARs for PCI 12/0/0
device emulation initialization error: Device busy

Here is my command:
bhyve -S -c sockets=1,cores=4,threads=2 -m 8G -H -w \
  -s 0,hostbridge \
  -s 3,passthru,8/0/0 \
  -s 4,virtio-blk,/bhyve/win-alt.img,sectorsize=512/4096 \
  -s 6,hda,rec=/dev/dsp,play=/dev/dsp \
  -s 8,passthru,12/0/0 \
  -s 29,fbuf,tcp=0.0.0.0:5900,w=1600,h=900,wait \
  -s 30,xhci,tablet \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  windows-10

Is there anything else that can be "reset" there without a complete machine reboot?

Comment 1 bergerkos 2022-03-30 13:00:49 UTC

NOTE: this started happening only since I migrated to STABLE.

Comment 2 bergerkos 2022-03-30 23:52:07 UTC

Tried this on BETA3.
Haven't experienced any boot crashes yet, but neither can I restart my VM after even normal shutdown.
On 13.0-RELEASE-p7 I was able to restart my VM after normal shutodwn only after bhyvectl --vm windows-10 --destroy. That was the problem on one of my machines which I learned to live with.

Now this doesn't work anymore. When trying to restart after --destroy, I get this same error as mentioned earlier:
bhyve: Failed to map MSI-X table BAR on 12/0/0: Device busy
bhyve: failed to initialize BARs for PCI 12/0/0
device emulation initialization error: Device busy

Comment 3 bergerkos 2022-04-16 10:23:10 UTC

Still the same on 13.1-RC3. So I'm changing it to 13.1-RELEASE because it's not very likely to change until then, is it.

Comment 4 bergerkos 2022-04-16 10:25:53 UTC

This evidently has to do with PASSTHRU, so maybe I should reflect it in the title?

Comment 5 bergerkos 2022-04-16 13:56:13 UTC

OK, I solved this by removing the USB 3.0 PCIe card PASSTHRU from my bhyve config, the one mentioned in the error message:
bhyve: Failed to map MSI-X table BAR on 12/0/0: Device busy
bhyve: failed to initialize BARs for PCI 12/0/0
device emulation initialization error: Device busy

I tried instead adding PASSTHRU for motherboard-based VIA VL805/806 xHCI USB 3.0 Controller instead. And it works!!! I don't even need the `bhyvectl --vm --restart` hack any more to restart the VM that was shut down, as I had to.

The irony is, this same VIA controller that now works fine caused me trouble with bhyve some 2 years ago (PASSTHRU would not attach; when it does, the machine spontaneously reboots). 

Still not clear what's causing the original problem reported here... For reference, the problematic device is
xhci0@pci0:12:0:0:	class=0x0c0330 rev=0x03 hdr=0x00 vendor=0x1912 device=0x0014 subvendor=0x1912 subdevice=0x0014
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720201 USB 3.0 Host Controller'
It USED to work fine with PASSTHRU, that's why I originally bought it.

THe one that works fine is:
ppt1@pci0:11:0:0:	class=0x0c0330 rev=0x01 hdr=0x00 vendor=0x1106 device=0x3483 subvendor=0x1106 subdevice=0x3483
    vendor     = 'VIA Technologies, Inc.'
    device     = 'VL805/806 xHCI USB 3.0 Controller'

Comment 6 bergerkos 2022-04-24 16:48:27 UTC

Well... difficult to tell now whether the originally reported problem was due to bhyve or Windows 10 Pro itself.

Still, I had this hangup again. 
Boot hangs dead early at UEFI stage, no response, no restart options work, with "bhyve" process hanging dead unresponsive. 
Only logging the process owner user out removed the hung process. But restarting bhyve ended up the same. Then the computer was rebooted, after which starting bhyve VM launched Windows Repairing attempt...

How can I make sure whether this is bhyve problem or internal Windows problem?