Created attachment 216037 [details]
Screen shot of kernel panic
Setup: Nested virtualisation with: Layer 0 - Ubuntu 20.0 KVM, Layer 1 - FreeBSD 12.1 with bhyve and UEFI, Layer 2 - Ubuntu 18.04 & 20.04 Guest VM
Starting both Ubuntu 20.04 & 18.04 (Server & Desktop CD-ROMs) VM with bhyve on FreeBSD 12.1 results in VM crashing with kernel panic.
(see attachment for UEFI VNC screen shot)
Ubuntu CD Install Boot to allow installation onto HD image.
I have done extensive testing of nested virtualisation:
Layer 0 - Ubuntu 20.04 with QEMU / KVM / libvirt
Layer 1 - FreeBSD 12.1 with OVMF Guest running bhyve hypervisor
Layer 2 - Guest Testing:
Layer 2 - FreeBSD 12.1 with UEFI rom + serial console - WORKS
Layer 2 - FreeBSD 12.1 with UEFI rom + graphical boot - WORKS
Layer 2 - Ubuntu 20.04 Server & Desktop with UEFI rom + graphical boot - FAILS
Layer 2 - Ubuntu 18.04 Server & Desktop with UEFI rom + graphical boot = FAILS
Bhyve command line:
# bhyve -c2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,./VM_DSK.img -s 4:0,ahci-cd,./ubuntu-18.04.4-live-server-amd64.iso -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait -s 30,xhci,tablet -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd VM_NAME
I have documented my testing extensively here: https://tips.graphica.com.au/freebsd-virtualisation-with-bhyve/
My configuration included work around to disable "virtual interrupt delivery" as this was required to get FreeBSD nested guest to boot successfully.
See bug report here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168
Happy to do other testing as required.
Thank you for any tips/advise/help/
Thanks for testing John.
There is a problem with your config:
With UEFI, this is required to be slot 31 to match the ACPI tables generated by the firmware. Try
The '-A' is not needed for this case since the UEFI firmware will be creating the ACPI tables and not bhyve itself, but this is harmless.
thanks for reviewing bug.
I do have some good news as I tested with Ubuntu 16.04 Desktop and this boots up ok via UEFI.
I have also retested with Ubuntu 20.04 as per your feedback:
# bhyve -c2 -m 2048 -H -s 0:0,hostbridge -s 31,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,./drum-hd1-01.img -s 4:0,ahci-cd,./ubuntu-20.04-desktop-amd64.iso -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,vga=io,wait -s 30,xhci,tablet -l com1,/dev/nmdm0A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd VM_NAME
and still get kernel panic on boot.
Created attachment 216050 [details]
Nested FreeBSD bhyve with libvirt / VMM
Here is screen shot with Ubuntu 16.04 booted successfully from CD
On an Ubuntu 20.04 host system (Thinkpad, Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz) and a FreeBSD 12.1-RELEASE/amd64 L1 guest, I was able to boot 2xvCPU L2
Ubuntu 20.04 (with caveat)
With the 20.04 L2 guest, I was hitting the MSI issue fixed in r360399 and not in 12.1. However, a workaround is to use 1 vCPU for the guest, and this worked.
With a recent snapshot for the L1 guest(FreeBSD-13.0-CURRENT-amd64-20200625-r362596-disc1.iso) a 2 vCPU 20.04 L2 guest was able to be booted.
thanks again for looking into this.
I found that while I can boot Ubuntu 16.04 from CD it crashed once I had installed on HD and rebooted. Ubuntu 16.04 appears to have erratic grub/boot installation occurring under bhyve for some reason (tested with 16.04.2 & 16.04.4).
I retested with 20.04 as per your advise:
# bhyve -c 1 -m 2048 -H -w -s 0:0,hostbridge -s 2:0,e1000,tap0 -s 3:0,ahci-hd,./drum-hd1-01.img -s 4:0,ahci-cd,./ubuntu-20.04-desktop-amd64.iso -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,vga=io,wait -s 30,xhci,tablet -s 31,lpc -l com1,/dev/nmdm0A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd drum
And it boots, install and runs on reboot.
So given that this is fixed with 13.0 CURRENT I would say issue is fixed.
I'll look at the issue with 16.*, though it will be low priority given that release is heading to out of support, and the following 2 LTS releases are working ok.
yes agree, I only went back to 16.x series as I had problems with 18.04 & 20.04.
So I am sure you have much better things to do.
Thanks for assisting.
I have raised defect/feature request on libvirt to help with improving bhyve support: https://gitlab.com/libvirt/libvirt/-/issues/45