Bug 247636 - Ubuntu 18.04 & 20.04 guest kernel panic on boot with 12.1 bhyve + graphical UEFI
Summary: Ubuntu 18.04 & 20.04 guest kernel panic on boot with 12.1 bhyve + graphical UEFI
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: 12.1-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-29 13:52 UTC by John Hartley
Modified: 2020-07-03 11:06 UTC (History)
1 user (show)

See Also:


Attachments
Screen shot of kernel panic (331.80 KB, image/png)
2020-06-29 13:52 UTC, John Hartley
no flags Details
Nested FreeBSD bhyve with libvirt / VMM (565.34 KB, image/png)
2020-06-30 01:49 UTC, John Hartley
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description John Hartley 2020-06-29 13:52:13 UTC
Created attachment 216037 [details]
Screen shot of kernel panic

BUG:

Setup: Nested virtualisation with: Layer 0 - Ubuntu 20.0 KVM, Layer 1 - FreeBSD 12.1 with bhyve and UEFI, Layer 2 - Ubuntu 18.04 & 20.04 Guest VM

Observered Behaviour:

Starting both Ubuntu 20.04 & 18.04 (Server & Desktop CD-ROMs) VM with bhyve on FreeBSD 12.1 results in VM crashing with kernel panic.
(see attachment for UEFI VNC screen shot)


Expected Behaviour:

Ubuntu CD Install Boot to allow installation onto HD image.

DIAGNOSTICS:

I have done extensive testing of nested virtualisation:
Layer 0 - Ubuntu 20.04 with QEMU / KVM / libvirt
Layer 1 - FreeBSD 12.1 with OVMF Guest running bhyve hypervisor

Layer 2 - Guest Testing:

Layer 2 - FreeBSD 12.1 with UEFI rom + serial console - WORKS
Layer 2 - FreeBSD 12.1 with UEFI rom + graphical boot - WORKS
Layer 2 - Ubuntu 20.04 Server & Desktop with UEFI rom + graphical boot - FAILS
Layer 2 - Ubuntu 18.04 Server & Desktop with UEFI rom + graphical boot = FAILS


Bhyve command line:

# bhyve -c2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,./VM_DSK.img -s 4:0,ahci-cd,./ubuntu-18.04.4-live-server-amd64.iso -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait -s 30,xhci,tablet -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd VM_NAME

I have documented my testing extensively here: https://tips.graphica.com.au/freebsd-virtualisation-with-bhyve/

My configuration included work around to disable "virtual interrupt delivery" as this was required to get FreeBSD nested guest to boot successfully.
See bug report here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168

Happy to do other testing as required.

Thank you for any tips/advise/help/

Regards,


John Hartley.
Comment 1 Peter Grehan freebsd_committer 2020-06-29 22:55:17 UTC
Thanks for testing John.

There is a problem with your config:

  -s 1:0,lpc

With UEFI, this is required to be slot 31 to match the ACPI tables generated by the firmware. Try

  -s 31,lpc


The '-A' is not needed for this case since the UEFI firmware will be creating the ACPI tables and not bhyve itself, but this is harmless.
Comment 2 John Hartley 2020-06-30 01:46:10 UTC
Hi Peter,

thanks for reviewing bug.

I do have some good news as I tested with Ubuntu 16.04 Desktop and this boots up ok via UEFI.

I have also retested with Ubuntu 20.04 as per your feedback:

# bhyve -c2 -m 2048 -H -s 0:0,hostbridge -s 31,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,./drum-hd1-01.img -s 4:0,ahci-cd,./ubuntu-20.04-desktop-amd64.iso -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,vga=io,wait -s 30,xhci,tablet -l com1,/dev/nmdm0A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd VM_NAME

and still get kernel panic on boot.

Cheers,


John Hartley.
Comment 3 John Hartley 2020-06-30 01:49:14 UTC
Created attachment 216050 [details]
Nested FreeBSD bhyve with libvirt / VMM

Here is screen shot with Ubuntu 16.04 booted successfully from CD
Comment 4 Peter Grehan freebsd_committer 2020-07-01 14:12:06 UTC
On an Ubuntu 20.04 host system (Thinkpad, Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz) and a FreeBSD 12.1-RELEASE/amd64 L1 guest, I was able to boot 2xvCPU L2
live-CD guests:

Ubuntu 16.04
Ubuntu 18.04
Ubuntu 20.04 (with caveat)

With the 20.04 L2 guest, I was hitting the MSI issue fixed in r360399 and not in 12.1. However, a workaround is to use 1 vCPU for the guest, and this worked.

With a recent snapshot for the L1 guest(FreeBSD-13.0-CURRENT-amd64-20200625-r362596-disc1.iso) a 2 vCPU 20.04 L2 guest was able to be booted.
Comment 5 John Hartley 2020-07-02 08:04:37 UTC
Hi Peter,

thanks again for looking into this.

I found that while I can boot Ubuntu 16.04 from CD it crashed once I had installed on HD and rebooted. Ubuntu 16.04 appears to have erratic grub/boot installation occurring under bhyve for some reason (tested with 16.04.2 & 16.04.4).

I retested with 20.04 as per your advise:

# bhyve -c 1 -m 2048 -H -w -s 0:0,hostbridge -s 2:0,e1000,tap0 -s 3:0,ahci-hd,./drum-hd1-01.img -s 4:0,ahci-cd,./ubuntu-20.04-desktop-amd64.iso -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,vga=io,wait -s 30,xhci,tablet -s 31,lpc -l com1,/dev/nmdm0A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd drum

And it boots, install and runs on reboot.

So given that this is fixed with 13.0 CURRENT I would say issue is fixed.

Cheers,

John.
Comment 6 Peter Grehan freebsd_committer 2020-07-03 09:00:55 UTC
I'll look at the issue with 16.*, though it will be low priority given that release is heading to out of support, and the following 2 LTS releases are working ok.
Comment 7 John Hartley 2020-07-03 11:06:59 UTC
Hi Peter,

yes agree, I only went back to 16.x series as I had problems with 18.04 & 20.04.

So I am sure you have much better things to do.

Thanks for assisting.

I have raised defect/feature request on libvirt to help with improving bhyve support: https://gitlab.com/libvirt/libvirt/-/issues/45

Cheers,

John.