Bug 247636 - Ubuntu 18.04 & 20.04 guest kernel panic on boot with 12.1 bhyve + graphical UEFI
Summary: Ubuntu 18.04 & 20.04 guest kernel panic on boot with 12.1 bhyve + graphical UEFI
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: 12.1-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-29 13:52 UTC by John Hartley
Modified: 2020-07-03 11:06 UTC (History)
1 user (show)

See Also:

Attachments
Screen shot of kernel panic (331.80 KB, image/png)
2020-06-29 13:52 UTC, John Hartley 		no flags Details
Nested FreeBSD bhyve with libvirt / VMM (565.34 KB, image/png)
2020-06-30 01:49 UTC, John Hartley 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description John Hartley 2020-06-29 13:52:13 UTC 
Created attachment 216037 [details]
Screen shot of kernel panic

BUG:

Setup: Nested virtualisation with: Layer 0 - Ubuntu 20.0 KVM, Layer 1 - FreeBSD 12.1 with bhyve and UEFI, Layer 2 - Ubuntu 18.04 & 20.04 Guest VM

Observered Behaviour:

Starting both Ubuntu 20.04 & 18.04 (Server & Desktop CD-ROMs) VM with bhyve on FreeBSD 12.1 results in VM crashing with kernel panic.
(see attachment for UEFI VNC screen shot)


Expected Behaviour:

Ubuntu CD Install Boot to allow installation onto HD image.

DIAGNOSTICS:

I have done extensive testing of nested virtualisation:
Layer 0 - Ubuntu 20.04 with QEMU / KVM / libvirt
Layer 1 - FreeBSD 12.1 with OVMF Guest running bhyve hypervisor

Layer 2 - Guest Testing:

Layer 2 - FreeBSD 12.1 with UEFI rom + serial console - WORKS
Layer 2 - FreeBSD 12.1 with UEFI rom + graphical boot - WORKS
Layer 2 - Ubuntu 20.04 Server & Desktop with UEFI rom + graphical boot - FAILS
Layer 2 - Ubuntu 18.04 Server & Desktop with UEFI rom + graphical boot = FAILS


Bhyve command line:

# bhyve -c2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,./VM_DSK.img -s 4:0,ahci-cd,./ubuntu-18.04.4-live-server-amd64.iso -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait -s 30,xhci,tablet -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd VM_NAME

I have documented my testing extensively here: https://tips.graphica.com.au/freebsd-virtualisation-with-bhyve/

My configuration included work around to disable "virtual interrupt delivery" as this was required to get FreeBSD nested guest to boot successfully.
See bug report here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168

Happy to do other testing as required.

Thank you for any tips/advise/help/

Regards,


John Hartley.
Comment 1 Peter Grehan freebsd_committer freebsd_triage 2020-06-29 22:55:17 UTC 
Thanks for testing John.

There is a problem with your config:

  -s 1:0,lpc

With UEFI, this is required to be slot 31 to match the ACPI tables generated by the firmware. Try

  -s 31,lpc


The '-A' is not needed for this case since the UEFI firmware will be creating the ACPI tables and not bhyve itself, but this is harmless.
Comment 2 John Hartley 2020-06-30 01:46:10 UTC 
Hi Peter,

thanks for reviewing bug.

I do have some good news as I tested with Ubuntu 16.04 Desktop and this boots up ok via UEFI.

I have also retested with Ubuntu 20.04 as per your feedback:

# bhyve -c2 -m 2048 -H -s 0:0,hostbridge -s 31,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,./drum-hd1-01.img -s 4:0,ahci-cd,./ubuntu-20.04-desktop-amd64.iso -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,vga=io,wait -s 30,xhci,tablet -l com1,/dev/nmdm0A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd VM_NAME

and still get kernel panic on boot.

Cheers,


John Hartley.
Comment 3 John Hartley 2020-06-30 01:49:14 UTC 
Created attachment 216050 [details]
Nested FreeBSD bhyve with libvirt / VMM

Here is screen shot with Ubuntu 16.04 booted successfully from CD
Comment 4 Peter Grehan freebsd_committer freebsd_triage 2020-07-01 14:12:06 UTC 
On an Ubuntu 20.04 host system (Thinkpad, Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz) and a FreeBSD 12.1-RELEASE/amd64 L1 guest, I was able to boot 2xvCPU L2
live-CD guests:

Ubuntu 16.04
Ubuntu 18.04
Ubuntu 20.04 (with caveat)

With the 20.04 L2 guest, I was hitting the MSI issue fixed in r360399 and not in 12.1. However, a workaround is to use 1 vCPU for the guest, and this worked.

With a recent snapshot for the L1 guest(FreeBSD-13.0-CURRENT-amd64-20200625-r362596-disc1.iso) a 2 vCPU 20.04 L2 guest was able to be booted.
Comment 5 John Hartley 2020-07-02 08:04:37 UTC 
Hi Peter,

thanks again for looking into this.

I found that while I can boot Ubuntu 16.04 from CD it crashed once I had installed on HD and rebooted. Ubuntu 16.04 appears to have erratic grub/boot installation occurring under bhyve for some reason (tested with 16.04.2 & 16.04.4).

I retested with 20.04 as per your advise:

# bhyve -c 1 -m 2048 -H -w -s 0:0,hostbridge -s 2:0,e1000,tap0 -s 3:0,ahci-hd,./drum-hd1-01.img -s 4:0,ahci-cd,./ubuntu-20.04-desktop-amd64.iso -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,vga=io,wait -s 30,xhci,tablet -s 31,lpc -l com1,/dev/nmdm0A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd drum

And it boots, install and runs on reboot.

So given that this is fixed with 13.0 CURRENT I would say issue is fixed.

Cheers,

John.
Comment 6 Peter Grehan freebsd_committer freebsd_triage 2020-07-03 09:00:55 UTC 
I'll look at the issue with 16.*, though it will be low priority given that release is heading to out of support, and the following 2 LTS releases are working ok.
Comment 7 John Hartley 2020-07-03 11:06:59 UTC 
Hi Peter,

yes agree, I only went back to 16.x series as I had problems with 18.04 & 20.04.

So I am sure you have much better things to do.

Thanks for assisting.

I have raised defect/feature request on libvirt to help with improving bhyve support: https://gitlab.com/libvirt/libvirt/-/issues/45

Cheers,

John.