Bug 246168 - Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Summary: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: 12.1-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-05-04 08:44 UTC by John Hartley
Modified: 2021-05-18 01:56 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description John Hartley 2020-05-04 08:44:34 UTC
BUG:

Starting FreeBSD Layer 2 bhyve Guest within Layer 1 FreeBSD VM Host on Layer 0 Ubuntu 20.04 KVM / QEMU Host result in Layer 1 Guest / Host Pausing with "Emulation Failure"

TESTING:

My test scenario is nested virtualisation:
Layer 0 - Ubuntu 20.04 Host
Layer 1 - FreeBSD 12.1 with OVMF + bhyve hypervisor Guest/Host
Layer 2 - FreeBSD 12.1 guest

Layer 0 Host is: Ubuntu 20.04 LTS KVM / QEMU / libvirt

<<START QEMU VERSION>>
$ virsh -c qemu:///system version --daemon
Compiled against library: libvirt 6.0.0
Using library: libvirt 6.0.0
Using API: QEMU 6.0.0
Running hypervisor: QEMU 4.2.0
Running against daemon: 6.0.0
<<END QEMU VERSION>

<<START Intel VMX Support & Nesting Enabled>>
$ cat /proc/cpuinfo | grep -c vmx
64
$ cat /sys/module/kvm_intel/parameters/nested
Y
<<END Intel VMS>>



Layer 1 Guest / Host is: FreeBSD Q35 v4.2 with OVMF:

Pass Host VMX support to Layer 1 Guest via <cpu mode='host-model>

<<LIBVIRT CONFIG SNIPPET>>
...
...
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/home/USER/swarm.bhyve.freebsd/OVMF_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'/>
...
...
<END LIBVIRT CONFIG SNIPPET>>

Checked that Layer 1 - FreeBSD Quest / Host has VMX feature available:

<<LAYER 1 - FreeBSD CPU Features>>
# uname -a
FreeBSD swarm.DOMAIN.HERE 12.1-RELEASE FreeBSD 12.1-RELEASE GENERIC  amd64

# grep Features /var/run/dmesg.boot 
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
  Structured Extended Features2=0x4<UMIP>
  Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
<<END LAYER 1 - FreeBSD CPU Features>

On Layer 1 FreeBSD Guest / Host start up the Layer 2 guest..

<<START LAYER 2 GUEST START>>
# ls
FreeBSD-11.2-RELEASE-amd64-bootonly.iso	FreeBSD-12.1-RELEASE-amd64-dvd1.iso	bee-hd1-01.img
# /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee
<<END LAYER 2 GUEST START>>

Result is that Layer 1 - FreeBSD Host guest "paused".

To Layer 1 machines freezes I cannot get any further diagnostics from this machine, so I run tail on libvirt log from Layer 0 - Ubuntu Host

<<LAYER 0 LOG TAIL>>
char device redirected to /dev/pts/29 (label charserial0)
2020-05-04T06:09:15.310474Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.310531Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.312533Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.312548Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.313828Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.313841Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.315185Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.315201Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 00000000 00008000 DPL=0 <hiword>
CS =0000 00000000 00000000 00008000 DPL=0 <hiword>
SS =0000 00000000 00000000 00008000 DPL=0 <hiword>
DS =0000 00000000 00000000 00008000 DPL=0 <hiword>
FS =0000 00000000 00000000 00008000 DPL=0 <hiword>
GS =0000 00000000 00000000 00008000 DPL=0 <hiword>
LDT=0000 00000000 00000000 00008000 DPL=0 <hiword>
TR =0000 00000000 00000000 00008000 DPL=0 <hiword>
GDT=     0000000000000000 00000000
IDT=     0000000000000000 00000000
CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2020-05-04T06:35:39.186799Z qemu-system-x86_64: terminating on signal 15 from pid 2155 (/usr/sbin/libvirtd)
2020-05-04 06:35:39.386+0000: shutting down, reason=destroyed
<<END LAYER 0 LOG TAIL>>


I am reporting this bug here as result is very similar to that seen with QEMU seabios failure reported here: https://bugs.launchpad.net/qemu/+bug/1866870

However in this case my VM Layer 1 VM is using OVMF.

NOTE 1: I have also tested with Q35 v3.1 and 2.12 and get the same result.
NOTE 2: Due to bug in FreeBSD networking code, I had to compile custom kernel with "netmap driver disabled".  This is known bug in FreeBSD that I have reported separately.
Comment 1 John Hartley 2020-05-04 08:51:50 UTC
NOTE:

I have cross posted this Bug Report Ubuntu Bug Tracking: https://bugs.launchpad.net/qemu/+bug/1876678

Also note I have done extensive testing of Ubuntu 20.04 Nested virtualisation with just Ubuntu hosts and OVMF and the nested virtualisation runs correctly, so problem is specific to using FreeBSD / bhyve guest / host.
Comment 2 Rodney W. Grimes freebsd_committer 2020-05-04 14:25:18 UTC
FreeBSD Bhyve does *not* support nested virtualization as a host, your operating beyond what has been implemented.
Comment 3 John Hartley 2020-05-05 03:03:15 UTC
Hi Rodney,

thanks for looking at this bug report.

Does your statement, "FreeBSD Bhyve does *not* support nested virtualization as a host" relate to this nested topology ?

Layer 0 - FreeBSD with bhyve hypervisor (on Bare Metal with Intel VT-x VMX support)
Layer 1 - FreeBSD with bhyve hypervisor (ie Guest of Layer 0, Host to Layer 2)
Layer 2 - FreeBSD or other OS Guest

I ask as bhyve wiki: https://wiki.freebsd.org/bhyve#Q:_Can_I_run_multiple_bhyve_hosts_under_VMware_nested_VT-x_EPT.3F

Indicates that there should be ability to run nested topology as per my example:

Layer 0 - VMWare or KVM with VT-x EPT
Layer 1 - FreeBSD with bhyve hypervisor (ie Guest of Layer 0, Host to Layer 2)
Layer 2 - FreeBSD or other OS Guest

I am trying to get this model up and running, as I am wanting to test metrics collection from Layer 1 FreeBSD guest/host.

Given scarcity of information I know that this is little off the beaten track, but I did find this example: https://github.com/Suhoy95/SNE-reports-2019-public/tree/4-LIA-2-xen-kvm which indicated that it had go similar nested topology going, but with Ubuntu 18.04 as Layer 0 rather than Ubuntu 20.04 as in my case.

Thanks you.

John Hartley.
Comment 4 Peter Grehan freebsd_committer 2020-05-05 03:25:30 UTC
Yours is a valid configuration.

This was reported a long while back and discussed extensively in bug 203994, where the fix was to use a kernel version >= 4.10.

So, this appears to be a regression.
Comment 5 Rodney W. Grimes freebsd_committer 2020-05-05 03:48:21 UTC
(In reply to Peter Grehan from comment #4)
How is running FreeBSD at layer 2, inside of a layer 1 FreeBSD (host) a valid configuration, I thought we had no support for nesting.  Does some magic happen because the layer 0/bare metal host is running KVM?

Now, running FreeBSD at layer 1, inside a layer 0 KVM, I'll agree is valid.
Comment 6 Rodney W. Grimes freebsd_committer 2020-05-05 03:53:04 UTC
(In reply to Rodney W. Grimes from comment #5)
Never mind, I see why, layer 2 is NOT running bhyve, it is just running FreeBSD
Comment 7 John Hartley 2020-05-05 04:50:35 UTC
Hi Peter & Co,

I reviewed the prior bug report and FYI here is some additional information on my machine:

<<START LAYER 1 - FreeBSD CPU Report from dmesg.boot>>
...
...
CPU: Intel Core Processor (Broadwell, IBRS) (2600.09-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306d2  Family=0x6  Model=0x3d  Stepping=2
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
  Structured Extended Features2=0x4<UMIP>
  Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
  IA32_ARCH_CAPS=0x8<SKIP_L1DFL_VME>
  AMD Extended Feature Extensions ID EBX=0x1001000
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Hypervisor: Origin = "KVMKVMKVM"
...
...
<END LAYER 1 - dimes.log>>


I also did cpuctl load and here are result from cpucontrol

(I am not sure how to interpret these or if provided vectors are still valid with FreeBSD 12.1)

<<START CPUCTL>>
# kldload cpuctl
# cpucontrol -m 0x480 /dev/cpuctl0
MSR 0x480: 0x00d81000 0x11e57ed0
# cpucontrol -m 0x482 /dev/cpuctl0
MSR 0x482: 0xfff9fffe 0x0401e172
# cpucontrol -m 0x48b /dev/cpuctl0
MSR 0x48b: 0x00037bff 0x00000000
# cpucontrol -m 0x48c /dev/cpuctl0
MSR 0x48c: 0x00000f01 0x06334041
<<END CPUCTL>>

I also saw that in the original but, it appear that issue was that the bhyve KLM was not loading.

I my case the vmm module is loading but I am getting runtime failure...

Any tips you can provide to help with diagnosis would be appreciated and I can chase upstream if needed, if this is KVM regression.

Thanks.


John Hartley
Comment 8 Peter Grehan freebsd_committer 2020-05-07 08:43:18 UTC
I was able to get this working fine, and the cause of the problem is clear looking back at the posted logs.

<<START LAYER 2 GUEST START>>
# ls
FreeBSD-11.2-RELEASE-amd64-bootonly.iso	FreeBSD-12.1-RELEASE-amd64-dvd1.iso	bee-hd1-01.img
# /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee

 bhyve has to be started with either a UEFI ROM image, or using a separate loader (bhyveload/grub-bhyve). This wasn't done in this case, which will result in registers partly set to power-on state, which will result in a triple-fault or some other error.

This can be seen in the register dump from KVM, where all the GP registers are zero including the RIP.

The fix to run a FreeBSD guest is to use bhyveload before running bhyve. For the example given, this would be

# /usr/sbin/bhyveload -m 2048 -d ./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee

...

and then run the bhyve command.
Comment 9 John Hartley 2020-05-09 07:42:19 UTC
(In reply to Peter Grehan from comment #8)

Hi Peter,

thanks for reviewing.

I have done further testing:

A. Running bhyveload first as per your feedback

Result:

Brings up FreeBSD Loader and then get to "/dev/entropy not found message and closes machine


<<BHYVECTL GET-STATS>>
# bhyvectl --vm=bee --get-stats > bee-vm-stats-01.txt
# cat bee-vm-stats-01.txt 
vcpu0 stats:
number of times in/out was intercepted  	0
number of times cpuid was intercepted   	0
vm exits due to nested page fault       	0
vm exits for instruction emulation      	0
number of vm exits for unknown reason   	0
number of times astpending at exit      	0
number of times idle requested at exit  	0
number of vm exits handled in userspace 	0
number of times rendezvous pending at exit	0
number of vm exits due to exceptions    	0
number of NMIs delivered to vcpu        	0
number of ExtINTs delivered to vcpu     	0
Resident memory                         	0
Wired memory                            	0
vcpu total runtime                      	0
EOI without any in-service interrupt    	0
error interrupts generated by vlapic    	0
timer interrupts generated by vlapic    	0
corrected machine check interrupts generated by vlapic	0
lvts triggered[0]                       	0
lvts triggered[1]                       	0
lvts triggered[2]                       	0
lvts triggered[3]                       	0
lvts triggered[4]                       	0
lvts triggered[5]                       	0
lvts triggered[6]                       	0
ipis sent to vcpu[0]                    	0
ipis sent to vcpu[1]                    	0
ipis sent to vcpu[2]                    	0
ipis sent to vcpu[3]                    	0
ipis sent to vcpu[4]                    	0
ipis sent to vcpu[5]                    	0
ipis sent to vcpu[6]                    	0
ipis sent to vcpu[7]                    	0
ipis sent to vcpu[8]                    	0
ipis sent to vcpu[9]                    	0
ipis sent to vcpu[10]                   	0
ipis sent to vcpu[11]                   	0
ipis sent to vcpu[12]                   	0
ipis sent to vcpu[13]                   	0
ipis sent to vcpu[14]                   	0
ipis sent to vcpu[15]                   	0
number of ticks vcpu was idle           	0
vcpu migration across host cpus         	0
total number of vm exits                	0
vm exits due to external interrupt      	0
Number of vpid invalidations saved      	0
Number of vpid invalidations done       	0
number of times hlt was intercepted     	0
number of times %cr access was intercepted	0
number of times rdmsr was intercepted   	0
number of times wrmsr was intercepted   	0
number of monitor trap exits            	0
number of times pause was intercepted   	0
vm exits due to interrupt window opening	0
vm exits due to nmi window opening      	0
<<END BHYVECTL GET-STATS>>

This was then followed with invoking Bhyve, as per original posting.

This again results in Layer 1 - FreeBSD VM going to pause with same error via Layer 0 - /var/log/libvirt/qemu/VM.log

<<LAYER 0 - LOG RESULT>>
char device redirected to /dev/pts/27 (label charserial0)
2020-05-09T06:28:07.870436Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T06:28:07.870499Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-09T06:28:07.872528Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T06:28:07.872543Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-09T06:28:07.873742Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T06:28:07.873757Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-09T06:28:07.874950Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T06:28:07.874964Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 00000000 00008000 DPL=0 <hiword>
CS =0000 00000000 00000000 00008000 DPL=0 <hiword>
SS =0000 00000000 00000000 00008000 DPL=0 <hiword>
DS =0000 00000000 00000000 00008000 DPL=0 <hiword>
FS =0000 00000000 00000000 00008000 DPL=0 <hiword>
GS =0000 00000000 00000000 00008000 DPL=0 <hiword>
LDT=0000 00000000 00000000 00008000 DPL=0 <hiword>
TR =0000 00000000 00000000 00008000 DPL=0 <hiword>
GDT=     0000000000000000 00000000
IDT=     0000000000000000 00000000
CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2020-05-09T06:42:45.518166Z qemu-system-x86_64: terminating on signal 15 from pid 2119 (/usr/sbin/libvirtd)
2020-05-09 06:42:45.919+0000: shutting down, reason=destroyed
<<END LAYER 0 - LOG RESULT>>


B. Running bhyve with UEFI rom rather than BIOS:

# bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd bee

<<LAYER 0 TAIL RESULT>>
$ cd /var/log/libvirt/qemu
$ sudo tail -f  swarm-bhyve-freebsd.log 
[sudo] password for XXX: 
-msg timestamp=on
char device redirected to /dev/pts/27 (label charserial0)
2020-05-09T07:21:12.156910Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T07:21:12.156975Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-09T07:21:12.158959Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T07:21:12.158973Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-09T07:21:12.160187Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T07:21:12.160201Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-09T07:21:12.161459Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T07:21:12.161471Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000f00
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 00000000 00008000 DPL=0 <hiword>
CS =0000 00000000 00000000 00008000 DPL=0 <hiword>
SS =0000 00000000 00000000 00008000 DPL=0 <hiword>
DS =0000 00000000 00000000 00008000 DPL=0 <hiword>
FS =0000 00000000 00000000 00008000 DPL=0 <hiword>
GS =0000 00000000 00000000 00008000 DPL=0 <hiword>
LDT=0000 00000000 00000000 00008000 DPL=0 <hiword>
TR =0000 00000000 00000000 00008000 DPL=0 <hiword>
GDT=     0000000000000000 00000000
IDT=     0000000000000000 00000000
CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2020-05-09T07:30:48.814754Z qemu-system-x86_64: terminating on signal 15 from pid 2119 (/usr/sbin/libvirtd)
2020-05-09 07:30:49.015+0000: shutting down, reason=destroyed
<<END LAYER 0 TAIL>>

So there must be some variation between my and your setup.

I am running test on: Lenovo x3650 M5 Server with 2 x CPU 
QEMU Q35 V4.2 with OVMF Layer 1 FreeBSD 12.1 VM
with 2 x vmxnet3 vNICs (1 for machine access and 1 for tap0 use)
Due to bug with FreeBSD 12.1 and QEMU with netmap I have recompiled my Layer 1 FreeBSD kernel with "dev netmap" removed.

Could you possibly suggest way to get better diagnostics

Thanks.

Cheers,

John Hartley.
Comment 10 Peter Grehan freebsd_committer 2020-05-09 09:50:04 UTC
I used a laptop-class CPU when trying the repro (core i5-8250U). In particular, there is a difference in VT-x features as reported by the L1 guest:

  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID

vs your system:

  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr

The one of interest here is that KVM is emulating APIC-virtualization (VID,PostIntr).

One thing to try is to disable this feature in the L1 guest. In /boot/loader.conf, put an entry 

hw.vmm.vmx.use_apic_vid=0

After (re)booting the L1 guest, the status of this parameter can be checked with:

# sysctl hw.vmm.vmx.cap.virtual_interrupt_delivery
hw.vmm.vmx.cap.virtual_interrupt_delivery: 0          <- should be zero
Comment 11 John Hartley 2020-05-09 13:55:02 UTC
(In reply to Peter Grehan from comment #10)

Hi Peter,

you have eagle eyes.

Your suggestion worked and I now have Layer 3 - FreeBSD UEFI guest doing an install.


I also tried to see if disabling X2APIC via QEMU fixed the problem via:

<cpu mode='host-model' check='partial'>
  <feature policy='disable' name='x2apic'/>
</cpu>

This did not work, but it did turn off the feature correct in Layer 1 - Guest / Host.

Here is output from sysctl on Layer 1 - FreeBSD Guest Host

<<LAYER 1 - SYSCTL>>
# sysctl hw.vmm
hw.vmm.amdvi.domain_id: 0
hw.vmm.amdvi.disable_io_fault: 0
hw.vmm.amdvi.ptp_level: 4
hw.vmm.amdvi.host_ptp: 1
hw.vmm.amdvi.enable: 0
hw.vmm.amdvi.count: 0
hw.vmm.npt.pmap_flags: 0
hw.vmm.svm.num_asids: 0
hw.vmm.svm.disable_npf_assist: 0
hw.vmm.svm.features: 4294967295
hw.vmm.svm.vmcb_clean: 1023
hw.vmm.vmx.l1d_flush_sw: 0
hw.vmm.vmx.l1d_flush: 0
hw.vmm.vmx.vpid_alloc_failed: 0
hw.vmm.vmx.posted_interrupt_vector: -1
hw.vmm.vmx.cap.posted_interrupts: 0
hw.vmm.vmx.cap.virtual_interrupt_delivery: 0
hw.vmm.vmx.cap.invpcid: 1
hw.vmm.vmx.cap.monitor_trap: 1
hw.vmm.vmx.cap.unrestricted_guest: 1
hw.vmm.vmx.cap.pause_exit: 1
hw.vmm.vmx.cap.halt_exit: 1
hw.vmm.vmx.initialized: 1
hw.vmm.vmx.cr4_zeros_mask: 18446744073705934848
hw.vmm.vmx.cr4_ones_mask: 8192
hw.vmm.vmx.cr0_zeros_mask: 18446744071025197056
hw.vmm.vmx.cr0_ones_mask: 32
hw.vmm.vmx.no_flush_rsb: 0
hw.vmm.ept.pmap_flags: 1531
hw.vmm.vrtc.flag_broken_time: 1
hw.vmm.ppt.devices: 0
hw.vmm.iommu.enable: 1
hw.vmm.iommu.initialized: 0
hw.vmm.bhyve_xcpuids: 0
hw.vmm.topology.cpuid_leaf_b: 1
hw.vmm.create: beavis
hw.vmm.destroy: beavis
hw.vmm.trace_guest_exceptions: 0
hw.vmm.ipinum: 251
hw.vmm.halt_detection: 1
<<END LAYER 1 - SYSCTL>>


So have work around.

Do think believe this needs fix within FreeBSD bhyve or is issue with QEMU / KVM ?

NOTE: I also tested with FreeBSD 13 CURRENT and this had same problem.

Thanks again for help on this.

Cheers,


John Hartley
Comment 12 Peter Grehan freebsd_committer 2020-05-09 21:33:06 UTC
Glad to hear that worked.

x2apic is fine - it has to be explicitly enabled in bhyve for a guest to use.

This would appear to be a bug in kvm since apic-virtualization in bhyve has been working fine on h/w for a number of years now. However, the fact that it fails almost immediately (on first guest entry perhaps) would indicate it may not be difficult to fix.
Comment 13 John Hartley 2021-05-18 01:56:29 UTC
(In reply to Peter Grehan from comment #12)

Hi Peter,

I have been working with Ubuntu team again on this: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1876678

and based on further testing have raise issue with upstream QEMU: https://gitlab.com/qemu-project/qemu/-/issues/337

Just cross linking so issue can be tracked across various components.

Thanks,

John Hartley