Bug 246168 - Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Summary: Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: 12.1-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-05-04 08:44 UTC by John Hartley
Modified: 2020-05-09 21:33 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description John Hartley 2020-05-04 08:44:34 UTC
BUG:

Starting FreeBSD Layer 2 bhyve Guest within Layer 1 FreeBSD VM Host on Layer 0 Ubuntu 20.04 KVM / QEMU Host result in Layer 1 Guest / Host Pausing with "Emulation Failure"

TESTING:

My test scenario is nested virtualisation:
Layer 0 - Ubuntu 20.04 Host
Layer 1 - FreeBSD 12.1 with OVMF + bhyve hypervisor Guest/Host
Layer 2 - FreeBSD 12.1 guest

Layer 0 Host is: Ubuntu 20.04 LTS KVM / QEMU / libvirt

<<START QEMU VERSION>>
$ virsh -c qemu:///system version --daemon
Compiled against library: libvirt 6.0.0
Using library: libvirt 6.0.0
Using API: QEMU 6.0.0
Running hypervisor: QEMU 4.2.0
Running against daemon: 6.0.0
<<END QEMU VERSION>

<<START Intel VMX Support & Nesting Enabled>>
$ cat /proc/cpuinfo | grep -c vmx
64
$ cat /sys/module/kvm_intel/parameters/nested
Y
<<END Intel VMS>>



Layer 1 Guest / Host is: FreeBSD Q35 v4.2 with OVMF:

Pass Host VMX support to Layer 1 Guest via <cpu mode='host-model>

<<LIBVIRT CONFIG SNIPPET>>
...
...
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/home/USER/swarm.bhyve.freebsd/OVMF_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'/>
...
...
<END LIBVIRT CONFIG SNIPPET>>

Checked that Layer 1 - FreeBSD Quest / Host has VMX feature available:

<<LAYER 1 - FreeBSD CPU Features>>
# uname -a
FreeBSD swarm.DOMAIN.HERE 12.1-RELEASE FreeBSD 12.1-RELEASE GENERIC  amd64

# grep Features /var/run/dmesg.boot 
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
  Structured Extended Features2=0x4<UMIP>
  Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
<<END LAYER 1 - FreeBSD CPU Features>

On Layer 1 FreeBSD Guest / Host start up the Layer 2 guest..

<<START LAYER 2 GUEST START>>
# ls
FreeBSD-11.2-RELEASE-amd64-bootonly.iso	FreeBSD-12.1-RELEASE-amd64-dvd1.iso	bee-hd1-01.img
# /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee
<<END LAYER 2 GUEST START>>

Result is that Layer 1 - FreeBSD Host guest "paused".

To Layer 1 machines freezes I cannot get any further diagnostics from this machine, so I run tail on libvirt log from Layer 0 - Ubuntu Host

<<LAYER 0 LOG TAIL>>
char device redirected to /dev/pts/29 (label charserial0)
2020-05-04T06:09:15.310474Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.310531Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.312533Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.312548Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.313828Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.313841Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-04T06:09:15.315185Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-04T06:09:15.315201Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 00000000 00008000 DPL=0 <hiword>
CS =0000 00000000 00000000 00008000 DPL=0 <hiword>
SS =0000 00000000 00000000 00008000 DPL=0 <hiword>
DS =0000 00000000 00000000 00008000 DPL=0 <hiword>
FS =0000 00000000 00000000 00008000 DPL=0 <hiword>
GS =0000 00000000 00000000 00008000 DPL=0 <hiword>
LDT=0000 00000000 00000000 00008000 DPL=0 <hiword>
TR =0000 00000000 00000000 00008000 DPL=0 <hiword>
GDT=     0000000000000000 00000000
IDT=     0000000000000000 00000000
CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2020-05-04T06:35:39.186799Z qemu-system-x86_64: terminating on signal 15 from pid 2155 (/usr/sbin/libvirtd)
2020-05-04 06:35:39.386+0000: shutting down, reason=destroyed
<<END LAYER 0 LOG TAIL>>


I am reporting this bug here as result is very similar to that seen with QEMU seabios failure reported here: https://bugs.launchpad.net/qemu/+bug/1866870

However in this case my VM Layer 1 VM is using OVMF.

NOTE 1: I have also tested with Q35 v3.1 and 2.12 and get the same result.
NOTE 2: Due to bug in FreeBSD networking code, I had to compile custom kernel with "netmap driver disabled".  This is known bug in FreeBSD that I have reported separately.
Comment 1 John Hartley 2020-05-04 08:51:50 UTC
NOTE:

I have cross posted this Bug Report Ubuntu Bug Tracking: https://bugs.launchpad.net/qemu/+bug/1876678

Also note I have done extensive testing of Ubuntu 20.04 Nested virtualisation with just Ubuntu hosts and OVMF and the nested virtualisation runs correctly, so problem is specific to using FreeBSD / bhyve guest / host.
Comment 2 Rodney W. Grimes freebsd_committer 2020-05-04 14:25:18 UTC
FreeBSD Bhyve does *not* support nested virtualization as a host, your operating beyond what has been implemented.
Comment 3 John Hartley 2020-05-05 03:03:15 UTC
Hi Rodney,

thanks for looking at this bug report.

Does your statement, "FreeBSD Bhyve does *not* support nested virtualization as a host" relate to this nested topology ?

Layer 0 - FreeBSD with bhyve hypervisor (on Bare Metal with Intel VT-x VMX support)
Layer 1 - FreeBSD with bhyve hypervisor (ie Guest of Layer 0, Host to Layer 2)
Layer 2 - FreeBSD or other OS Guest

I ask as bhyve wiki: https://wiki.freebsd.org/bhyve#Q:_Can_I_run_multiple_bhyve_hosts_under_VMware_nested_VT-x_EPT.3F

Indicates that there should be ability to run nested topology as per my example:

Layer 0 - VMWare or KVM with VT-x EPT
Layer 1 - FreeBSD with bhyve hypervisor (ie Guest of Layer 0, Host to Layer 2)
Layer 2 - FreeBSD or other OS Guest

I am trying to get this model up and running, as I am wanting to test metrics collection from Layer 1 FreeBSD guest/host.

Given scarcity of information I know that this is little off the beaten track, but I did find this example: https://github.com/Suhoy95/SNE-reports-2019-public/tree/4-LIA-2-xen-kvm which indicated that it had go similar nested topology going, but with Ubuntu 18.04 as Layer 0 rather than Ubuntu 20.04 as in my case.

Thanks you.

John Hartley.
Comment 4 Peter Grehan freebsd_committer 2020-05-05 03:25:30 UTC
Yours is a valid configuration.

This was reported a long while back and discussed extensively in bug 203994, where the fix was to use a kernel version >= 4.10.

So, this appears to be a regression.
Comment 5 Rodney W. Grimes freebsd_committer 2020-05-05 03:48:21 UTC
(In reply to Peter Grehan from comment #4)
How is running FreeBSD at layer 2, inside of a layer 1 FreeBSD (host) a valid configuration, I thought we had no support for nesting.  Does some magic happen because the layer 0/bare metal host is running KVM?

Now, running FreeBSD at layer 1, inside a layer 0 KVM, I'll agree is valid.
Comment 6 Rodney W. Grimes freebsd_committer 2020-05-05 03:53:04 UTC
(In reply to Rodney W. Grimes from comment #5)
Never mind, I see why, layer 2 is NOT running bhyve, it is just running FreeBSD
Comment 7 John Hartley 2020-05-05 04:50:35 UTC
Hi Peter & Co,

I reviewed the prior bug report and FYI here is some additional information on my machine:

<<START LAYER 1 - FreeBSD CPU Report from dmesg.boot>>
...
...
CPU: Intel Core Processor (Broadwell, IBRS) (2600.09-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306d2  Family=0x6  Model=0x3d  Stepping=2
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
  Structured Extended Features2=0x4<UMIP>
  Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
  IA32_ARCH_CAPS=0x8<SKIP_L1DFL_VME>
  AMD Extended Feature Extensions ID EBX=0x1001000
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Hypervisor: Origin = "KVMKVMKVM"
...
...
<END LAYER 1 - dimes.log>>


I also did cpuctl load and here are result from cpucontrol

(I am not sure how to interpret these or if provided vectors are still valid with FreeBSD 12.1)

<<START CPUCTL>>
# kldload cpuctl
# cpucontrol -m 0x480 /dev/cpuctl0
MSR 0x480: 0x00d81000 0x11e57ed0
# cpucontrol -m 0x482 /dev/cpuctl0
MSR 0x482: 0xfff9fffe 0x0401e172
# cpucontrol -m 0x48b /dev/cpuctl0
MSR 0x48b: 0x00037bff 0x00000000
# cpucontrol -m 0x48c /dev/cpuctl0
MSR 0x48c: 0x00000f01 0x06334041
<<END CPUCTL>>

I also saw that in the original but, it appear that issue was that the bhyve KLM was not loading.

I my case the vmm module is loading but I am getting runtime failure...

Any tips you can provide to help with diagnosis would be appreciated and I can chase upstream if needed, if this is KVM regression.

Thanks.


John Hartley
Comment 8 Peter Grehan freebsd_committer 2020-05-07 08:43:18 UTC
I was able to get this working fine, and the cause of the problem is clear looking back at the posted logs.

<<START LAYER 2 GUEST START>>
# ls
FreeBSD-11.2-RELEASE-amd64-bootonly.iso	FreeBSD-12.1-RELEASE-amd64-dvd1.iso	bee-hd1-01.img
# /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee

 bhyve has to be started with either a UEFI ROM image, or using a separate loader (bhyveload/grub-bhyve). This wasn't done in this case, which will result in registers partly set to power-on state, which will result in a triple-fault or some other error.

This can be seen in the register dump from KVM, where all the GP registers are zero including the RIP.

The fix to run a FreeBSD guest is to use bhyveload before running bhyve. For the example given, this would be

# /usr/sbin/bhyveload -m 2048 -d ./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee

...

and then run the bhyve command.
Comment 9 John Hartley 2020-05-09 07:42:19 UTC
(In reply to Peter Grehan from comment #8)

Hi Peter,

thanks for reviewing.

I have done further testing:

A. Running bhyveload first as per your feedback

Result:

Brings up FreeBSD Loader and then get to "/dev/entropy not found message and closes machine


<<BHYVECTL GET-STATS>>
# bhyvectl --vm=bee --get-stats > bee-vm-stats-01.txt
# cat bee-vm-stats-01.txt 
vcpu0 stats:
number of times in/out was intercepted  	0
number of times cpuid was intercepted   	0
vm exits due to nested page fault       	0
vm exits for instruction emulation      	0
number of vm exits for unknown reason   	0
number of times astpending at exit      	0
number of times idle requested at exit  	0
number of vm exits handled in userspace 	0
number of times rendezvous pending at exit	0
number of vm exits due to exceptions    	0
number of NMIs delivered to vcpu        	0
number of ExtINTs delivered to vcpu     	0
Resident memory                         	0
Wired memory                            	0
vcpu total runtime                      	0
EOI without any in-service interrupt    	0
error interrupts generated by vlapic    	0
timer interrupts generated by vlapic    	0
corrected machine check interrupts generated by vlapic	0
lvts triggered[0]                       	0
lvts triggered[1]                       	0
lvts triggered[2]                       	0
lvts triggered[3]                       	0
lvts triggered[4]                       	0
lvts triggered[5]                       	0
lvts triggered[6]                       	0
ipis sent to vcpu[0]                    	0
ipis sent to vcpu[1]                    	0
ipis sent to vcpu[2]                    	0
ipis sent to vcpu[3]                    	0
ipis sent to vcpu[4]                    	0
ipis sent to vcpu[5]                    	0
ipis sent to vcpu[6]                    	0
ipis sent to vcpu[7]                    	0
ipis sent to vcpu[8]                    	0
ipis sent to vcpu[9]                    	0
ipis sent to vcpu[10]                   	0
ipis sent to vcpu[11]                   	0
ipis sent to vcpu[12]                   	0
ipis sent to vcpu[13]                   	0
ipis sent to vcpu[14]                   	0
ipis sent to vcpu[15]                   	0
number of ticks vcpu was idle           	0
vcpu migration across host cpus         	0
total number of vm exits                	0
vm exits due to external interrupt      	0
Number of vpid invalidations saved      	0
Number of vpid invalidations done       	0
number of times hlt was intercepted     	0
number of times %cr access was intercepted	0
number of times rdmsr was intercepted   	0
number of times wrmsr was intercepted   	0
number of monitor trap exits            	0
number of times pause was intercepted   	0
vm exits due to interrupt window opening	0
vm exits due to nmi window opening      	0
<<END BHYVECTL GET-STATS>>

This was then followed with invoking Bhyve, as per original posting.

This again results in Layer 1 - FreeBSD VM going to pause with same error via Layer 0 - /var/log/libvirt/qemu/VM.log

<<LAYER 0 - LOG RESULT>>
char device redirected to /dev/pts/27 (label charserial0)
2020-05-09T06:28:07.870436Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T06:28:07.870499Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-09T06:28:07.872528Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T06:28:07.872543Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-09T06:28:07.873742Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T06:28:07.873757Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-09T06:28:07.874950Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T06:28:07.874964Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 00000000 00008000 DPL=0 <hiword>
CS =0000 00000000 00000000 00008000 DPL=0 <hiword>
SS =0000 00000000 00000000 00008000 DPL=0 <hiword>
DS =0000 00000000 00000000 00008000 DPL=0 <hiword>
FS =0000 00000000 00000000 00008000 DPL=0 <hiword>
GS =0000 00000000 00000000 00008000 DPL=0 <hiword>
LDT=0000 00000000 00000000 00008000 DPL=0 <hiword>
TR =0000 00000000 00000000 00008000 DPL=0 <hiword>
GDT=     0000000000000000 00000000
IDT=     0000000000000000 00000000
CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2020-05-09T06:42:45.518166Z qemu-system-x86_64: terminating on signal 15 from pid 2119 (/usr/sbin/libvirtd)
2020-05-09 06:42:45.919+0000: shutting down, reason=destroyed
<<END LAYER 0 - LOG RESULT>>


B. Running bhyve with UEFI rom rather than BIOS:

# bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd bee

<<LAYER 0 TAIL RESULT>>
$ cd /var/log/libvirt/qemu
$ sudo tail -f  swarm-bhyve-freebsd.log 
[sudo] password for XXX: 
-msg timestamp=on
char device redirected to /dev/pts/27 (label charserial0)
2020-05-09T07:21:12.156910Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T07:21:12.156975Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-09T07:21:12.158959Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T07:21:12.158973Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-09T07:21:12.160187Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T07:21:12.160201Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-09T07:21:12.161459Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-09T07:21:12.161471Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000f00
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 00000000 00008000 DPL=0 <hiword>
CS =0000 00000000 00000000 00008000 DPL=0 <hiword>
SS =0000 00000000 00000000 00008000 DPL=0 <hiword>
DS =0000 00000000 00000000 00008000 DPL=0 <hiword>
FS =0000 00000000 00000000 00008000 DPL=0 <hiword>
GS =0000 00000000 00000000 00008000 DPL=0 <hiword>
LDT=0000 00000000 00000000 00008000 DPL=0 <hiword>
TR =0000 00000000 00000000 00008000 DPL=0 <hiword>
GDT=     0000000000000000 00000000
IDT=     0000000000000000 00000000
CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2020-05-09T07:30:48.814754Z qemu-system-x86_64: terminating on signal 15 from pid 2119 (/usr/sbin/libvirtd)
2020-05-09 07:30:49.015+0000: shutting down, reason=destroyed
<<END LAYER 0 TAIL>>

So there must be some variation between my and your setup.

I am running test on: Lenovo x3650 M5 Server with 2 x CPU 
QEMU Q35 V4.2 with OVMF Layer 1 FreeBSD 12.1 VM
with 2 x vmxnet3 vNICs (1 for machine access and 1 for tap0 use)
Due to bug with FreeBSD 12.1 and QEMU with netmap I have recompiled my Layer 1 FreeBSD kernel with "dev netmap" removed.

Could you possibly suggest way to get better diagnostics

Thanks.

Cheers,

John Hartley.
Comment 10 Peter Grehan freebsd_committer 2020-05-09 09:50:04 UTC
I used a laptop-class CPU when trying the repro (core i5-8250U). In particular, there is a difference in VT-x features as reported by the L1 guest:

  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID

vs your system:

  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr

The one of interest here is that KVM is emulating APIC-virtualization (VID,PostIntr).

One thing to try is to disable this feature in the L1 guest. In /boot/loader.conf, put an entry 

hw.vmm.vmx.use_apic_vid=0

After (re)booting the L1 guest, the status of this parameter can be checked with:

# sysctl hw.vmm.vmx.cap.virtual_interrupt_delivery
hw.vmm.vmx.cap.virtual_interrupt_delivery: 0          <- should be zero
Comment 11 John Hartley 2020-05-09 13:55:02 UTC
(In reply to Peter Grehan from comment #10)

Hi Peter,

you have eagle eyes.

Your suggestion worked and I now have Layer 3 - FreeBSD UEFI guest doing an install.


I also tried to see if disabling X2APIC via QEMU fixed the problem via:

<cpu mode='host-model' check='partial'>
  <feature policy='disable' name='x2apic'/>
</cpu>

This did not work, but it did turn off the feature correct in Layer 1 - Guest / Host.

Here is output from sysctl on Layer 1 - FreeBSD Guest Host

<<LAYER 1 - SYSCTL>>
# sysctl hw.vmm
hw.vmm.amdvi.domain_id: 0
hw.vmm.amdvi.disable_io_fault: 0
hw.vmm.amdvi.ptp_level: 4
hw.vmm.amdvi.host_ptp: 1
hw.vmm.amdvi.enable: 0
hw.vmm.amdvi.count: 0
hw.vmm.npt.pmap_flags: 0
hw.vmm.svm.num_asids: 0
hw.vmm.svm.disable_npf_assist: 0
hw.vmm.svm.features: 4294967295
hw.vmm.svm.vmcb_clean: 1023
hw.vmm.vmx.l1d_flush_sw: 0
hw.vmm.vmx.l1d_flush: 0
hw.vmm.vmx.vpid_alloc_failed: 0
hw.vmm.vmx.posted_interrupt_vector: -1
hw.vmm.vmx.cap.posted_interrupts: 0
hw.vmm.vmx.cap.virtual_interrupt_delivery: 0
hw.vmm.vmx.cap.invpcid: 1
hw.vmm.vmx.cap.monitor_trap: 1
hw.vmm.vmx.cap.unrestricted_guest: 1
hw.vmm.vmx.cap.pause_exit: 1
hw.vmm.vmx.cap.halt_exit: 1
hw.vmm.vmx.initialized: 1
hw.vmm.vmx.cr4_zeros_mask: 18446744073705934848
hw.vmm.vmx.cr4_ones_mask: 8192
hw.vmm.vmx.cr0_zeros_mask: 18446744071025197056
hw.vmm.vmx.cr0_ones_mask: 32
hw.vmm.vmx.no_flush_rsb: 0
hw.vmm.ept.pmap_flags: 1531
hw.vmm.vrtc.flag_broken_time: 1
hw.vmm.ppt.devices: 0
hw.vmm.iommu.enable: 1
hw.vmm.iommu.initialized: 0
hw.vmm.bhyve_xcpuids: 0
hw.vmm.topology.cpuid_leaf_b: 1
hw.vmm.create: beavis
hw.vmm.destroy: beavis
hw.vmm.trace_guest_exceptions: 0
hw.vmm.ipinum: 251
hw.vmm.halt_detection: 1
<<END LAYER 1 - SYSCTL>>


So have work around.

Do think believe this needs fix within FreeBSD bhyve or is issue with QEMU / KVM ?

NOTE: I also tested with FreeBSD 13 CURRENT and this had same problem.

Thanks again for help on this.

Cheers,


John Hartley
Comment 12 Peter Grehan freebsd_committer 2020-05-09 21:33:06 UTC
Glad to hear that worked.

x2apic is fine - it has to be explicitly enabled in bhyve for a guest to use.

This would appear to be a bug in kvm since apic-virtualization in bhyve has been working fine on h/w for a number of years now. However, the fact that it fails almost immediately (on first guest entry perhaps) would indicate it may not be difficult to fix.