BUG: Starting FreeBSD Layer 2 bhyve Guest within Layer 1 FreeBSD VM Host on Layer 0 Ubuntu 20.04 KVM / QEMU Host result in Layer 1 Guest / Host Pausing with "Emulation Failure" TESTING: My test scenario is nested virtualisation: Layer 0 - Ubuntu 20.04 Host Layer 1 - FreeBSD 12.1 with OVMF + bhyve hypervisor Guest/Host Layer 2 - FreeBSD 12.1 guest Layer 0 Host is: Ubuntu 20.04 LTS KVM / QEMU / libvirt <<START QEMU VERSION>> $ virsh -c qemu:///system version --daemon Compiled against library: libvirt 6.0.0 Using library: libvirt 6.0.0 Using API: QEMU 6.0.0 Running hypervisor: QEMU 4.2.0 Running against daemon: 6.0.0 <<END QEMU VERSION> <<START Intel VMX Support & Nesting Enabled>> $ cat /proc/cpuinfo | grep -c vmx 64 $ cat /sys/module/kvm_intel/parameters/nested Y <<END Intel VMS>> Layer 1 Guest / Host is: FreeBSD Q35 v4.2 with OVMF: Pass Host VMX support to Layer 1 Guest via <cpu mode='host-model> <<LIBVIRT CONFIG SNIPPET>> ... ... <os> <type arch='x86_64' machine='pc-q35-4.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader> <nvram>/home/USER/swarm.bhyve.freebsd/OVMF_VARS.fd</nvram> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='host-model' check='partial'/> ... ... <END LIBVIRT CONFIG SNIPPET>> Checked that Layer 1 - FreeBSD Quest / Host has VMX feature available: <<LAYER 1 - FreeBSD CPU Features>> # uname -a FreeBSD swarm.DOMAIN.HERE 12.1-RELEASE FreeBSD 12.1-RELEASE GENERIC amd64 # grep Features /var/run/dmesg.boot Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS> Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x121<LAHF,ABM,Prefetch> Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP> Structured Extended Features2=0x4<UMIP> Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD> XSAVE Features=0x1<XSAVEOPT> <<END LAYER 1 - FreeBSD CPU Features> On Layer 1 FreeBSD Guest / Host start up the Layer 2 guest.. <<START LAYER 2 GUEST START>> # ls FreeBSD-11.2-RELEASE-amd64-bootonly.iso FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee-hd1-01.img # /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee <<END LAYER 2 GUEST START>> Result is that Layer 1 - FreeBSD Host guest "paused". To Layer 1 machines freezes I cannot get any further diagnostics from this machine, so I run tail on libvirt log from Layer 0 - Ubuntu Host <<LAYER 0 LOG TAIL>> char device redirected to /dev/pts/29 (label charserial0) 2020-05-04T06:09:15.310474Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-04T06:09:15.310531Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-04T06:09:15.312533Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-04T06:09:15.312548Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-04T06:09:15.313828Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-04T06:09:15.313841Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-04T06:09:15.315185Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-04T06:09:15.315201Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] KVM internal error. Suberror: 1 emulation failure EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 00000000 00008000 DPL=0 <hiword> CS =0000 00000000 00000000 00008000 DPL=0 <hiword> SS =0000 00000000 00000000 00008000 DPL=0 <hiword> DS =0000 00000000 00000000 00008000 DPL=0 <hiword> FS =0000 00000000 00000000 00008000 DPL=0 <hiword> GS =0000 00000000 00000000 00008000 DPL=0 <hiword> LDT=0000 00000000 00000000 00008000 DPL=0 <hiword> TR =0000 00000000 00000000 00008000 DPL=0 <hiword> GDT= 0000000000000000 00000000 IDT= 0000000000000000 00000000 CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000d01 Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 2020-05-04T06:35:39.186799Z qemu-system-x86_64: terminating on signal 15 from pid 2155 (/usr/sbin/libvirtd) 2020-05-04 06:35:39.386+0000: shutting down, reason=destroyed <<END LAYER 0 LOG TAIL>> I am reporting this bug here as result is very similar to that seen with QEMU seabios failure reported here: https://bugs.launchpad.net/qemu/+bug/1866870 However in this case my VM Layer 1 VM is using OVMF. NOTE 1: I have also tested with Q35 v3.1 and 2.12 and get the same result. NOTE 2: Due to bug in FreeBSD networking code, I had to compile custom kernel with "netmap driver disabled". This is known bug in FreeBSD that I have reported separately.
NOTE: I have cross posted this Bug Report Ubuntu Bug Tracking: https://bugs.launchpad.net/qemu/+bug/1876678 Also note I have done extensive testing of Ubuntu 20.04 Nested virtualisation with just Ubuntu hosts and OVMF and the nested virtualisation runs correctly, so problem is specific to using FreeBSD / bhyve guest / host.
FreeBSD Bhyve does *not* support nested virtualization as a host, your operating beyond what has been implemented.
Hi Rodney, thanks for looking at this bug report. Does your statement, "FreeBSD Bhyve does *not* support nested virtualization as a host" relate to this nested topology ? Layer 0 - FreeBSD with bhyve hypervisor (on Bare Metal with Intel VT-x VMX support) Layer 1 - FreeBSD with bhyve hypervisor (ie Guest of Layer 0, Host to Layer 2) Layer 2 - FreeBSD or other OS Guest I ask as bhyve wiki: https://wiki.freebsd.org/bhyve#Q:_Can_I_run_multiple_bhyve_hosts_under_VMware_nested_VT-x_EPT.3F Indicates that there should be ability to run nested topology as per my example: Layer 0 - VMWare or KVM with VT-x EPT Layer 1 - FreeBSD with bhyve hypervisor (ie Guest of Layer 0, Host to Layer 2) Layer 2 - FreeBSD or other OS Guest I am trying to get this model up and running, as I am wanting to test metrics collection from Layer 1 FreeBSD guest/host. Given scarcity of information I know that this is little off the beaten track, but I did find this example: https://github.com/Suhoy95/SNE-reports-2019-public/tree/4-LIA-2-xen-kvm which indicated that it had go similar nested topology going, but with Ubuntu 18.04 as Layer 0 rather than Ubuntu 20.04 as in my case. Thanks you. John Hartley.
Yours is a valid configuration. This was reported a long while back and discussed extensively in bug 203994, where the fix was to use a kernel version >= 4.10. So, this appears to be a regression.
(In reply to Peter Grehan from comment #4) How is running FreeBSD at layer 2, inside of a layer 1 FreeBSD (host) a valid configuration, I thought we had no support for nesting. Does some magic happen because the layer 0/bare metal host is running KVM? Now, running FreeBSD at layer 1, inside a layer 0 KVM, I'll agree is valid.
(In reply to Rodney W. Grimes from comment #5) Never mind, I see why, layer 2 is NOT running bhyve, it is just running FreeBSD
Hi Peter & Co, I reviewed the prior bug report and FYI here is some additional information on my machine: <<START LAYER 1 - FreeBSD CPU Report from dmesg.boot>> ... ... CPU: Intel Core Processor (Broadwell, IBRS) (2600.09-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306d2 Family=0x6 Model=0x3d Stepping=2 Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS> Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x121<LAHF,ABM,Prefetch> Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP> Structured Extended Features2=0x4<UMIP> Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD> XSAVE Features=0x1<XSAVEOPT> IA32_ARCH_CAPS=0x8<SKIP_L1DFL_VME> AMD Extended Feature Extensions ID EBX=0x1001000 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr Hypervisor: Origin = "KVMKVMKVM" ... ... <END LAYER 1 - dimes.log>> I also did cpuctl load and here are result from cpucontrol (I am not sure how to interpret these or if provided vectors are still valid with FreeBSD 12.1) <<START CPUCTL>> # kldload cpuctl # cpucontrol -m 0x480 /dev/cpuctl0 MSR 0x480: 0x00d81000 0x11e57ed0 # cpucontrol -m 0x482 /dev/cpuctl0 MSR 0x482: 0xfff9fffe 0x0401e172 # cpucontrol -m 0x48b /dev/cpuctl0 MSR 0x48b: 0x00037bff 0x00000000 # cpucontrol -m 0x48c /dev/cpuctl0 MSR 0x48c: 0x00000f01 0x06334041 <<END CPUCTL>> I also saw that in the original but, it appear that issue was that the bhyve KLM was not loading. I my case the vmm module is loading but I am getting runtime failure... Any tips you can provide to help with diagnosis would be appreciated and I can chase upstream if needed, if this is KVM regression. Thanks. John Hartley
I was able to get this working fine, and the cause of the problem is clear looking back at the posted logs. <<START LAYER 2 GUEST START>> # ls FreeBSD-11.2-RELEASE-amd64-bootonly.iso FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee-hd1-01.img # /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee bhyve has to be started with either a UEFI ROM image, or using a separate loader (bhyveload/grub-bhyve). This wasn't done in this case, which will result in registers partly set to power-on state, which will result in a triple-fault or some other error. This can be seen in the register dump from KVM, where all the GP registers are zero including the RIP. The fix to run a FreeBSD guest is to use bhyveload before running bhyve. For the example given, this would be # /usr/sbin/bhyveload -m 2048 -d ./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee ... and then run the bhyve command.
(In reply to Peter Grehan from comment #8) Hi Peter, thanks for reviewing. I have done further testing: A. Running bhyveload first as per your feedback Result: Brings up FreeBSD Loader and then get to "/dev/entropy not found message and closes machine <<BHYVECTL GET-STATS>> # bhyvectl --vm=bee --get-stats > bee-vm-stats-01.txt # cat bee-vm-stats-01.txt vcpu0 stats: number of times in/out was intercepted 0 number of times cpuid was intercepted 0 vm exits due to nested page fault 0 vm exits for instruction emulation 0 number of vm exits for unknown reason 0 number of times astpending at exit 0 number of times idle requested at exit 0 number of vm exits handled in userspace 0 number of times rendezvous pending at exit 0 number of vm exits due to exceptions 0 number of NMIs delivered to vcpu 0 number of ExtINTs delivered to vcpu 0 Resident memory 0 Wired memory 0 vcpu total runtime 0 EOI without any in-service interrupt 0 error interrupts generated by vlapic 0 timer interrupts generated by vlapic 0 corrected machine check interrupts generated by vlapic 0 lvts triggered[0] 0 lvts triggered[1] 0 lvts triggered[2] 0 lvts triggered[3] 0 lvts triggered[4] 0 lvts triggered[5] 0 lvts triggered[6] 0 ipis sent to vcpu[0] 0 ipis sent to vcpu[1] 0 ipis sent to vcpu[2] 0 ipis sent to vcpu[3] 0 ipis sent to vcpu[4] 0 ipis sent to vcpu[5] 0 ipis sent to vcpu[6] 0 ipis sent to vcpu[7] 0 ipis sent to vcpu[8] 0 ipis sent to vcpu[9] 0 ipis sent to vcpu[10] 0 ipis sent to vcpu[11] 0 ipis sent to vcpu[12] 0 ipis sent to vcpu[13] 0 ipis sent to vcpu[14] 0 ipis sent to vcpu[15] 0 number of ticks vcpu was idle 0 vcpu migration across host cpus 0 total number of vm exits 0 vm exits due to external interrupt 0 Number of vpid invalidations saved 0 Number of vpid invalidations done 0 number of times hlt was intercepted 0 number of times %cr access was intercepted 0 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 0 number of monitor trap exits 0 number of times pause was intercepted 0 vm exits due to interrupt window opening 0 vm exits due to nmi window opening 0 <<END BHYVECTL GET-STATS>> This was then followed with invoking Bhyve, as per original posting. This again results in Layer 1 - FreeBSD VM going to pause with same error via Layer 0 - /var/log/libvirt/qemu/VM.log <<LAYER 0 - LOG RESULT>> char device redirected to /dev/pts/27 (label charserial0) 2020-05-09T06:28:07.870436Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T06:28:07.870499Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T06:28:07.872528Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T06:28:07.872543Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T06:28:07.873742Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T06:28:07.873757Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T06:28:07.874950Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T06:28:07.874964Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] KVM internal error. Suberror: 1 emulation failure EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 00000000 00008000 DPL=0 <hiword> CS =0000 00000000 00000000 00008000 DPL=0 <hiword> SS =0000 00000000 00000000 00008000 DPL=0 <hiword> DS =0000 00000000 00000000 00008000 DPL=0 <hiword> FS =0000 00000000 00000000 00008000 DPL=0 <hiword> GS =0000 00000000 00000000 00008000 DPL=0 <hiword> LDT=0000 00000000 00000000 00008000 DPL=0 <hiword> TR =0000 00000000 00000000 00008000 DPL=0 <hiword> GDT= 0000000000000000 00000000 IDT= 0000000000000000 00000000 CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000d01 Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 2020-05-09T06:42:45.518166Z qemu-system-x86_64: terminating on signal 15 from pid 2119 (/usr/sbin/libvirtd) 2020-05-09 06:42:45.919+0000: shutting down, reason=destroyed <<END LAYER 0 - LOG RESULT>> B. Running bhyve with UEFI rom rather than BIOS: # bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd bee <<LAYER 0 TAIL RESULT>> $ cd /var/log/libvirt/qemu $ sudo tail -f swarm-bhyve-freebsd.log [sudo] password for XXX: -msg timestamp=on char device redirected to /dev/pts/27 (label charserial0) 2020-05-09T07:21:12.156910Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T07:21:12.156975Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T07:21:12.158959Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T07:21:12.158973Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T07:21:12.160187Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T07:21:12.160201Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T07:21:12.161459Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T07:21:12.161471Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] KVM internal error. Suberror: 1 emulation failure EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000f00 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 00000000 00008000 DPL=0 <hiword> CS =0000 00000000 00000000 00008000 DPL=0 <hiword> SS =0000 00000000 00000000 00008000 DPL=0 <hiword> DS =0000 00000000 00000000 00008000 DPL=0 <hiword> FS =0000 00000000 00000000 00008000 DPL=0 <hiword> GS =0000 00000000 00000000 00008000 DPL=0 <hiword> LDT=0000 00000000 00000000 00008000 DPL=0 <hiword> TR =0000 00000000 00000000 00008000 DPL=0 <hiword> GDT= 0000000000000000 00000000 IDT= 0000000000000000 00000000 CR0=80050033 CR2=0000000000000000 CR3=0000000000000000 CR4=00372060 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000d01 Code=<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 2020-05-09T07:30:48.814754Z qemu-system-x86_64: terminating on signal 15 from pid 2119 (/usr/sbin/libvirtd) 2020-05-09 07:30:49.015+0000: shutting down, reason=destroyed <<END LAYER 0 TAIL>> So there must be some variation between my and your setup. I am running test on: Lenovo x3650 M5 Server with 2 x CPU QEMU Q35 V4.2 with OVMF Layer 1 FreeBSD 12.1 VM with 2 x vmxnet3 vNICs (1 for machine access and 1 for tap0 use) Due to bug with FreeBSD 12.1 and QEMU with netmap I have recompiled my Layer 1 FreeBSD kernel with "dev netmap" removed. Could you possibly suggest way to get better diagnostics Thanks. Cheers, John Hartley.
I used a laptop-class CPU when trying the repro (core i5-8250U). In particular, there is a difference in VT-x features as reported by the L1 guest: VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID vs your system: VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr The one of interest here is that KVM is emulating APIC-virtualization (VID,PostIntr). One thing to try is to disable this feature in the L1 guest. In /boot/loader.conf, put an entry hw.vmm.vmx.use_apic_vid=0 After (re)booting the L1 guest, the status of this parameter can be checked with: # sysctl hw.vmm.vmx.cap.virtual_interrupt_delivery hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 <- should be zero
(In reply to Peter Grehan from comment #10) Hi Peter, you have eagle eyes. Your suggestion worked and I now have Layer 3 - FreeBSD UEFI guest doing an install. I also tried to see if disabling X2APIC via QEMU fixed the problem via: <cpu mode='host-model' check='partial'> <feature policy='disable' name='x2apic'/> </cpu> This did not work, but it did turn off the feature correct in Layer 1 - Guest / Host. Here is output from sysctl on Layer 1 - FreeBSD Guest Host <<LAYER 1 - SYSCTL>> # sysctl hw.vmm hw.vmm.amdvi.domain_id: 0 hw.vmm.amdvi.disable_io_fault: 0 hw.vmm.amdvi.ptp_level: 4 hw.vmm.amdvi.host_ptp: 1 hw.vmm.amdvi.enable: 0 hw.vmm.amdvi.count: 0 hw.vmm.npt.pmap_flags: 0 hw.vmm.svm.num_asids: 0 hw.vmm.svm.disable_npf_assist: 0 hw.vmm.svm.features: 4294967295 hw.vmm.svm.vmcb_clean: 1023 hw.vmm.vmx.l1d_flush_sw: 0 hw.vmm.vmx.l1d_flush: 0 hw.vmm.vmx.vpid_alloc_failed: 0 hw.vmm.vmx.posted_interrupt_vector: -1 hw.vmm.vmx.cap.posted_interrupts: 0 hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 hw.vmm.vmx.cap.invpcid: 1 hw.vmm.vmx.cap.monitor_trap: 1 hw.vmm.vmx.cap.unrestricted_guest: 1 hw.vmm.vmx.cap.pause_exit: 1 hw.vmm.vmx.cap.halt_exit: 1 hw.vmm.vmx.initialized: 1 hw.vmm.vmx.cr4_zeros_mask: 18446744073705934848 hw.vmm.vmx.cr4_ones_mask: 8192 hw.vmm.vmx.cr0_zeros_mask: 18446744071025197056 hw.vmm.vmx.cr0_ones_mask: 32 hw.vmm.vmx.no_flush_rsb: 0 hw.vmm.ept.pmap_flags: 1531 hw.vmm.vrtc.flag_broken_time: 1 hw.vmm.ppt.devices: 0 hw.vmm.iommu.enable: 1 hw.vmm.iommu.initialized: 0 hw.vmm.bhyve_xcpuids: 0 hw.vmm.topology.cpuid_leaf_b: 1 hw.vmm.create: beavis hw.vmm.destroy: beavis hw.vmm.trace_guest_exceptions: 0 hw.vmm.ipinum: 251 hw.vmm.halt_detection: 1 <<END LAYER 1 - SYSCTL>> So have work around. Do think believe this needs fix within FreeBSD bhyve or is issue with QEMU / KVM ? NOTE: I also tested with FreeBSD 13 CURRENT and this had same problem. Thanks again for help on this. Cheers, John Hartley
Glad to hear that worked. x2apic is fine - it has to be explicitly enabled in bhyve for a guest to use. This would appear to be a bug in kvm since apic-virtualization in bhyve has been working fine on h/w for a number of years now. However, the fact that it fails almost immediately (on first guest entry perhaps) would indicate it may not be difficult to fix.
(In reply to Peter Grehan from comment #12) Hi Peter, I have been working with Ubuntu team again on this: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1876678 and based on further testing have raise issue with upstream QEMU: https://gitlab.com/qemu-project/qemu/-/issues/337 Just cross linking so issue can be tracked across various components. Thanks, John Hartley