Bug 203994 - bhyve kernel module may need to relax some checks when running nested under KVM
Summary: bhyve kernel module may need to relax some checks when running nested under KVM
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-10-24 03:02 UTC by Peter Grehan
Modified: 2017-11-11 13:51 UTC (History)
10 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Grehan freebsd_committer 2015-10-24 03:02:03 UTC
It has been reported that bhyve doesn't run nested under KVM
   http://docs.FreeBSD.org/cgi/mid.cgi?78cd6c472dfa583c81442d766e620bf7

The thread showed that the issues were related to vmm.ko checking
for particular features that aren't implemented in KVM - the INS/OUTS vmexit capability, and also VPPID support in EPT.

These features (and perhaps others that haven't been hit) are not strictly
necessary to run bhyve. It may be worth relaxing the checks and implementing
workarounds for when the features aren't available, so that bhyve can run nested in this mode.
Comment 1 kay.diam 2016-08-24 14:14:10 UTC
+1

Have the same issue under KVM:

$ kldload vmm
$ dmesg | egrep 'vmx|vmm'
vmx_init: processor does not support desired basic capabilities
module_register_init: MOD_LOAD (vmm, 0xffffffff81a14230, 0) error 22
Comment 2 Sönke Schau 2017-01-04 15:49:20 UTC
Hello!
Any progress here?
This affects FreeeNAS 10 Installations running in KVM too.
Neither Dockers nor VM Support - so no plugins.
Best regards
Sönke
Comment 3 Peter Grehan freebsd_committer 2017-01-06 16:06:51 UTC
No progress. Tbanks for the reminder - I have some test code that I can repurpose from another project that can at least determine if ins/outs exits work correctly, so will give that a try.
Comment 4 Ahmad A. 2017-02-08 17:09:35 UTC
Hi. I thought I'd chime in and mention that bhyve doesn't work nested under Hyper-V either, at least not on my setup; the relevant lines from dmesg read:

vmx_init: processor does not support desired basic capabilities
module_register_init: MOD_LOAD (vmm, 0xffffffff826344e0, 0) error 22

VirtualBox (with VT-x and all) works fine though.
Comment 5 Roman Bogorodskiy freebsd_committer 2017-02-22 17:00:11 UTC
Will be happy to test bhyve with nested kvm when there are some patches ready.
Comment 6 Tom M 2017-03-27 16:40:26 UTC
Also willing to test patches when available.
Comment 7 wh 2017-03-27 18:52:34 UTC
Due to this bug I'm unable to run VMs and Docker-instances on FreeNAS Corral. I'm ready to test whatever patch you may throw at us.
Comment 8 Tom M 2017-04-05 17:17:42 UTC
Any update by chance?
Comment 9 Peter Grehan freebsd_committer 2017-04-05 20:13:10 UTC
I've not gotten to it: will carve out some time this weekend.

For those impacted, would you be able to post the output from the following commands from FreeBSD when running as a KVM guest ?

kldload cpuctl

cpucontrol -m 0x480 /dev/cpuctl0
cpucontrol -m 0x482 /dev/cpuctl0
cpucontrol -m 0x48b /dev/cpuctl0
cpucontrol -m 0x48c /dev/cpuctl0

Sample output from an Atom C2758:

 # cpucontrol -m 0x480 /dev/cpuctl0
 MSR 0x480: 0x00da0400 0x00000002
 # cpucontrol -m 0x482 /dev/cpuctl0
 MSR 0x482: 0xfff9fffe 0x0401e172
 # cpucontrol -m 0x48b /dev/cpuctl0
 MSR 0x48b: 0x000028ef 0x00000000
 # cpucontrol -m 0x48c /dev/cpuctl0
 MSR 0x48c: 0x00000f01 0x06114141
Comment 10 Roman Bogorodskiy freebsd_committer 2017-04-06 13:40:59 UTC
(In reply to Peter Grehan from comment #9)

Here's what I have in FreeBSD (12-CURRENT as of Feb 22 2017) inside KVM (Fedora 25):

# cpucontrol -m 0x480 /dev/cpuctl0
MSR 0x480: 0x00d81000 0x11e57ed0
# cpucontrol -m 0x482 /dev/cpuctl0
MSR 0x482: 0xfff9fffe 0x0401e172
# cpucontrol -m 0x48b /dev/cpuctl0
MSR 0x48b: 0x000000fb 0x00000000
# cpucontrol -m 0x48c /dev/cpuctl0
MSR 0x48c: 0x00000601 0x06114041
# 

in case if could be useful, cpu part of dmesg looks like this:


CPU: Intel(R) Core(TM) i5-4690 CPU @ 3.50GHz (3492.07-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306c3  Family=0x6  Model=0x3c  Stepping=3
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x72a<TSCADJ,BMI1,AVX2,BMI2,ERMS,INVPCID>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Hypervisor: Origin = "KVMKVMKVM"
Comment 11 Chris A 2017-04-06 17:56:35 UTC
(In reply to Peter Grehan from comment #9)

I have FreeNas running in Proxmox, and here's my output:

# cpucontrol -m 0x480 /dev/cpuctl0
MSR 0x480: 0x00981000 0x11e57ed0
# cpucontrol -m 0x482 /dev/cpuctl0
MSR 0x482: 0xfff9fffe 0x0401e172
# cpucontrol -m 0x48b /dev/cpuctl0
MSR 0x48b: 0x000000fb 0x00000000
# cpucontrol -m 0x48c /dev/cpuctl0
MSR 0x48c: 0x00000601 0x04114040

plus a couple of other bits that might be useful:

# uname -a
FreeBSD freenas.local 11.0-STABLE FreeBSD 11.0-STABLE #6 ...

# dmesg | grep CPU
CPU: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (3092.89-MHz K8-class CPU)
Comment 12 Tom M 2017-04-06 20:34:02 UTC
CPU: Intel(R) Xeon(R) CPU E3-1245 v3 @ 3.40GHz (3392.20-MHz K8-class CPU)

cpucontrol -m 0x480 /dev/cpuctl0
MSR 0x480: 0x00981000 0x11e57ed0
cpucontrol -m 0x482 /dev/cpuctl0
MSR 0x482: 0xfff9fffe 0x0401e172
cpucontrol -m 0x48b /dev/cpuctl0
MSR 0x48b: 0x000000fb 0x00000000
cpucontrol -m 0x48c /dev/cpuctl0
MSR 0x48c: 0x00000601 0x04114040


CPU: Intel(R) Atom(TM) CPU  C2750  @ 2.40GHz (2400.08-MHz K8-class CPU)

cpucontrol -m 0x480 /dev/cpuctl0
MSR 0x480: 0x00981000 0x11e57ed0
cpucontrol -m 0x482 /dev/cpuctl0
MSR 0x482: 0xfff9fffe 0x0401e172
cpucontrol -m 0x48b /dev/cpuctl0
MSR 0x48b: 0x000000eb 0x00000000
cpucontrol -m 0x48c /dev/cpuctl0
MSR 0x48c: 0x00000601 0x04114040
Comment 13 Peter Grehan freebsd_committer 2017-04-10 22:29:10 UTC
For completeness, here's the output from FreeBSD running as a KVM guest on Ubuntu 16.04, itself running on VMWare Fusion, on an Intel i5-3230M

MSR 0x480: 0x00981000 0x11e57ed0
MSR 0x482: 0xfff9fffe 0x0401e172
MSR 0x48b: 0x000000f2 0x00000000
MSR 0x48c: 0x00000601 0x04114040
Comment 14 Peter Grehan freebsd_committer 2017-04-10 22:32:53 UTC
Thanks to those who posted MSR output.

There are 3 issues:

 - the INST/OUTS capability isn't advertised on some KVM versions. I need to run a simple test to see if this matters or not in this environment. Hopefully it won't, and this test can be relaxed.

 - bhyve tests for all variants of INVVPID, but only uses the 2 that are advertised by KVM. This test will be dropped back to just those 2.

 - bhyve does use both forms of INVEPT, where some versions of KVM only advertise the coarse version. Since the finer-grained version is just an optimization, bhyve will be modified to only use the finer-grained version if it is available.

I'll try and get a patch together shortly so these mods can be tested.
Comment 15 Martin Lucina 2017-04-12 19:17:14 UTC
Hi, to add some more data points, this is Freebsd 11-RELEASE running in KVM (with nested=Y, Linux kernel 4.9.20, QEMU version 2.8.0(Debian 1:2.8+dfsg-3~bpo8+1):

CPU: Intel(R) Core(TM) i5-2540M CPU @ 2.60GHz (2594.17-MHz K8-class CPU)

MSR 0x480: 0x00d81000 0x11e57ed0
MSR 0x482: 0xfff9fffe 0x0401e172
MSR 0x48b: 0x000000fb 0x00000000
MSR 0x48c: 0x00000601 0x06114041
Comment 16 Keith 2017-04-14 14:04:31 UTC
Hi, Add me to the list of folks with this issue.

[root@freenas] ~# cpucontrol -m 0x480 /dev/cpuctl0
MSR 0x480: 0x00981000 0x11e57ed0
[root@freenas] ~# cpucontrol -m 0x482 /dev/cpuctl0
MSR 0x482: 0xfff9fffe 0x0401e172
[root@freenas] ~# cpucontrol -m 0x48b /dev/cpuctl0
MSR 0x48b: 0x000000fb 0x00000000
[root@freenas] ~# cpucontrol -m 0x48c /dev/cpuctl0
MSR 0x48c: 0x00000601 0x04114040
[root@freenas] ~# dmesg | grep CPU
CPU: Intel(R) Xeon(R) CPU           X5690  @ 3.47GHz (3458.07-MHz K8-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
Comment 17 Peter Grehan freebsd_committer 2017-04-30 18:17:42 UTC
Just an update on this. I used Ubuntu since it was easy to get a range of Linux kernel versions to test with.
 - Ubuntu 17.04 (kernel 4.10) works without any changes to bhyve
 - Ubuntu 15.10 doesn't advertize INVVPID capability. While it would be possible to modify bhyve to work in this mode, I'm loathe to do it since it's such a niche case
 - Ubuntu 16.04 seems to have the capabilities that most folk have reported.

After modifying bhyve to ignore the unused INVVPID capabilities and to fall-back to the all-context INVEPT if single-context wasn't available, the next issue was that Linux/KVM wasn't setting the GLA on EPT faults. This is used by bhyve to verify the GLA that is also obtained through the instruction emulation path for guest MMIO operations. This verification path hasn't thrown any errors for a few years now, so I relaxed this test when in this mode.

(Of note, this bug was reported here https://lkml.org/lkml/2016/9/4/156, and the author has submitted a diff review to relax some of the INVVPID/INVEPT checks at https://reviews.freebsd.org/D10049)

However, after doing all this, bhyve would still eventually panic with a kernel trap-9 fault when issuing a vmread instruction, so there may still be some additional KVM bugs that are being hit on this kernel rev :(

I'll do some more experimentation and then post the patch that I have.
Comment 18 Roman Bogorodskiy freebsd_committer 2017-11-06 16:07:35 UTC
Any updates?
Comment 19 Peter Grehan freebsd_committer 2017-11-09 17:08:45 UTC
No updates.

Do you have a specific kernel version that you are interested in ? If so I could restrict my testing to just that.

My concern was that there would be a never-ending set of fidelity issues in KVM with kernels <= 4.10.
Comment 20 Roman Bogorodskiy freebsd_committer 2017-11-11 13:51:27 UTC
(In reply to Peter Grehan from comment #19)

As for me, kernels >= 4.10 are fine for me. I use Fedora for my Linux-related stuff, Fedora 26 comes with 4.11, and Fedora 27 that will be released soon will have 4.13 I think.