Bug 218662

Summary: bhyve exposes CPU feature SDBG to guests, causing guest panic on OpenBSD 6.1
Product: Base System Reporter: todd.mortimer
Component: binAssignee: freebsd-virtualization mailing list <virtualization>
Status: New ---    
Severity: Affects Some People CC: allanjude, freebsd, grehan, maciej, olgeni, sascha.folie
Priority: ---    
Version: 10.3-RELEASE   
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
VMM patch (from HardenedBSD) none

Description todd.mortimer 2017-04-14 16:54:52 UTC
When attempting to create an OpenBSD 6.1 bhyve guest on 10.3-RELEASE-p18, the guest panics early in the boot process when attempting to disable the Silicon Debug feature identified by CPU feature bit SDBG. This feature bit is passed to guests if present on the host CPU. 

In my case, my CPU is this:

CPU: Intel(R) Core(TM) i7-4790S CPU @ 3.20GHz (3192.68-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306c3  Family=0x6  Model=0x3c  Stepping=3
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x27ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics


When OpenBSD 6.1 boots under byhve, it detects the SDBG feature and attempts to disable it, resulting in a protection fault / panic.

This was discussed on the OpenBSD misc list:

http://marc.info/?l=openbsd-misc&m=149136052120277&w=4

And a patch for HardenedBSD is here:

https://github.com/HardenedBSD/hardenedBSD/commit/cc91b57f4d1dabddfbf8b1e7655

As a workaround, passing -w to byhve allows the guest to boot, but the CPU feature should be masked out by the host.
Comment 1 Allan Jude freebsd_committer 2017-04-14 17:06:31 UTC
You can work around this by passing the -w flag to bhyve
Comment 2 Brandon Bergren 2017-10-29 21:45:28 UTC
Created attachment 187571 [details]
VMM patch (from HardenedBSD)

This affects me too.

HardenedBSD has the most elegant fix at https://github.com/HardenedBSD/hardenedBSD/commit/e76fcb77ba82649bc6aed808af06d6d2184847d8 -- I have attached a copy of the patch. I can confirm that with just that patch I can successfully run OpenBSD 6.1 in bhyve on my machine that is otherwise affected by this problem.

I believe it makes a lot more sense than emulating the MSR (i.e. what https://github.com/HardenedBSD/hardenedBSD/commit/cc91b57f4d1dabddfbf8b1e7655 does.)

My SuperMicro C7Z170-OCE board actually has bios options for enabling and disabling the silicon debug status and lock bit, so if I were to misconfigure my bios, the countermeasure would have actually been effective (when applied to the host).

I can verify though that with it disabled in the bios, it is indeed locked down, without needing the countermeasure.
root@narwhal:~ # kldload cpuctl
root@narwhal:~ # cpucontrol -m 0xc80 /dev/cpuctl0
MSR 0xc80: 0x00000000 0x40000000

I still think the countermeasure is a good idea though!
https://github.com/HardenedBSD/hardenedBSD/commit/da28546280938e19e866d2e11a9ccda3c4ca82fa is a version of the countermeasure that checks the virtualization flag too so it will avoid tickling the bhyve limitation while still providing protection.

But at the very least, filter the SDBG feature flag.