Bug 218662 - bhyve exposes CPU feature SDBG to guests, causing guest panic on OpenBSD 6.1
Summary: bhyve exposes CPU feature SDBG to guests, causing guest panic on OpenBSD 6.1
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 10.3-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization mailing list
Depends on:
Reported: 2017-04-14 16:54 UTC by todd.mortimer
Modified: 2019-04-08 15:33 UTC (History)
7 users (show)

See Also:

VMM patch (from HardenedBSD) (941 bytes, patch)
2017-10-29 21:45 UTC, Brandon Bergren
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description todd.mortimer 2017-04-14 16:54:52 UTC
When attempting to create an OpenBSD 6.1 bhyve guest on 10.3-RELEASE-p18, the guest panics early in the boot process when attempting to disable the Silicon Debug feature identified by CPU feature bit SDBG. This feature bit is passed to guests if present on the host CPU. 

In my case, my CPU is this:

CPU: Intel(R) Core(TM) i7-4790S CPU @ 3.20GHz (3192.68-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306c3  Family=0x6  Model=0x3c  Stepping=3
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x27ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG>
  XSAVE Features=0x1<XSAVEOPT>
  TSC: P-state invariant, performance statistics

When OpenBSD 6.1 boots under byhve, it detects the SDBG feature and attempts to disable it, resulting in a protection fault / panic.

This was discussed on the OpenBSD misc list:


And a patch for HardenedBSD is here:


As a workaround, passing -w to byhve allows the guest to boot, but the CPU feature should be masked out by the host.
Comment 1 Allan Jude freebsd_committer 2017-04-14 17:06:31 UTC
You can work around this by passing the -w flag to bhyve
Comment 2 Brandon Bergren 2017-10-29 21:45:28 UTC
Created attachment 187571 [details]
VMM patch (from HardenedBSD)

This affects me too.

HardenedBSD has the most elegant fix at https://github.com/HardenedBSD/hardenedBSD/commit/e76fcb77ba82649bc6aed808af06d6d2184847d8 -- I have attached a copy of the patch. I can confirm that with just that patch I can successfully run OpenBSD 6.1 in bhyve on my machine that is otherwise affected by this problem.

I believe it makes a lot more sense than emulating the MSR (i.e. what https://github.com/HardenedBSD/hardenedBSD/commit/cc91b57f4d1dabddfbf8b1e7655 does.)

My SuperMicro C7Z170-OCE board actually has bios options for enabling and disabling the silicon debug status and lock bit, so if I were to misconfigure my bios, the countermeasure would have actually been effective (when applied to the host).

I can verify though that with it disabled in the bios, it is indeed locked down, without needing the countermeasure.
root@narwhal:~ # kldload cpuctl
root@narwhal:~ # cpucontrol -m 0xc80 /dev/cpuctl0
MSR 0xc80: 0x00000000 0x40000000

I still think the countermeasure is a good idea though!
https://github.com/HardenedBSD/hardenedBSD/commit/da28546280938e19e866d2e11a9ccda3c4ca82fa is a version of the countermeasure that checks the virtualization flag too so it will avoid tickling the bhyve limitation while still providing protection.

But at the very least, filter the SDBG feature flag.