Bug 196093 - bhyve error "wrmsr to register 0x1a0" with Ubuntu 13.10
Summary: bhyve error "wrmsr to register 0x1a0" with Ubuntu 13.10
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-12-18 13:22 UTC by Diane Bruce
Modified: 2015-07-24 19:18 UTC (History)
3 users (show)

See Also:


Attachments
patch to bhyve/xmsr.c (739 bytes, patch)
2014-12-18 13:22 UTC, Diane Bruce
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Diane Bruce freebsd_committer 2014-12-18 13:22:36 UTC
Created attachment 150717 [details]
patch to bhyve/xmsr.c

FreeBSD VM's work fine, problem arose when running a linux emulation.

# bhyve -AI -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./linux.img -s 4:0,ahci-cd,./ubuntu-13.10-server-amd64.iso -l com1,stdio -c 4 -m 1024M linuxguest
wrmsr to register 0x1a0(0x801889) on vcpu 0
vm exit[0]
reason          VMX
rip              0x000000000154b760
inst_length     3
status         0                                                                                exit_reason     2
qualification   0x0000000000000000
inst_type      0                                                                                inst_error              0

The problem is in bhyve_xmsr.c

Patch attached.
Comment 1 Neel Natu freebsd_committer 2014-12-18 17:19:20 UTC
Can you dump the MISC_ENABLE MSR value on the host?

# kldload cpuctl
# cpucontrol -m 0x1A0 /dev/cpuctl0
Comment 2 Diane Bruce freebsd_committer 2014-12-18 17:47:16 UTC
# cpucontrol -m 0x1A0 /dev/cpuctl0
MSR 0x1a0: 0x00000004 0x00850089
Comment 3 Peter Grehan freebsd_committer 2014-12-18 20:08:04 UTC
FYI, it's not possible for bhyve to emulate every MSR, since many of these accessed by guests are specific to particular CPU models. The "-w" option (described in the man page) allows unimplemented MSRs to be ignored.

We usually look at MSRs on a case-by-case basis to determine if they are worth emulating or not - sometimes other unrelated issues are exposed.
Comment 4 Neel Natu freebsd_committer 2014-12-18 20:40:45 UTC
Thanks. Can you also provide the output of the following from the host?

# cpucontrol -i 0x80000001 /dev/cpuctl0
Comment 5 Diane Bruce freebsd_committer 2014-12-18 20:47:48 UTC
(In reply to Neel Natu from comment #4)
> Thanks. Can you also provide the output of the following from the host?
> 
> # cpucontrol -i 0x80000001 /dev/cpuctl0

# cpucontrol -i 0x80000001 /dev/cpuctl0 
cpuid level 0x80000001: 0x00000000 0x00000000 0x00000001 0x28000800
Comment 6 Diane Bruce freebsd_committer 2014-12-18 21:55:22 UTC
(In reply to Peter Grehan from comment #3)
> FYI, it's not possible for bhyve to emulate every MSR, since many of these
> accessed by guests are specific to particular CPU models. The "-w" option
> (described in the man page) allows unimplemented MSRs to be ignored.
> 
> We usually look at MSRs on a case-by-case basis to determine if they are
> worth emulating or not - sometimes other unrelated issues are exposed.

This should be mentioned in the handbook, but it does work. I was sure I tried it before with no luck. Close at will and I'll bug the doc folk.
Comment 7 Neel Natu freebsd_committer 2014-12-20 05:25:23 UTC
Diane, can you verify that the following patch fixes the problem for you?

https://people.freebsd.org/~neel/patches/bhyve_nx_disabled.patch

Of course, you'll want to do this without the "-w" option and also without your patch to xmsr.c.
Comment 8 Diane Bruce freebsd_committer 2014-12-20 16:03:39 UTC
(In reply to Neel Natu from comment #7)
> Diane, can you verify that the following patch fixes the problem for you?
> 
> https://people.freebsd.org/~neel/patches/bhyve_nx_disabled.patch
> 
> Of course, you'll want to do this without the "-w" option and also without
> your patch to xmsr.c.

Of course ;)

Rebuilding world and kernel now. I'll let you know.
Comment 9 Diane Bruce freebsd_committer 2014-12-20 17:26:18 UTC
(In reply to Diane Bruce from comment #8)
> (In reply to Neel Natu from comment #7)
> > Diane, can you verify that the following patch fixes the problem for you?
> > 
> > https://people.freebsd.org/~neel/patches/bhyve_nx_disabled.patch
> > 
> > Of course, you'll want to do this without the "-w" option and also without
> > your patch to xmsr.c.
> 
> Of course ;)
> 
> Rebuilding world and kernel now. I'll let you know.

It boots fine. Thanks!
Comment 10 Neel Natu freebsd_committer 2014-12-20 19:34:57 UTC
Thanks for verifying the fix.

The root cause was that the host system had the "no-execute" feature disabled (intentional?). This was passed through to the virtual machine and the guest kernel was trying to re-enable the feature by setting bit 34 of IA32_MISC_ENABLE msr.

bhyve was not handling writes to the IA32_MISC_ENABLE msr which caused it to exit.
Comment 11 Neel Natu freebsd_committer 2014-12-20 19:40:08 UTC
Also, in general you don't want to use the "-w" option to bhyve. Ignoring unimplemented MSRs might cause side-effects so it should only be used for debugging or as a quick and dirty workaround.

Please file PRs instead :-)
Comment 12 commit-hook freebsd_committer 2014-12-20 19:48:28 UTC
A commit references this bug:

Author: neel
Date: Sat Dec 20 19:47:52 UTC 2014
New revision: 275965
URL: https://svnweb.freebsd.org/changeset/base/275965

Log:
  Emulate writes to the IA32_MISC_ENABLE MSR.

  PR:		196093
  Reported by:	db
  Tested by:	db
  Discussed with:	grehan
  MFC after:	1 week

Changes:
  head/sys/amd64/vmm/intel/vmx_msr.c
Comment 13 Diane Bruce freebsd_committer 2014-12-20 21:18:11 UTC
(In reply to Neel Natu from comment #11)
> Also, in general you don't want to use the "-w" option to bhyve. Ignoring
> unimplemented MSRs might cause side-effects so it should only be used for
> debugging or as a quick and dirty workaround.
> 
> Please file PRs instead :-)

I always intended to ;-)

Close PR as you see fit. Thanks!
Comment 14 Conrad Meyer freebsd_committer 2015-07-24 01:06:39 UTC
Hey, I seem to be hitting this on a revision well after the fix: r283896.

The guest traps with:

...
kernel trap 9 with interrupts disabled
panic @ time 0.000, thread 0xffffffff813c2f20: Fatal trap 9: general protection fault while in kernel mode
...
Registers:
rax: 0000000000811889 rbx: ffffffff814077b8 rcx: 00000000000001a0 rdx: 0000000000000000
                                                              ^^^
                                               MSR_MISC_ENABLE or IA32_MISC_ENABLE

Are we missing some emulation? It looks like ECX is the MSR register, and
we're trying to write EDX:EAX (per [0]):

    00000000 00811889

Anything I can do to help fix this? Would you prefer a new bug? Thanks.

[0]: http://x86.renejeschke.de/html/file_module_x86_id_326.html
Comment 15 Conrad Meyer freebsd_committer 2015-07-24 01:10:34 UTC
Fwiw, the guest is attempting to configure Enhanced SpeedStep after detecting
the host as SandyBridge{,-E} or IveBridge{,-E}.

We are setting the SS_ENABLE bit and TB_DISABLE bits of IA32_MISC_ENABLE.

Finally, we also write the IA32_PERF_CTL register (haven't gotten that far
in boot yet, don't know if Bhyve handles that or not.

Pretty much all of these should just be ignored rather than trapping, IMO.
Comment 16 Conrad Meyer freebsd_committer 2015-07-24 01:27:41 UTC
Oh, also IA32_ENERGY_PERF_BIAS.
Comment 17 Peter Grehan freebsd_committer 2015-07-24 18:49:50 UTC
I guess Speedstep being disabled in CPUID isn't enough for some o/s versions :(

Shouldn't be too hard to ignore those bits in MISC_ENABLE and sink the writes to PERF_CTL and ENERGY_PERF_BIAS. Would you be able to test out a patch ?
Comment 18 Conrad Meyer freebsd_committer 2015-07-24 19:18:09 UTC
(In reply to Peter Grehan from comment #17)
> I guess Speedstep being disabled in CPUID isn't enough for some o/s versions :(
> Shouldn't be too hard to ignore those bits in MISC_ENABLE and sink the writes to PERF_CTL and ENERGY_PERF_BIAS. Would you be able to test out a patch ?

Yep, I'm happy to test out a patch.