Created attachment 150717 [details] patch to bhyve/xmsr.c FreeBSD VM's work fine, problem arose when running a linux emulation. # bhyve -AI -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./linux.img -s 4:0,ahci-cd,./ubuntu-13.10-server-amd64.iso -l com1,stdio -c 4 -m 1024M linuxguest wrmsr to register 0x1a0(0x801889) on vcpu 0 vm exit[0] reason VMX rip 0x000000000154b760 inst_length 3 status 0 exit_reason 2 qualification 0x0000000000000000 inst_type 0 inst_error 0 The problem is in bhyve_xmsr.c Patch attached.
Can you dump the MISC_ENABLE MSR value on the host? # kldload cpuctl # cpucontrol -m 0x1A0 /dev/cpuctl0
# cpucontrol -m 0x1A0 /dev/cpuctl0 MSR 0x1a0: 0x00000004 0x00850089
FYI, it's not possible for bhyve to emulate every MSR, since many of these accessed by guests are specific to particular CPU models. The "-w" option (described in the man page) allows unimplemented MSRs to be ignored. We usually look at MSRs on a case-by-case basis to determine if they are worth emulating or not - sometimes other unrelated issues are exposed.
Thanks. Can you also provide the output of the following from the host? # cpucontrol -i 0x80000001 /dev/cpuctl0
(In reply to Neel Natu from comment #4) > Thanks. Can you also provide the output of the following from the host? > > # cpucontrol -i 0x80000001 /dev/cpuctl0 # cpucontrol -i 0x80000001 /dev/cpuctl0 cpuid level 0x80000001: 0x00000000 0x00000000 0x00000001 0x28000800
(In reply to Peter Grehan from comment #3) > FYI, it's not possible for bhyve to emulate every MSR, since many of these > accessed by guests are specific to particular CPU models. The "-w" option > (described in the man page) allows unimplemented MSRs to be ignored. > > We usually look at MSRs on a case-by-case basis to determine if they are > worth emulating or not - sometimes other unrelated issues are exposed. This should be mentioned in the handbook, but it does work. I was sure I tried it before with no luck. Close at will and I'll bug the doc folk.
Diane, can you verify that the following patch fixes the problem for you? https://people.freebsd.org/~neel/patches/bhyve_nx_disabled.patch Of course, you'll want to do this without the "-w" option and also without your patch to xmsr.c.
(In reply to Neel Natu from comment #7) > Diane, can you verify that the following patch fixes the problem for you? > > https://people.freebsd.org/~neel/patches/bhyve_nx_disabled.patch > > Of course, you'll want to do this without the "-w" option and also without > your patch to xmsr.c. Of course ;) Rebuilding world and kernel now. I'll let you know.
(In reply to Diane Bruce from comment #8) > (In reply to Neel Natu from comment #7) > > Diane, can you verify that the following patch fixes the problem for you? > > > > https://people.freebsd.org/~neel/patches/bhyve_nx_disabled.patch > > > > Of course, you'll want to do this without the "-w" option and also without > > your patch to xmsr.c. > > Of course ;) > > Rebuilding world and kernel now. I'll let you know. It boots fine. Thanks!
Thanks for verifying the fix. The root cause was that the host system had the "no-execute" feature disabled (intentional?). This was passed through to the virtual machine and the guest kernel was trying to re-enable the feature by setting bit 34 of IA32_MISC_ENABLE msr. bhyve was not handling writes to the IA32_MISC_ENABLE msr which caused it to exit.
Also, in general you don't want to use the "-w" option to bhyve. Ignoring unimplemented MSRs might cause side-effects so it should only be used for debugging or as a quick and dirty workaround. Please file PRs instead :-)
A commit references this bug: Author: neel Date: Sat Dec 20 19:47:52 UTC 2014 New revision: 275965 URL: https://svnweb.freebsd.org/changeset/base/275965 Log: Emulate writes to the IA32_MISC_ENABLE MSR. PR: 196093 Reported by: db Tested by: db Discussed with: grehan MFC after: 1 week Changes: head/sys/amd64/vmm/intel/vmx_msr.c
(In reply to Neel Natu from comment #11) > Also, in general you don't want to use the "-w" option to bhyve. Ignoring > unimplemented MSRs might cause side-effects so it should only be used for > debugging or as a quick and dirty workaround. > > Please file PRs instead :-) I always intended to ;-) Close PR as you see fit. Thanks!
Hey, I seem to be hitting this on a revision well after the fix: r283896. The guest traps with: ... kernel trap 9 with interrupts disabled panic @ time 0.000, thread 0xffffffff813c2f20: Fatal trap 9: general protection fault while in kernel mode ... Registers: rax: 0000000000811889 rbx: ffffffff814077b8 rcx: 00000000000001a0 rdx: 0000000000000000 ^^^ MSR_MISC_ENABLE or IA32_MISC_ENABLE Are we missing some emulation? It looks like ECX is the MSR register, and we're trying to write EDX:EAX (per [0]): 00000000 00811889 Anything I can do to help fix this? Would you prefer a new bug? Thanks. [0]: http://x86.renejeschke.de/html/file_module_x86_id_326.html
Fwiw, the guest is attempting to configure Enhanced SpeedStep after detecting the host as SandyBridge{,-E} or IveBridge{,-E}. We are setting the SS_ENABLE bit and TB_DISABLE bits of IA32_MISC_ENABLE. Finally, we also write the IA32_PERF_CTL register (haven't gotten that far in boot yet, don't know if Bhyve handles that or not. Pretty much all of these should just be ignored rather than trapping, IMO.
Oh, also IA32_ENERGY_PERF_BIAS.
I guess Speedstep being disabled in CPUID isn't enough for some o/s versions :( Shouldn't be too hard to ignore those bits in MISC_ENABLE and sink the writes to PERF_CTL and ENERGY_PERF_BIAS. Would you be able to test out a patch ?
(In reply to Peter Grehan from comment #17) > I guess Speedstep being disabled in CPUID isn't enough for some o/s versions :( > Shouldn't be too hard to ignore those bits in MISC_ENABLE and sink the writes to PERF_CTL and ENERGY_PERF_BIAS. Would you be able to test out a patch ? Yep, I'm happy to test out a patch.