12.0-ALPHA8 on Phenom-II crashes with unemulated "clflush BYTE PTR [ebx] Host: ----- FreeBSD 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27 08:16:24 UTC 2018 amd64 CPU: AMD Phenom(tm) II X6 1075T Processor (3013.98-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x100fa0 Family=0x10 Model=0xa Stepping=0 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x802009<SSE3,MON,CX16,POPCNT> AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!> AMD Features2=0x837ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,NodeId> SVM: NP,NRIP,NAsids=64 TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33279193088 (31737 MB) Guest: ------ 12.0-ALPHA8-amd64-20180928-r338991 Result: ------- Host: ----- Launching virtual machine "vm14_fbsd" ... Failed to emulate instruction [0x0f 0xae 0x3b 0x8b 0x04 0x25 0xf8 0x5d 0xb8 0x81 0x48 0x01 0xc3 0x4c 0x39] at 0xffffffff8104abb0 Guest: ------ CPU: AMD Phenom(tm) II X6 1075T Processor (3013.42-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x100fa0 Family=0x10 Model=0xa Stepping=0 Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT> Features2=0x80802001<SSE3,CX16,POPCNT,HV> AMD Features=0xe6500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,LM,3DNow!+,3DNow!> AMD Features2=0x31fb<LAHF,CMP,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,SKINIT,WDT> TSC: P-state invariant [...] ACPI APIC Table: <BHYVE BVMADT > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 4 package(s) x 1 core(s) arc4random: no preloaded entropy cache
I suspect that the issue was uncovered by base r338807. It seems that CPUID_CLFSH feature is advertised to the guest because the host supports it. But there is some special code in initializecpucache() that clears the flag for VM guests. But ifuncs are resolved before initializecpucache() is called. So, clflush gets enabled since the switch to ifuncs. I guess that if bhyve does not emulate clflush and if the hardware does not support it in virtualized mode, then bhyve should clear CPUID_CLFSH.
Also, it seems that hw.clflush_disable=1, which is also handled by initializecpucache(), would be ignored now.
Created attachment 197961 [details] Tweak cache features bits before ifuncs are resolved. The patch should be applied to the guest. Lack of the clflush emulation is still the bug.
Also see https://reviews.freebsd.org/D17482
I do think we'd like to get both patches into 12.0 so that existing 10.x/11.x hosts can run 12.0 guests.
A commit references this bug: Author: kib Date: Fri Oct 12 15:30:15 UTC 2018 New revision: 339331 URL: https://svnweb.freebsd.org/changeset/base/339331 Log: bhyve: emulate CLFLUSH and CLFLUSHOPT. Apparently CLFLUSH on mmio can cause VM exit, as reported in the PR. I do not see that anything useful can be done except emulating page faults on invalid addresses. Due to the instruction encoding pecularity, also emulate SFENCE. PR: 232081 Reported by: phk Reviewed by: araujo, avg, jhb (all: previous version) Sponsored by: The FreeBSD Foundation Approved by: re (gjb) MFC after: 1 week Differential revision: https://reviews.freebsd.org/D17482 Changes: head/sys/amd64/vmm/vmm_instruction_emul.c
A commit references this bug: Author: kib Date: Fri Oct 12 16:00:21 UTC 2018 New revision: 339333 URL: https://svnweb.freebsd.org/changeset/base/339333 Log: Call initializecpucache() before ifuncs are resolved. The function tweaks CPU capabilities based on the VM platform and tunables, which affected selection of the cache flush method before ifuncs were used, and should affect the cache flush in the same way after ifunc. PR: 232081 Reported by: phk Analyzed by: avg Sponsored by: The FreeBSD Foundation Approved by: re (gjb) Changes: head/sys/amd64/amd64/machdep.c
Tried FreeBSD 12.0-ALPHA10 as guest, and that boots fine Thanks!