Created attachment 145450 [details] 9.3-RELEASE boot failure on 11.0-CURRENT r269644 When trying to boot FreeBSD 9.3-RELEASE or 10.0-RELEASE ISO in bhyve on 11.0-CURRENT r269644, I get "panic: CPU0 does not support X87 or SSE: 1". I tried the plainest possible configuration, using the sample vmrun.sh script. Booting Linux on my CURRENT machine works. Booting 9.3-RELEASE in bhyve on 10.0-RELEASE works. I'm not sure when this broke for me and it could have been months ago. I can help bisect or diagnose this if I'm pointed to some place to start. Full console log trying to boot 9.3-RELEASE on 11.0-CURRENT is attached.
Host CPU info from dmesg: CPU: Intel(R) Xeon(R) CPU E31235 @ 3.20GHz (3200.09-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206a7 Family=0x6 Model=0x2a Stepping=7 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,C MOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x1<LAHF> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics
Created attachment 146376 [details] sysctl hw.vmm output of "sysctl hw.vmm"
Created attachment 146377 [details] panic booting CURRENT This is trying to boot CURRENT snapshot, r270458
Created attachment 146380 [details] dmesg from CURRENT boot The boot sometimes works. Trying to boot CURRENT over and over again, I was able to boot it twice out of many tries. Here is a dmesg output from a successful boot. If FreeBSD boots, it works normally.
Since booting actually works once in a while, I tried pinning vcpu 0 to hostcpu 0. If I pin vcpu 0 to hostcpu 0, boot works every time.
Just to confirm the negative: does pinning to CPUs other than 0 result in the guest crashing ? If so, would you be able to install the cpuid port (pkg install cpuid) and run it on all cpus cpuset -l <cpu> cpuid .. and see if there are any differences in what is reported between CPUs, other than obvious ones such as APIC ID. later, Peter.
Pinning vcpu 0 on hostcpu 1 does not allow boot. It seems that all cpus other than 0 produce a similar diff. I'm about to attach it.
Created attachment 146425 [details] cpuid difference
Comment on attachment 146425 [details] cpuid difference Thanks - I think this explains it. -Vendor ID: "GenuineIntel"; CPUID level 13 +Vendor ID: "GenuineIntel"; CPUID level 2 The AP is coming up with a reduced number of CPU leaves compared to the BSP. FreeBSD (and bhyve) assumes that this is the same in all CPUs. FreeBSD itself pulls all information about the system from the BSP's leaf contents, and then assumes this is the same for all other CPUs. bhyve's assumption is that it can call do_cpuid() on any CPU and get the same information to pass to guests on CPUID exits. A lot of applications also assume this and will look at CPUID values to determine whether they can use SSE/SSE2/AVX libraries etc. What system is this ? Are there BIOS options to change max leaf values ? Is it running nested ? later, Peter.
This is exactly it! I had a BIOS setting called "Limit Max CPUID". After I turned it off, every core shows the same features and bhyve works reliably. It's still interesting that bhyve works despite this misconfiguration in 10.0, but not on CURRENT.
This issue can be closed. It's not valuable to fix, but anyone with a similar problem can find this bug report.