Created attachment 180919 [details] Intel 860 CPU Supports VT-x and EPT Intel 860 CPU ( http://intel.ly/1pktvfh ) Supports VT-x and EPT/ Only Shows Partial Support FreeNAS Corral. Submitted a bug report and was told to submit a bug to FreeBSD. History #1 Updated by Steve Revo about 3 hours ago 1 added project (FreeNAS Corral) #2 Updated by Kris Moore about 3 hours ago 1 deleted project (FreeNAS Corral) Project changed from FreeNAS to FreeNAS Corral Category changed from Hardware to Hardware Branch set to master #3 Updated by Jordan Hubbard about 2 hours ago Quote As noted in the docs (https://wiki.freenas.org/index.php/Create_and_Manage_VMs), CPU must support VT-x and EPT. #4 Updated by Jordan Hubbard about 2 hours ago Status changed from Unscreened to User Configuration Issue #5 Updated by Nikita Ozerov about 2 hours ago File Intel® Core™ i7 860 Processor 8M Cache 2.80 GHz Product Specifications.png View added 9967 Quote According to information he has provided, his CPU does support VT-x and EPT. #6 Updated by Steve Revo about 1 hour ago Subject changed from Intel 860 CPU Supports VT-X / Only Shows Partial Support FreeNAS Corral to Intel 860 CPU Supports VT-x and EPT/ Only Shows Partial Support FreeNAS Corral Hardware Configuration updated (diff) #7 Updated by Steve Revo about 1 hour ago Priority changed from No priority to Expected Seen in changed from 10.0-M1 to RELEASE #8 Updated by Jordan Hubbard 19 minutes ago Quote Did Intel make multiple versions of this chip? Honestly, not sure what else to say - if FreeBSD kernel says it doesn't support this particular CPU, and it passes that information on to us, we have no other source of information and have to honor it. This might be a good FreeBSD bug report, since 11.1 is due to come out in June.
As far as I know Intel only made on version of the 860 processor and it supports VT-x and EPT
Where are you getting the information that your CPU does not support these features? Can you post dmesg? Is it possible that the CPU supports the features, but they are disabled in your BIOS? This is common with some other features.
FREENAS states that it's "partially" supported as opposed to "Full" I double checked the BIOS and states "enabled" dmesg: CPU: Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz (2660.05-MHz K8-class CPU) FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 cpu4: <ACPI CPU> on acpi0 cpu5: <ACPI CPU> on acpi0 cpu6: <ACPI CPU> on acpi0 cpu7: <ACPI CPU> on acpi0 coretemp0: <CPU On-Die Thermal Sensors> on cpu0 est0: <Enhanced SpeedStep Frequency Control> on cpu0 coretemp1: <CPU On-Die Thermal Sensors> on cpu1 est1: <Enhanced SpeedStep Frequency Control> on cpu1 coretemp2: <CPU On-Die Thermal Sensors> on cpu2 est2: <Enhanced SpeedStep Frequency Control> on cpu2 coretemp3: <CPU On-Die Thermal Sensors> on cpu3 est3: <Enhanced SpeedStep Frequency Control> on cpu3 coretemp4: <CPU On-Die Thermal Sensors> on cpu4 est4: <Enhanced SpeedStep Frequency Control> on cpu4 coretemp5: <CPU On-Die Thermal Sensors> on cpu5 est5: <Enhanced SpeedStep Frequency Control> on cpu5 coretemp6: <CPU On-Die Thermal Sensors> on cpu6 est6: <Enhanced SpeedStep Frequency Control> on cpu6 coretemp7: <CPU On-Die Thermal Sensors> on cpu7 est7: <Enhanced SpeedStep Frequency Control> on cpu7 SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #7 Launched!
(In reply to Steve Revo from comment #3) > FREENAS states that it's "partially" supported as opposed to "Full" Ok, that would be a FreeNAS bug and has nothing to do with FreeBSD... can they fill us in on what that reporting means for FreeBSD? I'm looking for lines like these from dmesg: Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,<b11>,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x121<LAHF,ABM,Prefetch> Standard Extended Features=0x21cbfbb<GSFSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ENHMOVSB,INVPCID,RTM,RDSEED,ADX,SMAP> TSC: P-state invariant, performance statistics Those are the features your CPU (well, my CPU) self-reports and that would be my guess for what FreeNAS is determining on. However, without more info here, this will be closed as not a FreeBSD bug.
Thank Conrad. What dmesg command did you run to get that info?
(In reply to Steve Revo from comment #5) That's just ordinary "dmesg".
I have included sysctl -a for you to review. http://pastebin.com/yNNV5W9S
I think this ticket may have been closed a little too hastily. The FreeBSD bug is right there in the sysctl output: hw.vmm.vmx.cap.unrestricted_guest: 0 However, http://ark.intel.com/products/41316/Intel-Core-i7-860-Processor-8M-Cache-2_80-GHz on this product claims otherwise. Reference FreeNAS ticket: https://bugs.freenas.org/issues/22362 FreeNAS relies completely on the sysctl output for hw.vmm.vmx / hw.vmm.svm (AMD) to tell it whether chipset supports VT-x, EPT and AMD-V CPU features respectively.
(In reply to Jordan Hubbard from comment #8) > I think this ticket may have been closed a little too hastily. No one was providing any actionable information. FreeBSD can't diagnose vague FreeNAS issues. Identify where you think the actual bug in FreeBSD is and we can fix it. > The FreeBSD bug is right there in the sysctl output: > > hw.vmm.vmx.cap.unrestricted_guest: 0 Great. Unfortunately, until you said that, it was completely unclear.
647 cap_unrestricted_guest = (vmx_set_ctlreg(MSR_VMX_PROCBASED_CTLS2, 648 MSR_VMX_PROCBASED_CTLS2, 649 PROCBASED2_UNRESTRICTED_GUEST, 0, 650 &tmp) == 0); There doesn't seem to be a lot for FreeBSD to be doing wrong here. Does Linux claim to support the unrestricted_guest feature on your CPU? Is it possible Ark is simply wrong or incomplete?
I have used this CPU for years with other operating systems, including linux without issues. I have also used with various hypervisors like XEN and VMWARE and KVM without incident. I just recently switched to FREEBSD 11 to run FREENAS Corral and started having these issues with my processors capability being identified incorrectly.
Maybe FreeNAS is just being overly paranoid with the check, if other hypervisors do not require it.
(In reply to Conrad Meyer from comment #12) Nope, nothing to do with paranoia. When FreeBSD returns this value, it means that bhyve cannot boot a guest VM which requires EPT (which is any UEFI guest, among others). This machine would be failing even if it was running straight-up, raw FreeBSD. FreeNAS has nothing to do with this issue. It's simply a wrapper around FreeBSD and bhyve where this is concerned, and when unrestricted_guest=0 bhyve falls over if asked to do what this user is trying to do. We just provide a seat belt to give a better error message than that.
I'd like to toss my hat into the ring on this one. I'm running an Intel i7 processor and using FreeNAS Corral 10.0.2 (latest as of this writing). I am encountering the exact same problem, but based on everything I can dig up, my CPU should not have this problem. Here is a cut and paste from my original post to the FreeNAS bug (https://bugs.freenas.org/issues/17689): ------------------------------------- I'm glad I stumbled upon this because I am having the exact same issue that you are having. I was trying to fire up a plexpass docker container and got the same error you've posted except I'm using 10.0.2 and any updates as of this morning. As you showed in the Intel CPU specs link, your CPU has VT-x support as does mine. For unrestricted guest access (access to ring 0: https://en.wikipedia.org/wiki/X86_virtualization#Intel_virtualization_.28VT-x.29) your CPU must support VT-x with Extended Page Tables (EPT), which the specs clearly show. Here is the output from my dmesg. Notice the next to last line, it shows VT-x with EPT: CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz (2672.78-MHz K8-class CPU) Origin="GenuineIntel" Id=0x106a4 Family=0x6 Model=0x1a Stepping=4 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x98e3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x1<LAHF> VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID TSC: P-state invariant, performance statistics Although it doesn't show anything but VMX flags (VMX and ring 0 explaination: https://software.intel.com/sites/default/files/m/4/1/9/7/c/25039-final_cpu_1027.pdf), cpuid gives the following output (if it's useful): [root@freenas] ~# cpuid eax in eax ebx ecx edx 00000000 0000000b 756e6547 6c65746e 49656e69 00000001 000106a4 03100800 0098e3bd bfebfbff 00000002 55035a01 00f0b2e4 00000000 09ca212c 00000003 00000000 00000000 00000000 00000000 00000004 1c004121 01c0003f 0000003f 00000000 00000005 00000040 00000040 00000003 00001120 00000006 00000003 00000002 00000001 00000000 00000007 00000000 00000000 00000000 00000000 00000008 00000000 00000000 00000000 00000000 00000009 00000000 00000000 00000000 00000000 0000000a 07300403 00000040 00000000 00000603 0000000b 00000001 00000002 00000100 00000003 80000000 80000008 00000000 00000000 00000000 80000001 00000000 00000000 00000001 28100800 80000002 65746e49 2952286c 726f4320 4d542865 80000003 37692029 55504320 20202020 20202020 80000004 30323920 20402020 37362e32 007a4847 80000005 00000000 00000000 00000000 00000000 80000006 00000000 00000000 01006040 00000000 80000007 00000000 00000000 00000000 00000100 80000008 00003024 00000000 00000000 00000000 Vendor ID: "GenuineIntel"; CPUID level 11 Intel-specific functions: Version 000106a4: Type 0 - Original OEM Family 6 - Pentium Pro Model 26 - Stepping 4 Reserved 0 Extended brand string: "Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz" CLFLUSH instruction cache line size: 8 Initial APIC ID: 3 Hyper threading siblings: 16 Feature flags set 1 (CPUID.01H:EDX): bfebfbff: FPU Floating Point Unit VME Virtual 8086 Mode Enhancements DE Debugging Extensions PSE Page Size Extensions TSC Time Stamp Counter MSR Model Specific Registers PAE Physical Address Extension MCE Machine Check Exception CX8 COMPXCHG8B Instruction APIC On-chip Advanced Programmable Interrupt Controller present and enabled SEP Fast System Call MTRR Memory Type Range Registers PGE PTE Global Flag MCA Machine Check Architecture CMOV Conditional Move and Compare Instructions FGPAT Page Attribute Table PSE-36 36-bit Page Size Extension CLFSH CFLUSH instruction DS Debug store ACPI Thermal Monitor and Clock Ctrl MMX MMX instruction set FXSR Fast FP/MMX Streaming SIMD Extensions save/restore SSE Streaming SIMD Extensions instruction set SSE2 SSE2 extensions SS Self Snoop HT Hyper Threading TM Thermal monitor 31 Pending Break Enable Feature flags set 2 (CPUID.01H:ECX): 0098e3bd: SSE3 SSE3 extensions DTES64 64-bit debug store MONITOR MONITOR/MWAIT instructions DS-CPL CPL Qualified Debug Store VMX Virtual Machine Extensions EST Enhanced Intel SpeedStep Technology TM2 Thermal Monitor 2 SSSE3 Supplemental Streaming SIMD Extension 3 [root@freenas] ~# cpuid|less eax in eax ebx ecx edx 00000000 0000000b 756e6547 6c65746e 49656e69 00000001 000106a4 06100800 0098e3bd bfebfbff 00000002 55035a01 00f0b2e4 00000000 09ca212c 00000003 00000000 00000000 00000000 00000000 00000004 1c004121 01c0003f 0000003f 00000000 00000005 00000040 00000040 00000003 00001120 00000006 00000003 00000002 00000001 00000000 00000007 00000000 00000000 00000000 00000000 00000008 00000000 00000000 00000000 00000000 00000009 00000000 00000000 00000000 00000000 0000000a 07300403 00000040 00000000 00000603 0000000b 00000001 00000002 00000100 00000006 80000000 80000008 00000000 00000000 00000000 80000001 00000000 00000000 00000001 28100800 80000002 65746e49 2952286c 726f4320 4d542865 80000003 37692029 55504320 20202020 20202020 80000004 30323920 20402020 37362e32 007a4847 80000005 00000000 00000000 00000000 00000000 80000006 00000000 00000000 01006040 00000000 80000007 00000000 00000000 00000000 00000100 80000008 00003024 00000000 00000000 00000000 Vendor ID: "GenuineIntel"; CPUID level 11 Intel-specific functions: Version 000106a4: Type 0 - Original OEM Family 6 - Pentium Pro Model 26 - Stepping 4 Reserved 0 Extended brand string: "Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz" CLFLUSH instruction cache line size: 8 Initial APIC ID: 6 Hyper threading siblings: 16 Feature flags set 1 (CPUID.01H:EDX): bfebfbff: FPU Floating Point Unit VME Virtual 8086 Mode Enhancements DE Debugging Extensions PSE Page Size Extensions TSC Time Stamp Counter MSR Model Specific Registers PAE Physical Address Extension MCE Machine Check Exception CX8 COMPXCHG8B Instruction APIC On-chip Advanced Programmable Interrupt Controller present and enabled SEP Fast System Call MTRR Memory Type Range Registers PGE PTE Global Flag MCA Machine Check Architecture CMOV Conditional Move and Compare Instructions FGPAT Page Attribute Table PSE-36 36-bit Page Size Extension CLFSH CFLUSH instruction DS Debug store ACPI Thermal Monitor and Clock Ctrl MMX MMX instruction set FXSR Fast FP/MMX Streaming SIMD Extensions save/restore SSE Streaming SIMD Extensions instruction set SSE2 SSE2 extensions SS Self Snoop HT Hyper Threading TM Thermal monitor 31 Pending Break Enable Feature flags set 2 (CPUID.01H:ECX): 0098e3bd: SSE3 SSE3 extensions DTES64 64-bit debug store MONITOR MONITOR/MWAIT instructions DS-CPL CPL Qualified Debug Store VMX Virtual Machine Extensions EST Enhanced Intel SpeedStep Technology TM2 Thermal Monitor 2 SSSE3 Supplemental Streaming SIMD Extension 3 Intel-specific functions: Version 000106a4: Type 0 - Original OEM Family 6 - Pentium Pro Model 26 - Stepping 4 Reserved 0 Extended brand string: "Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz" CLFLUSH instruction cache line size: 8 Initial APIC ID: 6 Hyper threading siblings: 16 Feature flags set 1 (CPUID.01H:EDX): bfebfbff: FPU Floating Point Unit VME Virtual 8086 Mode Enhancements DE Debugging Extensions PSE Page Size Extensions TSC Time Stamp Counter MSR Model Specific Registers PAE Physical Address Extension MCE Machine Check Exception CX8 COMPXCHG8B Instruction APIC On-chip Advanced Programmable Interrupt Controller present and enabled SEP Fast System Call MTRR Memory Type Range Registers PGE PTE Global Flag MCA Machine Check Architecture CMOV Conditional Move and Compare Instructions FGPAT Page Attribute Table PSE-36 36-bit Page Size Extension CLFSH CFLUSH instruction DS Debug store ACPI Thermal Monitor and Clock Ctrl MMX MMX instruction set FXSR Fast FP/MMX Streaming SIMD Extensions save/restore SSE Streaming SIMD Extensions instruction set SSE2 SSE2 extensions SS Self Snoop HT Hyper Threading TM Thermal monitor 31 Pending Break Enable Feature flags set 2 (CPUID.01H:ECX): 0098e3bd: SSE3 SSE3 extensions DTES64 64-bit debug store MONITOR MONITOR/MWAIT instructions DS-CPL CPL Qualified Debug Store VMX Virtual Machine Extensions EST Enhanced Intel SpeedStep Technology TM2 Thermal Monitor 2 SSSE3 Supplemental Streaming SIMD Extension 3 CX16 CMPXCHG16B xTPR Send Task Priority messages PDCM Perfmon and debug capability SSE4.1 Streaming SIMD Extension 4.1 SSE4.2 Streaming SIMD Extension 4.2 POPCNT POPCNT instruction Extended feature flags set 1 (CPUID.80000001H:EDX): 28100800 SYSCALL SYSCALL/SYSRET instructions XD-bit Execution Disable bit RDTSCP RDTSCP and IA32_TSC_AUX are available EM64T Intel Extended Memory 64 Technology Extended feature flags set 2 (CPUID.80000001H:ECX): 00000001 LAHF LAHF/SAHF available in IA-32e mode Old-styled TLB and cache info: 5a: Data TLB: 2MB or 4MB pages, 4-way set associative, 32 entries 03: Data TLB: 4KB pages, 4-way set assoc, 64 entries 55: Instruction TLB: 2MB or 4MB pages, fully assoc., 7 entries e4: 3rd-level cache: 8MB, 16-way set associative, 64-byte line size b2: Instruction TLB: 4-KB Pages, 4-way set associative, 64 entries f0: 64-byte prefetching 2c: 1st-level data cache: 32-KB, 8-way set associative, 64-byte line size 21: 256-KB L2 (MLC), 8-way set associative, 64 byte line size ca: Shared 2nd-level TLB: 4-KB Pages, 4-way set associative, 512 entries 09: 1st-level instruction cache: 32KB, 4-way set assoc, 64 byte line size Processor serial: 0001-06A4-0000-0000-0000-0000 Deterministic Cache Parameters: index=0: eax=1c004121 ebx=01c0003f ecx=0000003f edx=00000000 > Data cache, level 1, self initializing > 64 sets, 8 ways, 1 partitions, line size 64 > full size 32768 bytes > shared between up to 2 threads > NB this package has up to 8 threads index=1: eax=1c004122 ebx=00c0003f ecx=0000007f edx=00000000 > Instruction cache, level 1, self initializing > 128 sets, 4 ways, 1 partitions, line size 64 > full size 32768 bytes > shared between up to 2 threads index=2: eax=1c004143 ebx=01c0003f ecx=000001ff edx=00000000 > Unified cache, level 2, self initializing > 512 sets, 8 ways, 1 partitions, line size 64 > full size 262144 bytes > shared between up to 2 threads index=3: eax=1c03c163 ebx=03c0003f ecx=00001fff edx=00000002 > Unified cache, level 3, self initializing > 8192 sets, 16 ways, 1 partitions, line size 64 > full size 8388608 bytes > shared between up to 16 threads ------------------------------------- The thing that has me confused is dmesg clearly shows VT-x with EPT, but hw.vmm.vmx.cap.unrestricted_guest returns 0: [root@freenas] ~# sysctl hw.vmm.vmx.cap.unrestricted_guest hw.vmm.vmx.cap.unrestricted_guest: 0 Please let me know if you need any other information. Thanks for looking into this for us.
Same issue here on an Intel X5560.
So after more research and talking with the FreeNAS guys it suddenly clicked for me. To work properly you need VT-x with EPT to be able to use unrestricted guests. Great, dmesg clearly shows that for me: VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID However, someone posted theirs and it looked like this: VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID It finally dawned on me the UG flag stands for "Unrestricted Guest". So, hopefully this will help everyone else in the same situation as we were, being, Intel specs said we have VT-x and EPT (as did FreeBSD), but the CPU flags didn't have UG. Upon more investigation it appears that UG came about in the Westmere and later processors. Mine was Bloomfield, the architecture just previous. So, I bought an Intel i7 860x which should have UG support. Many thanks guys and hopefully this will help clear up the issue for others and give a more clear answer as to why in the future.
Thanks Nathan Benson for investigating (comment #16). The CPU needs to support VT-x, EPT, and UG (Unrestricted Guest). See https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html for more information, and specifically this part: Hosting Linux® guests or FreeBSD guests with more than one vCPU requires VMX unrestricted mode support (UG). Most newer processors, specifically the Intel® Core™ i3/i5/i7 and Intel® Xeon™ E3/E5/E7, support these features. UG support was introduced with Intel's Westmere micro-architecture.