Bug 227880
| Summary: | `kldstat -m` does not work for vmm.ko (while `kldstat -n` does) | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Mateusz Piotrowski <0mp> |
| Component: | misc | Assignee: | freebsd-virtualization (Nobody) <virtualization> |
| Status: | New --- | ||
| Severity: | Affects Only Me | CC: | anish, cem, emaste, fbsd44, grehan, rgrimes, rstone |
| Priority: | --- | Keywords: | bhyve, needs-qa |
| Version: | CURRENT | ||
| Hardware: | Any | ||
| OS: | Any | ||
| URL: | https://github.com/churchers/vm-bhyve/issues/199 | ||
Hmmm, works fine on a recent current. I'll upgrade and see if there's been a regression root@ryzen1:/home/grehan # uname -a FreeBSD ryzen1 12.0-CURRENT FreeBSD 12.0-CURRENT #2 r332986M: Wed Apr 25 06:04:26 PDT 2018 grehan@ryzen1:/usr/obj/usr/home/grehan/proj/freebsd/current/amd64.amd64/sys/GENERIC amd64 root@ryzen1:/home/grehan # kldload nmdm root@ryzen1:/home/grehan # kldstat -m vmm Id Refs Name 515 1 vmm root@ryzen1:/home/grehan # kldstat -m nmdm Id Refs Name 518 1 nmdm root@ryzen1:/home/grehan # kldstat -n nmdm Id Refs Address Size Name 9 1 0xffffffff83d5d000 af0 nmdm.ko root@ryzen1:/home/grehan # kldstat -m nmdm Id Refs Name 518 1 nmdm root@ryzen1:/home/grehan # kldstat -n vmm Id Refs Address Size Name 8 1 0xffffffff83820000 53c440 vmm.ko root@ryzen1:/home/grehan # kldstat -m vmm Id Refs Name 515 1 vmm I presume `kldstat -f` in the summary is a typo. I tried this with a fresh -current build in QEMU; in that case the module loads but vmm fails: SVM: Nested Paging feature not available. module_register_init: MOD_LOAD (vmm, 0xffffffff82621550, 0) error 6 driver bug: Unable to set devclass (class: fdc devname: (unknown)) Here I get the same results as the initial report with kldstat -m / kldstat -n. (Also, I haven't looked into the fdc error report triggered by trying to load vmm.) On my desktop at r333113 plus my WIP: # kldstat -m vmm Id Refs Name 514 1 vmm # kldstat -n vmm Id Refs Address Size Name 12 1 0xffffffff832ee000 53a160 vmm.ko > kldstat: can't find module vmm: No such file or directory
vmm module is not loaded, can you provide relevant console messages? For example in Ed case it failed with
" SVM: Nested Paging feature not available."
Bhyve works on processors with nested paging(EPT/NPT) support.
Machine 1 ========= * Laptop (Yoga 3 14) * FreeBSD 12.0-CURRENT r332889 * Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz Everything works as expected. ---- Machine 2 ========= * Virtual machine (libvirt guest) * FreeBSD 12.0-CURRENT r333017 * AMD Opteron 63xx class CPU * libvirt host machine: * Debain 9.3 * Linux devmachine 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux) * AMD FX(tm)-8350 Eight-Core Processor) kldload vmm results in the following messages in /var/log/messages: > May 2 18:10:46 freebsd kernel: SVM: not available. > May 2 18:10:46 freebsd kernel: module_register_init: MOD_LOAD (vmm, 0xffffffff82c30490, 0) error 6 > May 2 18:10:46 freebsd kernel: driver bug: Unable to set devclass (class: fdc devname: (unknown)) Subsequent executions of kldstat look like this: > $ kldstat -n vmm > Id Refs Address Size Name > 20 1 0xffffffff82c2c000 53c440 vmm.ko > $ kldstat -m vmm > kldstat: can't find module vmm: No such file or directory I can't explain the fdc driver bug -- that seems unrelated -- but this is the expected outcome of the vmm.ko *file* containing several modules: vmm, ppt, and ivhd, and only one of them failing to load (vmm). The vmm.ko file will stay loaded unless all modules it contains fail to load. The usual thing to do here is to split vmm.ko into three separate .ko's. I don't know the history here of why are one file. Possibly just convenience. If they share significant code, maybe space savings. (Although, that last one can be solved by moving the shared code to a 4th module and depending on it from the other three.) (In reply to Conrad Meyer from comment #6) To clarify, there is a distinction between a *module* which is something declared by (sometimes DRIVER_MODULE ->) DECLARE_MODULE -> MODULE_METADATA, and a loadable kernel *file* ("foo.ko"), which can contain 0 or more *module* objects. Yes, confusingly they are often both referred to as "modules" and are often interchangeable. However, vmm.ko happens to contain three kernel module objects, and the two that succeed to load keep the .ko loaded. To confirm, try 'kldstat -m ppt' or 'kldstat -m ivhd' on the system that demonstrates the reported bug. This is sort of a WONTFIX, but it's also possible the Bhyve folks want to use this confusion as a reason to split up vmm.ko. Up to you guys. (In reply to Conrad Meyer from comment #8) On the system that demonstrates the reported bug: # kldunload vmm # kldload ppt kldload: can't load ppt: No such file or directory # kldload ivhd kldload: can't load ivhd: No such file or directory # kldload vmm # kldstat -m ppt kldstat: can't find module ppt: No such file or directory # kldstat -m ivhd kldstat: can't find module ivhd: No such file or directory (In reply to Conrad Meyer from comment #8) Thank you for that insight on kernel modules, I was not aware that we could have multiple modules in a single file. That helps clarify some things, and also brings up some questions on where else this is being done. (In reply to Mateusz Piotrowski from comment #0) Can you provide us some dmesg info from the host? The vmm.ko/svm component complained at one point that you did not have nested page table support, that would prevent bhyve from working. Looking for this: CPU: Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz (2494.39-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306a9 Family=0x6 Model=0x3a Stepping=9 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7fbae3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x1<LAHF> Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS> XSAVE Features=0x1<XSAVEOPT> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 17179869184 (16384 MB) avail memory = 16278343680 (15524 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <LENOVO TP-G2 > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads This is a bit of a lose-lose situation :( ppt was a separate kld at one point but caused even more problems with it not being loaded, having to deal with separate loads/unloads etc, so it was moved back into vmm. It has to be a module because it is a PCI device. Fortunately the cases where vmm.ko is loaded and fails are few and far between these days. (In reply to Rodney W. Grimes from comment #10) Fragments of /var/log/messages from the FreeBSD guest: CPU: AMD Opteron 63xx class CPU (4027.12-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x600f20 Family=0x15 Model=0x2 Stepping=0 Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2> Features2=0xbeb83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV> AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM> AMD Features2=0x2109e1<LAHF,ABM,SSE4A,MAS,Prefetch,XOP,FMA4,TBM> Hypervisor: Origin = "KVMKVMKVM" real memory = 17179869184 (16384 MB) avail memory = 16576606208 (15808 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <BOCHS BXPCAPIC> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 8 package(s) x 1 core(s) Fragments of /proc/cat from the Debian host: processor : 7 vendor_id : AuthenticAMD cpu family : 21 model : 2 model name : AMD FX(tm)-8350 Eight-Core Processor stepping : 0 microcode : 0x600081c cpu MHz : 1400.000 cache size : 2048 KB physical id : 0 siblings : 8 core id : 7 cpu cores : 4 apicid : 7 initial apicid : 7 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold bugs : fxsave_leak sysret_ss_attrs null_seg bogomips : 8054.23 TLB size : 1536 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro (In reply to Mateusz Piotrowski from comment #12) Features2=0xbeb83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV> AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM> AMD Features2=0x2109e1<LAHF,ABM,SSE4A,MAS,Prefetch,XOP,FMA4,TBM> Hypervisor: Origin = "KVMKVMKVM" ^^^^^^^^^ Your attempting to run bhyve inside of kvm? That is nested hypervisors, and is most likely not supported. real memory = 17179869184 (16384 MB) avail memory = 16576606208 (15808 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <BOCHS BXPCAPIC> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 8 package(s) x 1 core(s) I see no VT-x information here, so you trying to run bhyve on a cpu that has no vt-x. You might be able to configure kvm to provide this, not sure. (In reply to Rodney W. Grimes from comment #13) Yeah, it's a nested hypervisor situation. That was covered in comment #5. Does not change the fact that vmm.ko load failure results in the half-loaded state. (In reply to Peter Grehan from comment #11) What if you made ppt.ko a separate module again, but had vmm.ko MODULE_DEPEND on it? I think that would solve the problem of ppt.ko not being loaded but still allow vmm.ko to fail to load on an error. Using 12.1-P8 in August of 2020, this is still happening. cbsd, and possibly other ports, rely on /sbin/kldstat -qm vmm to return an exit status of 0. kldstat -m does not recognize vmm. If this is the intended behavior of kldstat with the vmm.ko module, then please add the appropriate comments and close this report. Thanks for all of the awesome work you do! (In reply to Bob Martin from comment #16) Can you be more specific? kldstat -qm vmm *does* return zero if the module is present. If kldstat does not recognize vmm, it's not present. At least in the earlier reports in this bug, kldstat is functioning as intended; vmm.ko could behave more nicely. |
I've got vmm.ko and nmdm.ko loaded. I can find out if nmdm.ko is loaded by running either `kldstat -n nmdm` or `kldstat -m nmdm`. When it comes to vmm.ko, however, only `kldstat -n vmm` works. In case of `kldstat -m vmm` the following message is printed: > kldstat: can't find module vmm: No such file or directory Why is it so? ---- I'm running 12.0-CURRENT amd64 r333017.