Bug 227880 - `kldstat -m` does not work for vmm.ko (while `kldstat -n` does)
Summary: `kldstat -m` does not work for vmm.ko (while `kldstat -n` does)
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-virtualization mailing list
URL: https://github.com/churchers/vm-bhyve...
Keywords: bhyve, needs-qa
Depends on:
Blocks:
 
Reported: 2018-04-30 20:21 UTC by Mateusz Piotrowski
Modified: 2018-05-11 03:16 UTC (History)
6 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mateusz Piotrowski freebsd_committer 2018-04-30 20:21:40 UTC
I've got vmm.ko and nmdm.ko loaded.

I can find out if nmdm.ko is loaded by running either `kldstat -n nmdm` or `kldstat -m nmdm`. When it comes to vmm.ko, however, only `kldstat -n vmm` works.

In case of `kldstat -m vmm` the following message is printed:

> kldstat: can't find module vmm: No such file or directory

Why is it so?

----

I'm running 12.0-CURRENT amd64 r333017.
Comment 1 Peter Grehan freebsd_committer 2018-04-30 22:49:11 UTC
Hmmm, works fine on a recent current. I'll upgrade and see if there's been a regression 

root@ryzen1:/home/grehan # uname -a
FreeBSD ryzen1 12.0-CURRENT FreeBSD 12.0-CURRENT #2 r332986M: Wed Apr 25 06:04:26 PDT 2018     grehan@ryzen1:/usr/obj/usr/home/grehan/proj/freebsd/current/amd64.amd64/sys/GENERIC  amd64
root@ryzen1:/home/grehan # kldload nmdm
root@ryzen1:/home/grehan # kldstat -m vmm
Id  Refs Name
515    1 vmm
root@ryzen1:/home/grehan # kldstat -m nmdm
Id  Refs Name
518    1 nmdm
root@ryzen1:/home/grehan # kldstat -n nmdm
Id Refs Address            Size     Name
 9    1 0xffffffff83d5d000 af0      nmdm.ko
root@ryzen1:/home/grehan # kldstat -m nmdm
Id  Refs Name
518    1 nmdm
root@ryzen1:/home/grehan # kldstat -n vmm
Id Refs Address            Size     Name
 8    1 0xffffffff83820000 53c440   vmm.ko
root@ryzen1:/home/grehan # kldstat -m vmm
Id  Refs Name
515    1 vmm
Comment 2 Ed Maste freebsd_committer 2018-05-01 01:57:08 UTC
I presume `kldstat -f` in the summary is a typo.

I tried this with a fresh -current build in QEMU; in that case the module loads but vmm fails:

SVM: Nested Paging feature not available.
module_register_init: MOD_LOAD (vmm, 0xffffffff82621550, 0) error 6
driver bug: Unable to set devclass (class: fdc devname: (unknown))

Here I get the same results as the initial report with kldstat -m / kldstat -n.  (Also, I haven't looked into the fdc error report triggered by trying to load vmm.)
Comment 3 Ed Maste freebsd_committer 2018-05-01 14:39:36 UTC
On my desktop at r333113 plus my WIP:

# kldstat -m vmm
Id  Refs Name
514    1 vmm
# kldstat -n vmm
Id Refs Address            Size     Name
12    1 0xffffffff832ee000 53a160   vmm.ko
Comment 4 Anish Gupta freebsd_committer 2018-05-01 16:53:14 UTC
> kldstat: can't find module vmm: No such file or directory
vmm module is not loaded, can you provide relevant console messages? For example in Ed case it failed with
" SVM: Nested Paging feature not available."

Bhyve works on processors with nested paging(EPT/NPT) support.
Comment 5 Mateusz Piotrowski freebsd_committer 2018-05-02 16:19:30 UTC
Machine 1
=========

 * Laptop (Yoga 3 14)
 * FreeBSD 12.0-CURRENT r332889
 * Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz

Everything works as expected.

----

Machine 2
=========

 * Virtual machine (libvirt guest)
 * FreeBSD 12.0-CURRENT r333017
 * AMD Opteron 63xx class CPU
 * libvirt host machine:
  * Debain 9.3
  * Linux devmachine 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux) 
  * AMD FX(tm)-8350 Eight-Core Processor) 

kldload vmm results in the following messages in /var/log/messages:

> May  2 18:10:46 freebsd kernel: SVM: not available.
> May  2 18:10:46 freebsd kernel: module_register_init: MOD_LOAD (vmm, 0xffffffff82c30490, 0) error 6
> May  2 18:10:46 freebsd kernel: driver bug: Unable to set devclass (class: fdc devname: (unknown))

Subsequent executions of kldstat look like this:

> $ kldstat -n vmm
> Id Refs Address            Size     Name
> 20    1 0xffffffff82c2c000 53c440   vmm.ko
> $ kldstat -m vmm
> kldstat: can't find module vmm: No such file or directory
Comment 6 Conrad Meyer freebsd_committer 2018-05-02 17:40:50 UTC
I can't explain the fdc driver bug -- that seems unrelated -- but this is the expected outcome of the vmm.ko *file* containing several modules: vmm, ppt, and ivhd, and only one of them failing to load (vmm).

The vmm.ko file will stay loaded unless all modules it contains fail to load.
Comment 7 Conrad Meyer freebsd_committer 2018-05-02 17:43:09 UTC
The usual thing to do here is to split vmm.ko into three separate .ko's.  I don't know the history here of why are one file.  Possibly just convenience.  If they share significant code, maybe space savings.  (Although, that last one can be solved by moving the shared code to a 4th module and depending on it from the other three.)
Comment 8 Conrad Meyer freebsd_committer 2018-05-02 17:52:36 UTC
(In reply to Conrad Meyer from comment #6)
To clarify, there is a distinction between a *module* which is something declared by (sometimes DRIVER_MODULE ->) DECLARE_MODULE -> MODULE_METADATA, and a loadable kernel *file* ("foo.ko"), which can contain 0 or more *module* objects.

Yes, confusingly they are often both referred to as "modules" and are often interchangeable.  However, vmm.ko happens to contain three kernel module objects, and the two that succeed to load keep the .ko loaded.

To confirm, try 'kldstat -m ppt' or 'kldstat -m ivhd' on the system that demonstrates the reported bug.

This is sort of a WONTFIX, but it's also possible the Bhyve folks want to use this confusion as a reason to split up vmm.ko.  Up to you guys.
Comment 9 Mateusz Piotrowski freebsd_committer 2018-05-02 18:06:09 UTC
(In reply to Conrad Meyer from comment #8)

On the system that demonstrates the reported bug:

    # kldunload vmm
    # kldload ppt
    kldload: can't load ppt: No such file or directory
    # kldload ivhd
    kldload: can't load ivhd: No such file or directory
    # kldload vmm
    # kldstat -m ppt
    kldstat: can't find module ppt: No such file or directory
    # kldstat -m ivhd
    kldstat: can't find module ivhd: No such file or directory
Comment 10 Rodney W. Grimes freebsd_committer 2018-05-02 18:09:00 UTC
(In reply to Conrad Meyer from comment #8)

Thank you for that insight on kernel modules, I was not aware that we could have multiple modules in a single file.  That helps clarify some things, and also brings up some questions on where else this is being done.

(In reply to Mateusz Piotrowski from comment #0)
Can you provide us some dmesg info from the host?
The vmm.ko/svm component complained at one point that you did not have nested page table support, that would prevent bhyve from working.

Looking for this:
CPU: Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz (2494.39-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306a9  Family=0x6  Model=0x3a  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7fbae3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16278343680 (15524 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <LENOVO TP-G2   >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
Comment 11 Peter Grehan freebsd_committer 2018-05-02 18:14:35 UTC
This is a bit of a lose-lose situation :( ppt was a separate kld at one point but caused even more problems with it not being loaded, having to deal with separate loads/unloads etc, so it was moved back into vmm. It has to be a module because it is a PCI device.

Fortunately the cases where vmm.ko is loaded and fails are few and far between these days.
Comment 12 Mateusz Piotrowski freebsd_committer 2018-05-02 20:39:54 UTC
(In reply to Rodney W. Grimes from comment #10)

Fragments of /var/log/messages from the FreeBSD guest:

    CPU: AMD Opteron 63xx class CPU (4027.12-MHz K8-class CPU)
      Origin="AuthenticAMD"  Id=0x600f20  Family=0x15  Model=0x2  Stepping=0
      Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
      Features2=0xbeb83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV>
      AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM>
      AMD Features2=0x2109e1<LAHF,ABM,SSE4A,MAS,Prefetch,XOP,FMA4,TBM>
    Hypervisor: Origin = "KVMKVMKVM"
    real memory  = 17179869184 (16384 MB)
    avail memory = 16576606208 (15808 MB)
    Event timer "LAPIC" quality 600
    ACPI APIC Table: <BOCHS  BXPCAPIC>
    FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
    FreeBSD/SMP: 8 package(s) x 1 core(s)

Fragments of /proc/cat from the Debian host:

    processor       : 7
    vendor_id       : AuthenticAMD
    cpu family      : 21
    model           : 2
    model name      : AMD FX(tm)-8350 Eight-Core Processor
    stepping        : 0
    microcode       : 0x600081c
    cpu MHz         : 1400.000
    cache size      : 2048 KB
    physical id     : 0
    siblings        : 8
    core id         : 7
    cpu cores       : 4
    apicid          : 7
    initial apicid  : 7
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
    bugs            : fxsave_leak sysret_ss_attrs null_seg
    bogomips        : 8054.23
    TLB size        : 1536 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro
Comment 13 Rodney W. Grimes freebsd_committer 2018-05-02 21:03:01 UTC
(In reply to Mateusz Piotrowski from comment #12)

Features2=0xbeb83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV>
      AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM>
      AMD Features2=0x2109e1<LAHF,ABM,SSE4A,MAS,Prefetch,XOP,FMA4,TBM>
    Hypervisor: Origin = "KVMKVMKVM"
                           ^^^^^^^^^

Your attempting to run bhyve inside of kvm?
That is nested hypervisors, and is most likely not supported.

    real memory  = 17179869184 (16384 MB)
    avail memory = 16576606208 (15808 MB)
    Event timer "LAPIC" quality 600
    ACPI APIC Table: <BOCHS  BXPCAPIC>
    FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
    FreeBSD/SMP: 8 package(s) x 1 core(s)

I see no VT-x information here, so you trying to run bhyve on a cpu that has no vt-x.  You might be able to configure kvm to provide this, not sure.
Comment 14 Conrad Meyer freebsd_committer 2018-05-02 21:13:37 UTC
(In reply to Rodney W. Grimes from comment #13)
Yeah, it's a nested hypervisor situation.  That was covered in comment #5.  Does not change the fact that vmm.ko load failure results in the half-loaded state.
Comment 15 Ryan Stone freebsd_committer 2018-05-11 03:16:27 UTC
(In reply to Peter Grehan from comment #11)

What if you made ppt.ko a separate module again, but had vmm.ko MODULE_DEPEND on it?  I think that would solve the problem of ppt.ko not being loaded but still allow vmm.ko to fail to load on an error.