Bug 228773 - hang on boot before mounting / when 'verbose boot' not selected
Summary: hang on boot before mounting / when 'verbose boot' not selected
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: arm (show other bugs)
Version: CURRENT
Hardware: arm Any
: --- Affects Only Me
Assignee: freebsd-arm mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-06-06 05:00 UTC by Bob Frazier
Modified: 2018-08-24 18:24 UTC (History)
0 users

See Also:


Attachments
unedited console output for VERBOSE_SYSINIT (50.27 KB, text/plain)
2018-06-09 21:34 UTC, Bob Frazier
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Bob Frazier 2018-06-06 05:00:27 UTC
using rev 334665 on an RPi 2 using the older bcm2836 CPU.

I have been able to consistently demonstrate this problem more than half of the time when I boot up from a power up, or a 'reboot', or by pressing a key in the serial console after 'poweroff'.

The hang appears to happen immediately after the following text displays on the console

  Releasing APs

Consistently, whenever I enable verbose output as a boot option in the 'beastie' menu, the reboot takes place normally, i.e. without hanging.

uname output:
FreeBSD pi2c 12.0-CURRENT FreeBSD 12.0-CURRENT #2 r334665M: Tue Jun  5 21:17:17 PDT 2018     bobf@hack.SFT.local:/e-drive/obj.current/e-drive/src.current/arm.armv7/sys/RPI2  arm

The problem first appeared when I obtained source yesterday and built a fresh version.  Prior to this I had been using code that was around 3 or 4 weeks old.


This problem was reproduced several times with unmodified code and no additional overlays or kernel modules being loaded.


Typical console output demonstrating problem:

/boot/dtb/bcm2836-rpi-2-b.dtb size=0x4c49
Loaded DTB from file 'bcm2836-rpi-2-b.dtb'.
Kernel entry at 0x1200100...
Kernel args: (null)
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #2 r334665M: Tue Jun  5 21:17:17 PDT 2018
    bobf@hack.SFT.local:/e-drive/obj.current/e-drive/src.current/arm.armv7/sys/RPI2 arm
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
CPU: ARM Cortex-A7 r0p5 (ECO: 0x00000000)
CPU Features: 
  Multiprocessing, Thumb2, Security, Virtualization, Generic Timer, VMSAv7,
  PXN, LPAE, Coherent Walk
Optional instructions: 
  SDIV/UDIV, UMULL, SMULL, SIMD(ext)
LoUU:2 LoC:3 LoUIS:2 
Cache level 1:
 32KB/64B 4-way data cache WB Read-Alloc Write-Alloc
 32KB/32B 2-way instruction cache Read-Alloc
Cache level 2:
 512KB/64B 8-way unified cache WB Read-Alloc Write-Alloc
real memory  = 0 (0 MB)
avail memory = 1008672768 (961 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
random: unblocking device.
random: entropy device external interface
kbd0 at kbdmux0
ofwbus0: <Open Firmware Device Tree>
simplebus0: <Flattened device tree simple bus> on ofwbus0
local_intc0: <BCM2836 Interrupt Controller> mem 0x40000000-0x400000ff on simplebus0
intc0: <BCM2835 Interrupt Controller> mem 0x7e00b200-0x7e00b3ff irq 25 on simplebus0
gpio0: <BCM2708/2835 GPIO controller> mem 0x7e200000-0x7e2000b3 irq 27,28,29,30 on simplebus0
gpio0: read-only pins: 46-53.
gpio0: reserved pins: 48-53.
gpiobus0: <OFW GPIO bus> on gpio0
generic_timer0: <ARMv7 Generic Timer> irq 1,2,3,4 on ofwbus0
Timecounter "ARM MPCore Timecounter" frequency 19200000 Hz quality 1000
Event timer "ARM MPCore Eventtimer" frequency 19200000 Hz quality 1000
bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007eff irq 9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24 on simplebus0
bcmwd0: <BCM2708/2835 Watchdog> mem 0x7e100000-0x7e100027 on simplebus0
bcmrng0: <Broadcom BCM2835 RNG> mem 0x7e104000-0x7e10400f on simplebus0
mbox0: <BCM2835 VideoCore Mailbox> mem 0x7e00b880-0x7e00b8bf irq 26 on simplebus0
gpioc0: <GPIO controller> on gpio0
uart0: <PrimeCell UART (PL011)> mem 0x7e201000-0x7e201fff irq 31 on simplebus0
uart0: console (115200,n,8,1)
spi0: <BCM2708/2835 SPI controller> mem 0x7e204000-0x7e204fff irq 33 on simplebus0
spibus0: <OFW SPI bus> on spi0
iichb0: <BCM2708/2835 BSC controller> mem 0x7e205000-0x7e205fff irq 34 on simplebus0
sdhci_bcm0: <Broadcom 2708 SDHCI controller> mem 0x7e300000-0x7e3000ff irq 41 on simplebus0
mmc0: <MMC/SD bus> on sdhci_bcm0
iichb1: <BCM2708/2835 BSC controller> mem 0x7e804000-0x7e804fff irq 44 on simplebus0
iichb2: <BCM2708/2835 BSC controller> mem 0x7e805000-0x7e805fff irq 45 on simplebus0
bcm283x_dwcotg0: <DWC OTG 2.0 integrated USB controller (bcm283x)> mem 0x7e980000-0x7e98ffff irq 50 on simplebus0
usbus0 on bcm283x_dwcotg0
vchiq0: <BCM2835 VCHIQ> mem 0x7e00b840-0x7e00b84e irq 52 on simplebus0
vchiq: local ver 8 (min 3), remote ver 8.
pcm0: <VCHIQ audio> on vchiq0
fb0: <BCM2835 VT framebuffer driver> on simplebus0
fbd0 on fb0
VT: initialize with new VT driver "fb".
fb0: 656x416(656x416@0,0) 24bpp
fb0: fbswap: 0, pitch 1968, base rx3ef34000, screen_size 818688
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
bcm2835_cpufreq0: <CPU Frequency Control> on cpu0
cpu1: <Open Firmware CPU> on cpulist0
cpu2: <Open Firmware CPU> on cpulist0
cpu3: <Open Firmware CPU> on cpulist0
gpioled0: <GPIO LEDs> on ofwbus0
cryptosoft0: <software crypto>
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
iicbus0: <OFW I2C bus> on iichb0
iic0: <I2C generic I/O> on iicbus0
ugen0.1: <DWCOTG OTG Root HUB> at usbus0
uhub0: <DWCOTG OTG Root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
mmcsd0: 4GB <SDHC SS04G 8.0 SN 032B0249 MFG 11/2014 by 3 SD> at mmc0 50.0MHz/4bit/65535-block
iicbus1: <OFW I2C bus> on iichb1
iic1: <I2C generic I/O> on iicbus1
iicbus2: <OFW I2C bus> on iichb2
iic2: <I2C generic I/O> on iicbus2
bcm2835_cpufreq0: ARM 600MHz, Core 250MHz, SDRAM 400MHz, Turbo OFF
Release APs

(this is where the hang takes place; no response to keyboard input on the serial console at this point.  power cycle is only option)
Comment 1 Bob Frazier 2018-06-08 17:56:00 UTC
was able to type ~ then ctrl+b and dropped into the debugger

KDB: enter: Break to debugger
[ thread pid 10 tid 100002 ]
Stopped at      $d.5:   ldrb    r15, [r15, r15, ror r15]!

did 'bt' got the following:

Tracing pid 10 tid 100002 td 0xc4a69b40
db_trace_self() at db_trace_self
         pc = 0xc055c420  lr = 0xc00c6150 (db_stack_trace+0x10c)
         sp = 0xc2dbe870  fp = 0xc2dbe888
db_stack_trace() at db_stack_trace+0x10c
         pc = 0xc00c6150  lr = 0xc00c5da0 (db_command+0x258)
         sp = 0xc2dbe890  fp = 0xc2dbe930
         r4 = 0x00000001  r5 = 0x00000000
         r6 = 0xc060444a r10 = 0xc08c0efc
db_command() at db_command+0x258
         pc = 0xc00c5da0  lr = 0xc00c5b38 (db_command_loop+0x74)
         sp = 0xc2dbe938  fp = 0xc2dbe948
         r4 = 0xc05d6759  r5 = 0xc05fb38a
         r6 = 0xc08c0ee8  r7 = 0xc08589e0
         r8 = 0xc0734288  r9 = 0xc08589c0
        r10 = 0xc06dd208
db_command_loop() at db_command_loop+0x74
         pc = 0xc00c5b38  lr = 0xc00c9530 (db_trap+0x12c)
         sp = 0xc2dbe950  fp = 0xc2dbea68
         r4 = 0x00000000  r5 = 0xc08c0ef4
         r6 = 0xc2dbeb20 r10 = 0xc06dd208
db_trap() at db_trap+0x12c
         pc = 0xc00c9530  lr = 0xc02a8c2c (kdb_trap+0x19c)
         sp = 0xc2dbea70  fp = 0xc2dbea98
         r4 = 0x00000000  r5 = 0x00000001
         r6 = 0xc2dbeb20  r7 = 0xc08589e0
kdb_trap() at kdb_trap+0x19c
         pc = 0xc02a8c2c  lr = 0xc057b5e4 (undefinedinstruction+0x214)
         sp = 0xc2dbeaa0  fp = 0xc2dbeb18
         r4 = 0x00000000  r5 = 0x00000000
         r6 = 0xc057b320  r7 = 0xe7ffffff
         r8 = 0xc2dbeb20  r9 = 0x00000001
        r10 = 0xc02a8534
undefinedinstruction() at undefinedinstruction+0x214
         pc = 0xc057b5e4  lr = 0xc055eda4 (exception_exit)
         sp = 0xc2dbeb20  fp = 0xc2dbebc0
         r4 = 0x60000193  r5 = 0xd87ad490
         r6 = 0x00000001  r7 = 0x00040000
         r8 = 0x00000000  r9 = 0x00000000
        r10 = 0xc06ea12c
exception_exit() at exception_exit
         pc = 0xc055eda4  lr = 0xc02a851c (kdb_alt_break_internal+0x1b0)
         sp = 0xc2dbebb0  fp = 0xc2dbebc0
         r0 = 0xc08589d0  r1 = 0x00000000
         r2 = 0xc2dbeaec  r3 = 0x60000193
         r4 = 0xd87ad400  r5 = 0xd87ad490
         r6 = 0x00000001  r7 = 0x00040000
         r8 = 0x00000000  r9 = 0x00000000
        r10 = 0xc06ea12c r12 = 0x00000001
$a.6() at $a.6
         pc = 0xc02a8538  lr = 0xc02a8364 (kdb_alt_break+0x10)
         sp = 0xc2dbebc8  fp = 0xc2dbebc8
         r4 = 0xd87ad400  r5 = 0xd87ad490
         r6 = 0x00000001  r7 = 0x00040000
kdb_alt_break() at kdb_alt_break+0x10
         pc = 0xc02a8364  lr = 0xc012c1b4 (uart_intr_rxready+0x8c)
         sp = 0xc2dbebd0  fp = 0xc2dbebe0
uart_intr_rxready() at uart_intr_rxready+0x8c
         pc = 0xc012c1b4  lr = 0xc012cff0 (uart_intr+0x11c)
         sp = 0xc2dbebe8  fp = 0xc2dbec20
         r4 = 0xc4aaf800  r5 = 0xd87ad400
         r6 = 0xd87ad59c r10 = 0xc06ea12c
uart_intr() at uart_intr+0x11c
         pc = 0xc012cff0  lr = 0xc0226054 (intr_event_handle+0xa4)
         sp = 0xc2dbec28  fp = 0xc2dbec48
         r4 = 0xc4aaf800  r5 = 0xc2dbed28
         r6 = 0xd879d058  r7 = 0xc4a69b40
         r8 = 0xc2e80600  r9 = 0x00000000
        r10 = 0xc2dbed28
intr_event_handle() at intr_event_handle+0xa4
         pc = 0xc0226054  lr = 0xc0560cb4 (intr_isrc_dispatch+0x68)
         sp = 0xc2dbec50  fp = 0xc2dbec58
         r4 = 0xd879d058  r5 = 0xc2dbed28
         r6 = 0xd879d058  r7 = 0x00000041
         r8 = 0xc0638409  r9 = 0xd879c018
        r10 = 0x001fffff
intr_isrc_dispatch() at intr_isrc_dispatch+0x68
         pc = 0xc0560cb4  lr = 0xc05892e4 (bcm2835_intc_intr+0x88)
         sp = 0xc2dbec60  fp = 0xc2dbec80
         r4 = 0xd879c000  r5 = 0x00000000
bcm2835_intc_intr() at bcm2835_intc_intr+0x88
         pc = 0xc05892e4  lr = 0xc0226054 (intr_event_handle+0xa4)
         sp = 0xc2dbec88  fp = 0xc2dbeca8
         r4 = 0xc4ab0600  r5 = 0xc2dbed28
         r6 = 0x00000000  r7 = 0xc4a69b40
         r8 = 0xc2e80980  r9 = 0x00000000
        r10 = 0xc2dbed28
intr_event_handle() at intr_event_handle+0xa4
         pc = 0xc0226054  lr = 0xc0560cb4 (intr_isrc_dispatch+0x68)
         sp = 0xc2dbecb0  fp = 0xc2dbecb8
         r4 = 0xd879a204  r5 = 0xc2dbed28
         r6 = 0x00000000  r7 = 0x00000000
         r8 = 0x00000100  r9 = 0xc0856cc4
        r10 = 0xc2dbed28
intr_isrc_dispatch() at intr_isrc_dispatch+0x68
         pc = 0xc0560cb4  lr = 0xc0580dac (bcm_lintc_intr+0x268)
         sp = 0xc2dbecc0  fp = 0xc2dbed00
         r4 = 0xd879a000  r5 = 0x000000c0
bcm_lintc_intr() at bcm_lintc_intr+0x268
         pc = 0xc0580dac  lr = 0xc0560b14 (intr_irq_handler+0x90)
         sp = 0xc2dbed08  fp = 0xc2dbed20
         r4 = 0xc2dbed28  r5 = 0xc08bba60
         r6 = 0xc4a69b40  r7 = 0x00000000
         r8 = 0xc072bef8  r9 = 0xc0856cc4
        r10 = 0x00000000
intr_irq_handler() at intr_irq_handler+0x90
         pc = 0xc0560b14  lr = 0xc055eda4 (exception_exit)
         sp = 0xc2dbed28  fp = 0xc2dbedc0
         r4 = 0xc4a69b40  r5 = 0x00000000
         r6 = 0xc0856cd4  r7 = 0x00000000
         r8 = 0xffffffff r10 = 0x00000000
exception_exit() at exception_exit
         pc = 0xc055eda4  lr = 0xc0563b68 (cpu_idle+0x74)
         sp = 0xc2dbedb8  fp = 0xc2dbedc0
         r0 = 0x00000000  r1 = 0x000001c0
         r2 = 0x600000d3  r3 = 0x60000013
         r4 = 0xc4a69b40  r5 = 0x00000000
         r6 = 0xc0856cd4  r7 = 0x00000000
         r8 = 0xffffffff  r9 = 0xc0856cc4
        r10 = 0x00000000 r12 = 0xd874f180
cpu_idle() at cpu_idle+0xa0
         pc = 0xc0563b94  lr = 0xc0290564 (sched_idletd+0x510)
         sp = 0xc2dbedc8  fp = 0xc2dbee20
         r4 = 0xc0856cc8 r10 = 0x00000000
sched_idletd() at sched_idletd+0x510
         pc = 0xc0290564  lr = 0xc0223064 (fork_exit+0xa0)
         sp = 0xc2dbee28  fp = 0xc2dbee40
         r4 = 0xc4a69b40  r5 = 0xc4a60ac8
         r6 = 0xc0290054  r7 = 0x00000000
         r8 = 0xc2dbee48  r9 = 0x00000000
        r10 = 0x00000000
fork_exit() at fork_exit+0xa0
         pc = 0xc0223064  lr = 0xc055ed34 (swi_exit)
         sp = 0xc2dbee48  fp = 0x00000000
         r4 = 0xc0290054  r5 = 0x00000000
         r6 = 0x00000000  r7 = 0x00000000
         r8 = 0x00000000 r10 = 0x00000000
swi_exit() at swi_exit
         pc = 0xc055ed34  lr = 0xc055ed34 (swi_exit)
         sp = 0xc2dbee48  fp = 0x00000000
Comment 2 Bob Frazier 2018-06-08 18:09:04 UTC
'show witness' output:

Sleep locks:
mmcsd disk (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/mmc/mmcsd.c:1446
USB control transfer SX lock (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/usb/usb_device.c:2970
 usb_def_mtx (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/usb/usb_request.c:488
USB config SX lock (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/usb/usb_device.c:2884
 Giant (type: sleep mutex, depth: 1, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_module.c:116
  pipe mutex (type: sleep mutex, depth: 2, active refs: 1) -- never acquired
   sigio lock (type: sleep mutex, depth: 3, active refs: 2) -- never acquired
    process group (type: sleep mutex, depth: 4, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_fork.c:589
     pmc-sleep (type: sleep mutex, depth: 5, active refs: 1) -- never acquired
      process lock (type: sleep mutex, depth: 6, active refs: 42) -- last acquired @ /e-drive/src.current/sys/kern/kern_thread.c:1230
       session (type: sleep mutex, depth: 7, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_fork.c:599
        uidinfo hash (type: rw, depth: 8, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_resource.c:1281
         time lock (type: sleep mutex, depth: 9, active refs: 2) -- never acquired
       jail mutex (type: sleep mutex, depth: 7, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_jail.c:2680
        osd_object (type: rm, depth: 8, active refs: 3) -- last acquired @ /e-drive/src.current/sys/kern/kern_osd.c:227
         osd_list (type: sleep mutex, depth: 9, active refs: 3) -- last acquired @ /e-drive/src.current/sys/kern/kern_osd.c:270
       sigacts (type: sleep mutex, depth: 7, active refs: 29) -- last acquired @ /e-drive/src.current/sys/kern/kern_kthread.c:112
     process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed)
  process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed)
  uidinfo hash (type: rw, depth: 8, active refs: 2) -- (already displayed)
  igmp_mtx (type: sleep mutex, depth: 3, active refs: 2) -- last acquired @ /e-drive/src.current/sys/netinet/igmp.c:2165
   ifnet_rw (type: rw, depth: 4, active refs: 3) -- last acquired @ /e-drive/src.current/sys/net/if.c:1420
    if_addr_lock (type: sleep mutex, depth: 5, active refs: 3) -- last acquired @ /e-drive/src.current/sys/net/if.c:1463
  mld_mtx (type: sleep mutex, depth: 3, active refs: 2) -- last acquired @ /e-drive/src.current/sys/netinet6/mld6.c:1751
   ifnet_rw (type: rw, depth: 4, active refs: 3) -- (already displayed)
  vm map (system) (type: sleep mutex, depth: 2, active refs: 146) -- last acquired @ /e-drive/src.current/sys/vm/vm_map.c:1577
   vnode interlock (type: sleep mutex, depth: 3, active refs: 42) -- never acquired
    cdev (type: sleep mutex, depth: 4, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_conf.c:74
     devfs dirlist lock (type: sleep mutex, depth: 5, active refs: 1) -- last acquired @ /e-drive/src.current/sys/fs/devfs/devfs_dir.c:66
    ncneg (type: sleep mutex, depth: 4, active refs: 1) -- never acquired
   vm page free queue (type: sleep mutex, depth: 7, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/vm_page.c:1841
    vm pagequeue (type: sleep mutex, depth: 8, active refs: 5) -- never acquired
   SYSMAPS (type: sleep mutex, depth: 4, active refs: 4) -- last acquired @ /e-drive/src.current/sys/arm/arm/pmap-v6.c:1633
   128 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:3856
   kernel vm object (type: rw, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/vm_kern.c:437
    pmap pv global (type: rw, depth: 4, active refs: 2) -- last acquired @ /e-drive/src.current/sys/arm/arm/pmap-v6.c:3865
     pmap (type: sleep mutex, depth: 5, active refs: 67) -- last acquired @ /e-drive/src.current/sys/arm/arm/pmap-v6.c:1942
      pmap pv list (type: rw, depth: 6, active refs: 1) -- never acquired
       vm page free queue (type: sleep mutex, depth: 7, active refs: 2) -- (already displayed)
    vm page free queue (type: sleep mutex, depth: 7, active refs: 2) -- (already displayed)
    SYSMAPS (type: sleep mutex, depth: 4, active refs: 4) -- (already displayed)
    UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:2323
    16 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:2323
    256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:2565
    vm reserv (type: sleep mutex, depth: 4, active refs: 963) -- last acquired @ /e-drive/src.current/sys/vm/vm_reserv.c:869
     VM reserv domain (type: sleep mutex, depth: 5, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/vm_reserv.c:539
     resv obj lock (type: sleep mutex, depth: 5, active refs: 4) -- last acquired @ /e-drive/src.current/sys/vm/vm_reserv.c:394
    resv obj lock (type: sleep mutex, depth: 5, active refs: 4) -- (already displayed)
   KMAP ENTRY (type: sleep mutex, depth: 3, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:3856
   kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_vmem.c:1180
    UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
    16 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
    256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
  cdev (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
  vm object (type: rw, depth: 2, active refs: 139) -- last acquired @ /e-drive/src.current/sys/vm/vm_glue.c:387
   vm page (type: sleep mutex, depth: 3, active refs: 33) -- never acquired
    pmap pv global (type: rw, depth: 4, active refs: 2) -- (already displayed)
   vm page free queue (type: sleep mutex, depth: 7, active refs: 2) -- (already displayed)
   SYSMAPS (type: sleep mutex, depth: 4, active refs: 4) -- (already displayed)
   UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
  pmap pv global (type: rw, depth: 4, active refs: 2) -- (already displayed)
  pmap (type: sleep mutex, depth: 5, active refs: 67) -- (already displayed)
  vm page free queue (type: sleep mutex, depth: 7, active refs: 2) -- (already displayed)
  SYSMAPS (type: sleep mutex, depth: 4, active refs: 4) -- (already displayed)
  UMA lock (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:1559
   UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
  UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
  4 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:3856
  8 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:3856
  12 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:3856
  16 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
  32 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:2565
  64 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:3856
  128 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed)
  256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
  vmem list lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_vmem.c:1067
  btag lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_vmem.c:634
   kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
   kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
  vm object_list (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/vm_object.c:229
  kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
  kernel arena (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_vmem.c:883
   UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
   16 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
   256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
  kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
  swapdev (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/swap_pager.c:208
  malloc (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_malloc.c:1001
  OF xreflist lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/ofw/openfirm.c:164
  vtdev (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_mutex.c:186
   kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_environment.c:506
  kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed)
  USB quirk (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/usb/quirk/usb_quirk.c:671
  protect sysfilt_ops (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_event.c:1204
  encapmtx (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet/ip_encap.c:369
  domain list (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/uipc_domain.c:264
  intr config (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_autoconf.c:159
  devstat (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_devstat.c:97
  intr event list (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_intr.c:267
  jail mutex (type: sleep mutex, depth: 7, active refs: 1) -- (already displayed)
  root_holds (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/vfs_mountroot.c:178
  XPT topology lock (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/cam/cam_xpt.c:5360
   kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed)
   CAM bus lock (type: sleep mutex, depth: 4, active refs: 1) -- last acquired @ /e-drive/src.current/sys/cam/cam_xpt.c:4861
    CAM queue lock (type: sleep mutex, depth: 5, active refs: 1) -- last acquired @ /e-drive/src.current/sys/cam/cam_xpt.c:4913
  kstkch (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/vm_glue.c:330
  if_cloners lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/net/if_clone.c:367
  et_mtx (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_clocksource.c:672
  loginclasses lock (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_loginclass.c:158
  lltable_list_lock (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/net/if_llatbl.c:642
  chacha20_mtx (type: sleep mutex, depth: 2, active refs: 4) -- last acquired @ /e-drive/src.current/sys/libkern/arc4random.c:172
  kobj (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_kobj.c:288
  bounce pages lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/arm/arm/busdma_machdep-v6.c:1593
  eventhandler (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_eventhandler.c:118
  eventhandler list (type: sleep mutex, depth: 2, active refs: 32) -- last acquired @ /e-drive/src.current/sys/kern/subr_eventhandler.c:129
  vnet_rwlock (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet6/mld6.c:1313
   igmp_mtx (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed)
   mld_mtx (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed)
   IP reassembly (type: sleep mutex, depth: 3, active refs: 64) -- last acquired @ /e-drive/src.current/sys/netinet/ip_reass.c:492
   tcptw (type: rw, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet/tcp_timewait.c:697
   ip6qlock (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet6/frag6.c:752
  USB ref mutex (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/usb/usb_hub.c:2243
  buffer arena (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_vmem.c:1020
   UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
  transient arena (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_vmem.c:1344
   UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
  bufq empty lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/vfs_bio.c:1958
  reseed mutex (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/random/fortuna.c:223
  TID lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_unit.c:644
  tidhash (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_thread.c:1223
   process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed)
  ui_vmsize (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/swap_pager.c:225
  intr map table (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_intr.c:1521
  intr pic list (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_intr.c:729
  intr isrc table (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_intr.c:1205
  taskqgroup (type: sleep mutex, depth: 2, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/subr_gtaskqueue.c:929
  intr event (type: sleep mutex, depth: 2, active refs: 29) -- last acquired @ /e-drive/src.current/sys/kern/kern_intr.c:581
  thread (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_taskqueue.c:467
  GEOM orphanage (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/geom/geom_event.c:402
  devd (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_bus.c:617
  crypto driver table (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/opencrypto/crypto.c:744
  LED mtx (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/led/led.c:295
  XPT lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/cam/cam_xpt.c:936
   XPT topology lock (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
   CAM XPT task (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_taskqueue.c:467
   CAM queue lock (type: sleep mutex, depth: 5, active refs: 1) -- (already displayed)
   CAM bus lock (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed)
  CAM bus lock (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed)
  CAM device lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/cam/cam_xpt.c:2338
   XPT topology lock (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
   CAM bus lock (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed)
  rman head (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_rman.c:150
  rman (type: sleep mutex, depth: 2, active refs: 4) -- last acquired @ /e-drive/src.current/sys/kern/subr_rman.c:670
  bcmdma (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/arm/broadcom/bcm2835/bcm2835_dma.c:331
  vcio mbox (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/arm/broadcom/bcm2835/bcm2835_mbox.c:243
  ttymtx (type: sleep mutex, depth: 2, active refs: 13) -- last acquired @ /e-drive/src.current/sys/kern/subr_terminal.c:274
  bcm_bsc (type: sleep mutex, depth: 2, active refs: 3) -- last acquired @ /e-drive/src.current/sys/arm/broadcom/bcm2835/bcm2835_bsc.c:679
  sdhci (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/sdhci/sdhci.c:2006
   kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed)
   kobj (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
   mmc (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/mmc/mmc.c:427
  mmc (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
  usb_def_mtx (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
  VCHI spinlock &service_spinlock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_core.c:308
  VCHI rwlock (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_arm.c:2741
   condvar (type: sleep mutex, depth: 3, active refs: 5) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/compat/vchi_bsd.c:271
  condvar (type: sleep mutex, depth: 3, active refs: 5) -- (already displayed)
  bcm_audio_lock (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/arm/broadcom/bcm2835/bcm2835_audio.c:487
  vnode_free_list (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/vfs_subr.c:1417
  newnfsd_mtx (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/fs/nfsclient/nfs_clkrpc.c:296
  ncl_iod_mutex (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/fs/nfsclient/nfs_clnfsiod.c:220
  if_clone lock (type: sleep mutex, depth: 2, active refs: 3) -- last acquired @ /e-drive/src.current/sys/net/if_clone.c:354
  if_afdata (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/net/if.c:943
  IPsec transforms list (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netipsec/key.c:8628
  cc_list (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet/cc/cc.c:264
  sound cdev (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/sound/pcm/sound.c:784
  pcm play channel (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/sound/pcm/channel.c:1813
   UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
   8 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed)
   kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
   kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
   kobj (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
   bcm_audio_lock (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
  pcm virtual play channel (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/sound/pcm/channel.c:1226
   kobj (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
sctp_it_wq (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet/sctp_bsd_addr.c:88
shpsx (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_sharedpage.c:88
soaio jobs (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/sys_socket.c:462
Syncer mtx (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/vfs_subr.c:2199
&g_connected_mutex (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_connected.c:108
&state->recycle_mutex (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_core.c:1124
&state->mutex (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_core.c:2614
 proctree (type: sx, depth: 1, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_fork.c:588
  allproc (type: sx, depth: 2, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_fork.c:959
   allprison (type: sx, depth: 3, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/sysv_msg.c:289
   process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed)
  process group (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
  process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed)
 process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed)
 vm object (type: rw, depth: 2, active refs: 139) -- (already displayed)
 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
 4 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed)
 64 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed)
 vm object_list (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
 kernel arena (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
 kstkch (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 eventhandler list (type: sleep mutex, depth: 2, active refs: 32) -- (already displayed)
 TID lock (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 tidhash (type: rw, depth: 2, active refs: 1) -- (already displayed)
 filedesc structure (type: sx, depth: 1, active refs: 29) -- last acquired @ /e-drive/src.current/sys/kern/kern_descrip.c:1956
 VCHI spinlock &service_spinlock (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 &state->slot_mutex (type: sx, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_core.c:785
  VCHI spinlock &quota_spinlock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_core.c:919
 VCHI rwlock (type: rw, depth: 2, active refs: 1) -- (already displayed)
mboxprop (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/arm/broadcom/bcm2835/bcm2835_mbox.c:402
 Giant (type: sleep mutex, depth: 1, active refs: 2) -- (already displayed)
 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
 64 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed)
 256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
 kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
 kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
 vcio mbox (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
CAM doneq (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/cam/cam_xpt.c:5510
LED sx (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/led/led.c:286
 cdev (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
 unit# allocation (type: sleep mutex, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_unit.c:644
crypto return queues (type: sleep mutex, depth: 0, active refs: 4) -- last acquired @ /e-drive/src.current/sys/opencrypto/crypto.c:1584
crypto (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_taskqueue.c:754
crypto op queues (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/opencrypto/crypto.c:1438
GEOM topology (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/geom/geom_event.c:290
 cdev (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
 32 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed)
 64 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed)
 kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
 kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
 sysctl lock (type: sleepable rm, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_sysctl.c:824
  UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
  256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
  kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
  kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
  kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed)
 kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed)
 sleep mtxpool (type: sleep mutex, depth: 1, active refs: 128) -- last acquired @ /e-drive/src.current/sys/kern/vfs_bio.c:4364
 devstat (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 root_holds (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 bio queue (type: sleep mutex, depth: 1, active refs: 2) -- last acquired @ /e-drive/src.current/sys/geom/geom_io.c:103
 GEOM orphanage (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 devd (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
sndstat lock (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/sound/pcm/sndstat.c:264
kqueue_ctx (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_taskqueue.c:754
aiod_kick (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_taskqueue.c:754
user map (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/vm_map.c:3429
pbuf mutex (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/swap_pager.c:512
kernel linker (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_linker.c:592
 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
 4 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed)
 kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
 kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
 kobj (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
 module subsystem sx lock (type: sx, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_module.c:133
  UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
  128 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed)
  kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
  kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
USB sym mutex (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/usb/usb_dev.c:2320
vnet_sysinit_sxlock (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/net/vnet.c:482
 proctree (type: sx, depth: 1, active refs: 2) -- (already displayed)
 Giant (type: sleep mutex, depth: 1, active refs: 2) -- (already displayed)
 process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed)
 bpf global lock (type: sx, depth: 1, active refs: 2) -- last acquired @ /e-drive/src.current/sys/net/bpf.c:2598
  bpf interface lock (type: rw, depth: 2, active refs: 2) -- never acquired
   bpf cdev lock (type: sleep mutex, depth: 3, active refs: 1) -- never acquired
 vm object (type: rw, depth: 2, active refs: 139) -- (already displayed)
 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
 4 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed)
 8 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed)
 16 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
 32 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed)
 128 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed)
 256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
 kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
 kernel arena (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
 umadrain (type: sx, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:2033
  UMA lock (type: rw, depth: 2, active refs: 1) -- (already displayed)
  UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
  kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
  kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
 kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed)
 hhook_head list lock (type: sleep mutex, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_hhook.c:320
 helper list lock (type: rw, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_khelp.c:296
 kstkch (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 ifnet_sx (type: sx, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet/sctp_bsd_addr.c:209
  ifnet_rw (type: rw, depth: 4, active refs: 3) -- (already displayed)
 if_cloners lock (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 pfil_head_list lock (type: sleep mutex, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/net/pfil.c:211
 chacha20_mtx (type: sleep mutex, depth: 2, active refs: 4) -- (already displayed)
 eventhandler (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 eventhandler list (type: sleep mutex, depth: 2, active refs: 32) -- (already displayed)
 vnet_rwlock (type: rw, depth: 2, active refs: 1) -- (already displayed)
 reseed mutex (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 TID lock (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 tidhash (type: rw, depth: 2, active refs: 1) -- (already displayed)
 filedesc structure (type: sx, depth: 1, active refs: 29) -- (already displayed)
 devd (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 if_clone lock (type: sleep mutex, depth: 2, active refs: 3) -- (already displayed)
 tcp_func_lock (type: rw, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet/tcp_subr.c:825
 rib head lock (type: rw, depth: 1, active refs: 2) -- last acquired @ /e-drive/src.current/sys/net/route.c:989
vnet_sxlock (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/net/vnet.c:260
 vnet_rwlock (type: rw, depth: 2, active refs: 1) -- (already displayed)
osd_module (type: sx, depth: 0, active refs: 3) -- last acquired @ /e-drive/src.current/sys/kern/kern_osd.c:110
 osd_object (type: rm, depth: 8, active refs: 3) -- (already displayed)
vm daemon (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/vm_swapout.c:391
vfsconf (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/vfs_init.c:190
 cdev (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
 4 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed)
 256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed)
 kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
 kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
 umadrain (type: sx, depth: 1, active refs: 1) -- (already displayed)
 sysctl lock (type: sleepable rm, depth: 1, active refs: 1) -- (already displayed)
 jail mutex (type: sleep mutex, depth: 7, active refs: 1) -- (already displayed)
 eventhandler (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 eventhandler list (type: sleep mutex, depth: 2, active refs: 32) -- (already displayed)
 devd (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed)
 pfs_fileno (type: sleep mutex, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_unit.c:644
 pfs_node (type: sleep mutex, depth: 1, active refs: 20) -- last acquired @ /e-drive/src.current/sys/fs/pseudofs/pseudofs_internal.h:105
tty list (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/tty.c:1357
TCP ID tree (type: rw, depth: 0, active refs: 1) -- never acquired
 tcp log id bucket (type: sleep mutex, depth: 1, active refs: 1) -- never acquired
  tcpinp (type: rw, depth: 2, active refs: 2) -- never acquired
   so_snd (type: sleep mutex, depth: 3, active refs: 5) -- never acquired
    so_rcv (type: sleep mutex, depth: 4, active refs: 2) -- never acquired
     sellck (type: sleep mutex, depth: 5, active refs: 1) -- never acquired
     radix node head (type: rw, depth: 5, active refs: 1) -- never acquired
      rtentry (type: sleep mutex, depth: 6, active refs: 1) -- never acquired
       ifaddr (type: sleep mutex, depth: 7, active refs: 1) -- never acquired
   TCP log expireq (type: sleep mutex, depth: 3, active refs: 1) -- never acquired
dn->dn_mtx (type: sx, depth: 0, active refs: 1) -- never acquired
 dr->dt.di.dr_mtx (type: sx, depth: 1, active refs: 1) -- never acquired
  db->db_mtx (type: sx, depth: 2, active refs: 1) -- never acquired
ncvn (type: sleep mutex, depth: 0, active refs: 513) -- never acquired
 ncbuc (type: rw, depth: 1, active refs: 513) -- never acquired
  vnode interlock (type: sleep mutex, depth: 3, active refs: 42) -- (already displayed)
kqueue (type: sleep mutex, depth: 0, active refs: 1) -- never acquired
 struct mount mtx (type: sleep mutex, depth: 1, active refs: 1) -- never acquired
  vnode interlock (type: sleep mutex, depth: 3, active refs: 42) -- (already displayed)
vm map (user) (type: sx, depth: 0, active refs: 146) -- last acquired @ /e-drive/src.current/sys/vm/vm_kern.c:576
 vm object (type: rw, depth: 2, active refs: 139) -- (already displayed)
 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed)
 128 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed)
 kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed)
 kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed)
ng_node (type: sleep mutex, depth: 0, active refs: 1) -- never acquired
 ng_worklist (type: sleep mutex, depth: 1, active refs: 1) -- never acquired
network driver (type: sleep mutex, depth: 0, active refs: 1) -- never acquired
802.11 com lock (type: sleep mutex, depth: 0, active refs: 1) -- never acquired
nfsd_mtx (type: sleep mutex, depth: 0, active refs: 1) -- never acquired
 so_snd (type: sleep mutex, depth: 3, active refs: 5) -- (already displayed)
tcp (type: rw, depth: 0, active refs: 2) -- never acquired
 tcpinp (type: rw, depth: 2, active refs: 2) -- (already displayed)
udp (type: rw, depth: 0, active refs: 2) -- never acquired
 udpinp (type: rw, depth: 1, active refs: 3) -- never acquired
  so_snd (type: sleep mutex, depth: 3, active refs: 5) -- (already displayed)
  in_multi_list_mtx (type: sleep mutex, depth: 2, active refs: 2) -- never acquired
   igmp_mtx (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed)
  in6_multi_list_mtx (type: sleep mutex, depth: 2, active refs: 2) -- never acquired
   mld_mtx (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed)
unp_link_rwlock (type: rw, depth: 0, active refs: 2) -- never acquired
 unp_list_lock (type: sleep mutex, depth: 1, active refs: 1) -- never acquired
  unp (type: sleep mutex, depth: 2, active refs: 1) -- never acquired
   so_snd (type: sleep mutex, depth: 3, active refs: 5) -- (already displayed)
in6_multi_sx (type: sx, depth: 0, active refs: 2) -- never acquired
 udpinp (type: rw, depth: 1, active refs: 3) -- (already displayed)
in_multi_sx (type: sx, depth: 0, active refs: 2) -- never acquired
 udpinp (type: rw, depth: 1, active refs: 3) -- (already displayed)
accept (type: sleep mutex, depth: 0, active refs: 2) -- never acquired
 so_snd (type: sleep mutex, depth: 3, active refs: 5) -- (already displayed)
umtx lock (type: sleep mutex, depth: 0, active refs: 2) -- never acquired

Spin locks:

Locks which were never acquired:
select mtxpool (type: sleep mutex, depth: 0)
g_dev (type: sleep mutex, depth: 0)
g_disk_done (type: sleep mutex, depth: 0)
g_disk_start (type: sleep mutex, depth: 0)
primary pcm mixer (type: sleep mutex, depth: 0)
&service->bulk_mutex (type: sx, depth: 0)
&instance->bulk_waiter_list_mutex (type: sx, depth: 0)
mmcsd IOCTL (type: sleep mutex, depth: 0)
USB HUB mutex (type: sleep mutex, depth: 0)
iicbus (type: sleep mutex, depth: 0)
USB device mutex (type: sleep mutex, depth: 0)
MD config lock (type: sx, depth: 0)
ip_id_mtx (type: sleep mutex, depth: 0)
unp_defer (type: sleep mutex, depth: 0)
nd6 DAD queue (type: rw, depth: 0)
nd6 list (type: rw, depth: 0)
nd6 onlink (type: sleep mutex, depth: 0)
addrsel_sxlock (type: sx, depth: 0)
addrsel_lock (type: sleep mutex, depth: 0)
scope6_lock (type: sleep mutex, depth: 0)
fast ipsec security policy acquire list (type: sleep mutex, depth: 0)
ipsec SA acquiring list (type: sleep mutex, depth: 0)
sahtree (type: rm, depth: 0)
fast ipsec regtree (type: sleep mutex, depth: 0)
sptree (type: rm, depth: 0)
rip (type: rw, depth: 0)
udplite (type: rw, depth: 0)
iterator (type: sleep mutex, depth: 0)
sctp_addr_wq (type: sleep mutex, depth: 0)
sctp-addr (type: rw, depth: 0)
sctp-info (type: rw, depth: 0)
isn_mtx (type: sleep mutex, depth: 0)
tcp_hc_entry (type: sleep mutex, depth: 0)
tcp_sc_head (type: sleep mutex, depth: 0)
pcbinfolist (type: rw, depth: 0)
pcbinfohash (type: rw, depth: 0)
hhook_head rm lock (type: rm, depth: 0)
PFil shared rmlock (type: rm, depth: 0)
exec args kva (type: sleep mutex, depth: 0)
pfs_vncache (type: sleep mutex, depth: 0)
if send queue (type: sleep mutex, depth: 0)
msq (type: sleep mutex, depth: 0)
semu (type: sleep mutex, depth: 0)
sem (type: sleep mutex, depth: 0)
semid (type: sleep mutex, depth: 0)
shm dictionary (type: sx, depth: 0)
shm timestamps (type: sleep mutex, depth: 0)
sysvshmsx (type: sx, depth: 0)
nlm_global_lock (type: sleep mutex, depth: 0)
aio_job (type: sleep mutex, depth: 0)
nfsuserd (type: sleep mutex, depth: 0)
nfs_req_mutex (type: sleep mutex, depth: 0)
nfs_slock_mutex (type: sleep mutex, depth: 0)
nfs_sockl_mutex (type: sleep mutex, depth: 0)
nfs_nameid_mutex (type: sleep mutex, depth: 0)
dirhash list (type: sleep mutex, depth: 0)
dqhlock (type: sleep mutex, depth: 0)
ffs_susp (type: sx, depth: 0)
ncnegs (type: sleep mutex, depth: 0)
ncneglh (type: sleep mutex, depth: 0)
ncnegl (type: sleep mutex, depth: 0)
crossmp (type: lockmgr, depth: 0)
bufobj interlock (type: rw, depth: 0)
vnode (type: lockmgr, depth: 0)
vfs hash (type: rw, depth: 0)
sg_lock (type: sleep mutex, depth: 0)
sp_lock (type: sleep mutex, depth: 0)
nfs_clstate_mutex (type: sleep mutex, depth: 0)
nfs_state_mutex (type: sleep mutex, depth: 0)
mntid (type: sleep mutex, depth: 0)
swcr_sessions_lock (type: rw, depth: 0)
cpufreq lock (type: sx, depth: 0)
&state->bulk_transfer_mutex (type: sx, depth: 0)
&state->sync_mutex (type: sx, depth: 0)
VCHI spinlock &msg_queue_spinlock (type: sleep mutex, depth: 0)
vchiq (type: sleep mutex, depth: 0)
VCHI spinlock &bulk_waiter_spinlock (type: sleep mutex, depth: 0)
spigen2 (type: sleep mutex, depth: 0)
spigen1 (type: sleep mutex, depth: 0)
spigen0 (type: sleep mutex, depth: 0)
bcm_spi (type: sleep mutex, depth: 0)
bcmwd (type: sleep mutex, depth: 0)
gpiobus (type: sleep mutex, depth: 0)
CAM LUNs lock (type: sleep mutex, depth: 0)
XPT highpower lock (type: sleep mutex, depth: 0)
nfslock (type: sleep mutex, depth: 0)
sysmouse (type: sleep mutex, depth: 0)
midistat lock (type: sleep mutex, depth: 0)
netisr_mtx (type: sleep mutex, depth: 0)
swi_giant (type: sleep mutex, depth: 0)
swi (type: sleep mutex, depth: 0)
sf_buf (type: sleep mutex, depth: 0)
ktrace_sx (type: sx, depth: 0)
ktrace (type: sleep mutex, depth: 0)
system map (type: sleep mutex, depth: 0)
p_peers (type: sleep mutex, depth: 0)
bufspace daemon run lock (type: sleep mutex, depth: 0)
bufq clean subqueue lock (type: sleep mutex, depth: 0)
bufq dirty lock (type: sleep mutex, depth: 0)
bufq clean lock (type: sleep mutex, depth: 0)
bufwait (type: lockmgr, depth: 0)
dirty buf lock (type: sleep mutex, depth: 0)
buffer daemon lock (type: sleep mutex, depth: 0)
runningbufspace lock (type: sleep mutex, depth: 0)
vnet_data alloc lock (type: sx, depth: 0)
dpcpu alloc lock (type: sx, depth: 0)
umtxshm (type: sleep mutex, depth: 0)
umtxql (type: sleep mutex, depth: 0)
ipsec_ioctl (type: sx, depth: 0)
if_ipsec softc list (type: rm, depth: 0)
CAM SIM free lock (type: sleep mutex, depth: 0)
NFS create verifier mutex (type: sleep mutex, depth: 0)
mem4map (type: sx, depth: 0)
PCM channel sync group lock (type: sleep mutex, depth: 0)
in6_ifaddr_lock (type: rm, depth: 0)
in6_multi_free_mtx (type: sleep mutex, depth: 0)
ifnet descr (type: sx, depth: 0)
DEVFS ruleset lock (type: sx, depth: 0)
rtc list (type: sx, depth: 0)
Global Softdep Lock (type: sleep mutex, depth: 0)
mountlist (type: sleep mutex, depth: 0)
db_script_mtx (type: sleep mutex, depth: 0)
msgbuf lock (type: sleep mutex, depth: 0)
pmc-sx (type: sx, depth: 0)
tty snoop registration (type: sx, depth: 0)
db_capture_sx (type: sx, depth: 0)
so_glabel (type: sleep mutex, depth: 0)
cdevpriv lock (type: sleep mutex, depth: 0)
fail point set sx (type: sx, depth: 0)
in_ifaddr_lock (type: rm, depth: 0)
acct_sx (type: sx, depth: 0)
in_multi_free_mtx (type: sleep mutex, depth: 0)
accept_filter_mtx (type: sleep mutex, depth: 0)
UUID generator mutex lock (type: sleep mutex, depth: 0)
snapdata free list (type: sleep mutex, depth: 0)
clone events drain lock (type: sx, depth: 0)
knlist lock for lockless objects (type: sleep mutex, depth: 0)
in_control (type: sx, depth: 0)
kqueue order (type: sleep mutex, depth: 0)
rtsock route_cb lock (type: sleep mutex, depth: 0)
devfs interlock (type: sleep mutex, depth: 0)
rawcb (type: sleep mutex, depth: 0)
NLM svid lock (type: sleep mutex, depth: 0)
owner graph lock (type: sx, depth: 0)
lock owners lock (type: sx, depth: 0)
lock states lock (type: sx, depth: 0)
auth_unix_lock (type: sx, depth: 0)
sysctl mem (type: sx, depth: 0)
phys_pager list (type: sleep mutex, depth: 0)
dev_pager list (type: sleep mutex, depth: 0)
swsysc (type: sx, depth: 0)
swspsx (type: sx, depth: 0)
vm map sleep mutex (type: sleep mutex, depth: 0)
vmfctr (type: rw, depth: 0)
6 Bucket (type: sleep mutex, depth: 0)
vm pageout lock (type: sleep mutex, depth: 0)
vm domainset lock (type: sleep mutex, depth: 0)
PMAP2 (type: sleep mutex, depth: 0)
Comment 3 Bob Frazier 2018-06-08 18:10:42 UTC
ps output:

  pid  ppid  pgrp   uid  state   wmesg   wchan       cmd
   28     0     0     0  DL      syncer  0xc08b7520  [syncer]
   27     0     0     0  LL     *kernel  0xc2e77880  [vnlru]
   26     0     0     0  LL     *kernel  0xc2e77880  [bufdaemon]
   25     0     0     0  DL      psleep  0xc08bb8ec  [vmdaemon]
   24     0     0     0  LL     *kernel  0xc2e77880  [dom0]
   23     0     0     0  SL      worker_ 0xc2e99b90  [bcm2835_audio_worke]
   22     0     0     0  SL      VCHI co 0xd87df80c  [VCHIQka-0]
   21     0     0     0  DL      mmcsd d 0xc2e99a00  [mmcsd0: mmc/sd card]
   20     0     0     0  DL      -       0xc08b6aa4  [soaiod4]
   19     0     0     0  DL      -       0xc08b6aa4  [soaiod3]
   18     0     0     0  DL      -       0xc08b6aa4  [soaiod2]
   17     0     0     0  DL      -       0xc08b6aa4  [soaiod1]
   16     0     0     0  DL      -       0xc0745120  [rand_harvestq]
   15     0     0     0  DL      waiting 0xc08c4fa8  [sctp_iterator]
   14     0     0     0  SL      sema cv 0xc08dceb0  [VCHIQs-0]
    9     0     0     0  SL      sema cv 0xc08dce8c  [VCHIQr-0]
    8     0     0     0  SL      sema cv 0xc08dce68  [VCHIQ-0]
   13     0     0     0  LL      (threaded)          [usb]
100043                   D       -       0xd87b302c  [usbus0]
100044                   D       -       0xd87b305c  [usbus0]
100045                   D       -       0xd87b308c  [usbus0]
100046                   L      *Giant   0xc2e7a800  [usbus0]
100047                   D       -       0xd87b30ec  [usbus0]
    7     0     0     0  DL      (threaded)          [cam]
100030                   D       -       0xc0742140  [doneq0]
100055                   D       -       0xc074206c  [scanner]
    6     0     0     0  DL      crypto_ 0xc2ee52c4  [crypto returns 3]
    5     0     0     0  DL      crypto_ 0xc2ee528c  [crypto returns 2]
    4     0     0     0  DL      crypto_ 0xc2ee5254  [crypto returns 1]
    3     0     0     0  DL      crypto_ 0xc2ee521c  [crypto returns 0]
    2     0     0     0  DL      crypto_ 0xc08bb07c  [crypto]
   12     0     0     0  LL      (threaded)          [geom]
100018                   L      *kernel  0xc2e77880  [g_event]
100019                   D       -       0xc08c3704  [g_up]
100020                   D       -       0xc08c3708  [g_down]
   11     0     0     0  LL      (threaded)          [intr]
100006                   I                           [swi5: fast taskq]
100009                   I                           [swi6: task queue]
100010                   I                           [swi6: Giant taskq]
100012                   I                           [swi1: netisr 0]
100013                   I                           [swi3: vm]
100014                   L      *Giant   0xc2e7a800  [swi4: clock (0)]
100015                   I                           [swi4: clock (1)]
100016                   I                           [swi4: clock (2)]
100017                   I                           [swi4: clock (3)]
100031                   I                           [intc0,28: bcm_dma0]
100032                   I                           [intc0,29: bcm_dma0]
100033                   I                           [intc0,32: bcm_dma0]
100034                   I                           [intc0,33: bcm_dma0]
100035                   I                           [intc0,34: bcm_dma0]
100036                   I                           [intc0,35: bcm_dma0]
100037                   I                           [intc0,1: mbox0]
100038                   I                           [swi0: uart]
100039                   I                           [intc0,62: spi0]
100040                   I                           [intc0,61: iichb0++]
100041                   I                           [intc0,70: +]
100042                   I                           [intc0,17: +]
100048                   I                           [intc0,2: vchiq0]
   10     0     0     0  RL      (threaded)          [idle]
100002                   Run     CPU 0               [idle: cpu0]
100003                   Run     CPU 1               [idle: cpu1]
100004                   Run     CPU 2               [idle: cpu2]
100005                   CanRun                      [idle: cpu3]
    1     0     0     0  RL                          [kernel]
    0     0     0     0  LLs     (threaded)          [kernel]
100000                   L      *kernel  0xc2e77880  [swapper]
100007                   D       -       0xc2ea3a00  [aiod_kick taskq]
100008                   D       -       0xc2ea3980  [kqueue_ctx taskq]
100011                   D       -       0xc2ea3800  [thread taskq]
100021                   D       -       0xc2ea3680  [crypto_0]
100022                   D       -       0xc2ea3680  [crypto_1]
100023                   D       -       0xc2ea3680  [crypto_2]
100024                   D       -       0xc2ea3680  [crypto_3]
100054                   D       -       0xc2ea3580  [CAM taskq]
100068                   L      *kernel  0xc2e77880  [softirq_0]
100069                   L      *kernel  0xc2e77880  [softirq_1]
100070                   L      *kernel  0xc2e77880  [softirq_2]
100071                   D       -       0xc2ea3200  [softirq_3]
Comment 4 Bob Frazier 2018-06-08 18:40:56 UTC
'show alllocks' output:

Process 13 (usb) thread 0xd87c63c0 (100046)
exclusive sx USB config SX lock (USB config SX lock) r = 0 (0xd8b8c028) locked @ /e-drive/src.current/sys/dev/usb/usb_device.c:2884
Process 1 (kernel) thread 0xc4a6a000 (100001)
exclusive sleep mutex kernel arena domain (kernel arena domain) r = 0 (0xc0b52000) locked @ /e-drive/src.current/sys/kern/subr_vmem.c:1180
Process 0 (kernel) thread 0xc08c3ac0 (100000)
exclusive sleep mutex Giant (Giant) r = 0 (0xc0735140) locked @ /e-drive/src.current/sys/kern/kern_module.c:116
Comment 5 Bob Frazier 2018-06-08 18:47:39 UTC
stack back trace for thread ID 100000 (which has giant lock)

Tracing pid 0 tid 100000 td 0xc08c3ac0
cpu_switch() at cpu_switch+24
         pc = 0xc0579a8c  lr = 0xc028d544 (sched_switch+1672)
         sp = 0xc0b13a60  fp = 0xc0b13ab8
sched_switch() at sched_switch+1672
         pc = 0xc028d544  lr = 0xc026a014 (mi_switch+572)
         sp = 0xc0b13ac0  fp = 0xc0b13ae8
         r4 = 0x0000000a  r5 = 0x00000000
         r6 = 0xc08c3ac0  r7 = 0x0000f248
         r8 = 0x00000000  r9 = 0x00000103
        r10 = 0xc072bef8
mi_switch() at mi_switch+572
         pc = 0xc026a014  lr = 0xc02bf8f4 (turnstile_wait+924)
         sp = 0xc0b13af0  fp = 0xc0b13b18
         r4 = 0xc08c3ac0  r5 = 0xc2e77880
         r6 = 0xc06035f4  r7 = 0x00000000
         r8 = 0xc085d0d8  r9 = 0x00000000
        r10 = 0xc4a6a000
turnstile_wait() at turnstile_wait+924
         pc = 0xc02bf8f4  lr = 0xc0243398 (__mtx_lock_sleep+572)
         sp = 0xc0b13b20  fp = 0xc0b13b70
         r4 = 0x00000000  r5 = 0xc0b52010
         r6 = 0x00000000  r7 = 0xc0b13b48
         r8 = 0xc0b52000  r9 = 0xc2e77880
        r10 = 0xc4a6a000
__mtx_lock_sleep() at __mtx_lock_sleep+572
         pc = 0xc0243398  lr = 0xc0243060 (__mtx_lock_flags+236)
         sp = 0xc0b13b78  fp = 0xc0b13ba0
         r4 = 0x00000000  r5 = 0xc0603ea3
         r6 = 0xc0b52010  r7 = 0x00000000
         r8 = 0xc0b52000  r9 = 0x00000000
        r10 = 0x0000049c
__mtx_lock_flags() at __mtx_lock_flags+236
         pc = 0xc0243060  lr = 0xc02c4464 (vmem_xalloc+364)
         sp = 0xc0b13ba8  fp = 0xc0b13c08
         r4 = 0x00000000  r5 = 0x00002000
         r6 = 0x00002001  r7 = 0x00001000
         r8 = 0xc0b13c4c  r9 = 0x00000001
        r10 = 0xc0b52000
vmem_xalloc() at vmem_xalloc+364
         pc = 0xc02c4464  lr = 0xc02c428c (vmem_alloc+180)
         sp = 0xc0b13c10  fp = 0xc0b13c40
         r4 = 0x00003603  r5 = 0xc0b52000
         r6 = 0x00001000  r7 = 0x00002101
         r8 = 0xc0b13c4c  r9 = 0x00000001
        r10 = 0xc05256b8
vmem_alloc() at vmem_alloc+180
         pc = 0xc02c428c  lr = 0xc052bc48 (kmem_malloc_domain+72)
         sp = 0xc0b13c48  fp = 0xc0b13c68
         r4 = 0xc0b52000  r5 = 0x00001000
         r6 = 0x00000101  r7 = 0x00000000
         r8 = 0x00001000 r10 = 0xc05256b8
kmem_malloc_domain() at kmem_malloc_domain+72
         pc = 0xc052bc48  lr = 0xc05233cc (keg_alloc_slab+224)
         sp = 0xc0b13c70  fp = 0xc0b13ca8
         r4 = 0x00000101  r5 = 0xc0b4a9a0
         r6 = 0xd8bd3420  r7 = 0xc0b4a9b0
         r8 = 0x00001000 r10 = 0xc05256b8
keg_alloc_slab() at keg_alloc_slab+224
         pc = 0xc05233cc  lr = 0xc0525f78 (keg_fetch_slab+324)
         sp = 0xc0b13cb0  fp = 0xc0b13d00
         r4 = 0xc0b4a9a0  r5 = 0x00000000
         r6 = 0x00000001  r7 = 0xc0b4d000
         r8 = 0x00000000  r9 = 0xc2e25100
        r10 = 0x00000000
keg_fetch_slab() at keg_fetch_slab+324
         pc = 0xc0525f78  lr = 0xc0525768 (zone_fetch_slab+92)
         sp = 0xc0b13d08  fp = 0xc0b13d20
         r4 = 0x00000001  r5 = 0xc0b4d000
         r6 = 0xc0b4d000  r7 = 0xc0b4a9a0
         r8 = 0xffffffff  r9 = 0xc2e25100
        r10 = 0x0000000e
zone_fetch_slab() at zone_fetch_slab+92
         pc = 0xc0525768  lr = 0xc0525818 (zone_import+76)
         sp = 0xc0b13d28  fp = 0xc0b13d60
         r4 = 0x00000000  r5 = 0xc0b4d000
         r6 = 0xffffffff  r7 = 0xc052570c
         r8 = 0x00000000 r10 = 0x0000000e
zone_import() at zone_import+76
         pc = 0xc0525818  lr = 0xc0521950 (uma_zalloc_arg+1008)
         sp = 0xc0b13d68  fp = 0xc0b13da0
         r4 = 0x00000000  r5 = 0xc072938c
         r6 = 0xc05257cc  r7 = 0xc2e2510c
         r8 = 0xc0b4d000  r9 = 0xc2e25100
        r10 = 0xffffffff
uma_zalloc_arg() at uma_zalloc_arg+1008
         pc = 0xc0521950  lr = 0xc05218f0 (uma_zalloc_arg+912)
         sp = 0xc0b13da8  fp = 0xc0b13de0
         r4 = 0x00000000  r5 = 0xc07293bc
         r6 = 0xc2e10864  r7 = 0x00000000
         r8 = 0xc2e10780  r9 = 0xc062d1c2
        r10 = 0xffffffff
uma_zalloc_arg() at uma_zalloc_arg+912
         pc = 0xc05218f0  lr = 0xc023df04 (malloc+236)
         sp = 0xc0b13de8  fp = 0xc0b13e10
         r4 = 0xc0707a70  r5 = 0xc2e10780
         r6 = 0xc0707a70  r7 = 0x00000020
         r8 = 0x00000000  r9 = 0x00000001
        r10 = 0x00000002
malloc() at malloc+236
         pc = 0xc023df04  lr = 0xc02a6790 (taskqgroup_adjust+2248)
         sp = 0xc0b13e18  fp = 0xc0b13e68
         r4 = 0xc0707a70  r5 = 0xc2ea3b80
         r6 = 0x00000003  r7 = 0xc02a71dc
         r8 = 0x00000000  r9 = 0xc05c79a2
        r10 = 0xc06005af
taskqgroup_adjust() at taskqgroup_adjust+2248
         pc = 0xc02a6790  lr = 0xc01fa1d4 (mi_startup+396)
         sp = 0xc0b13e70  fp = 0xc0b13e90
         r4 = 0xc08c3f9c  r5 = 0x00000001
         r6 = 0xc0732ac8  r7 = 0x00000000
         r8 = 0xc08c3f98  r9 = 0xc2ea5e6c
        r10 = 0xc08c3ea0
mi_startup() at mi_startup+396
         pc = 0xc01fa1d4  lr = 0xc0000244 (_start+324)
         sp = 0xc0b13e98  fp = 0x00000000
         r4 = 0xc0000378  r5 = 0xc0914000
         r6 = 0x01090e00  r7 = 0x00c52078
         r8 = 0xc0a6a000  r9 = 0x3effb800
        r10 = 0x00000040
_start() at _start+324
         pc = 0xc0000244  lr = 0xc0000244 (_start+324)
         sp = 0xc0b13e98  fp = 0x00000000
Comment 6 Bob Frazier 2018-06-08 18:56:52 UTC
back trace for thread 100001 (which has a locked at sys/kern/subr_vmem.c:1180 )

Tracing pid 1 tid 100001 td 0xc4a6a000
cpu_switch() at cpu_switch+24
         pc = 0xc0579a8c  lr = 0xc028d544 (sched_switch+1672)
         sp = 0xc2dbb858  fp = 0xc2dbb8b0
sched_switch() at sched_switch+1672
         pc = 0xc028d544  lr = 0xc026a014 (mi_switch+572)
         sp = 0xc2dbb8b8  fp = 0xc2dbb8e0
         r4 = 0x00000008  r5 = 0x00000000
         r6 = 0xc4a6a000  r7 = 0x000014ff
         r8 = 0x00000000  r9 = 0x00000602
        r10 = 0xc072bef8
mi_switch() at mi_switch+572
         pc = 0xc026a014  lr = 0xc02664f8 (critical_exit_preempt+112)
         sp = 0xc2dbb8e8  fp = 0xc2dbb8f0
         r4 = 0xc4a6a000  r5 = 0xc05fc082
         r6 = 0xc4a6a000  r7 = 0x00000000
         r8 = 0xc072bef8  r9 = 0xd89df000
        r10 = 0xd89f5fff
critical_exit_preempt() at critical_exit_preempt+112
         pc = 0xc02664f8  lr = 0xc0560b1c (intr_irq_handler+152)
         sp = 0xc2dbb8f8  fp = 0xc2dbb910
         r4 = 0xc2dbb918  r5 = 0xc08bba60
intr_irq_handler() at intr_irq_handler+152
         pc = 0xc0560b1c  lr = 0xc055eda4 (exception_exit)
         sp = 0xc2dbb918  fp = 0xc2dbb9b8
         r4 = 0x00000133  r5 = 0xc0603ea3
         r6 = 0xc0b52010  r7 = 0x00000004
         r8 = 0xc0b52000 r10 = 0xd89f5fff
exception_exit() at exception_exit
         pc = 0xc055eda4  lr = 0xc02c6620 (bt_alloc+36)
         sp = 0xc2dbb9a8  fp = 0xc2dbb9b8
         r0 = 0x00000000  r1 = 0xc4a6a000
         r2 = 0xc0603ea3  r3 = 0x00000133
         r4 = 0x00000133  r5 = 0xc0603ea3
         r6 = 0xc0b52010  r7 = 0x00000004
         r8 = 0xc0b52000  r9 = 0xd89df000
        r10 = 0xd89f5fff r12 = 0x00000000
$a.2() at $a.2+24
         pc = 0xc0243a0c  lr = 0xc02c6620 (bt_alloc+36)
         sp = 0xc2dbb9c0  fp = 0xc2dbb9c8
         r4 = 0xc0b52000  r5 = 0xd89df000
         r6 = 0xd8764d20  r7 = 0xd89dffff
bt_alloc() at bt_alloc+36
         pc = 0xc02c6620  lr = 0xc02c4a78 (vmem_xalloc+1920)
         sp = 0xc2dbb9d0  fp = 0xc2dbba30
         r4 = 0x00001000  r5 = 0xd89df000
vmem_xalloc() at vmem_xalloc+1920
         pc = 0xc02c4a78  lr = 0xc02c428c (vmem_alloc+180)
         sp = 0xc2dbba38  fp = 0xc2dbba68
         r4 = 0x00003603  r5 = 0xc0b52000
         r6 = 0x00001000  r7 = 0x00002101
         r8 = 0xc2dbba74  r9 = 0x00000001
        r10 = 0xc05256b8
vmem_alloc() at vmem_alloc+180
         pc = 0xc02c428c  lr = 0xc052bc48 (kmem_malloc_domain+72)
         sp = 0xc2dbba70  fp = 0xc2dbba90
         r4 = 0xc0b52000  r5 = 0x00001000
         r6 = 0x00000101  r7 = 0x00000000
         r8 = 0x00001000 r10 = 0xc05256b8
kmem_malloc_domain() at kmem_malloc_domain+72
         pc = 0xc052bc48  lr = 0xc05233cc (keg_alloc_slab+224)
         sp = 0xc2dbba98  fp = 0xc2dbbad0
         r4 = 0x00000101  r5 = 0xc0b4a9a0
         r6 = 0xd8bd3318  r7 = 0xc0b4a9b0
         r8 = 0x00001000 r10 = 0xc05256b8
keg_alloc_slab() at keg_alloc_slab+224
         pc = 0xc05233cc  lr = 0xc0525f78 (keg_fetch_slab+324)
         sp = 0xc2dbbad8  fp = 0xc2dbbb28
         r4 = 0xc0b4a9a0  r5 = 0x00000000
         r6 = 0x00000001  r7 = 0xc0b4d000
         r8 = 0x00000000  r9 = 0xd8bd32c0
        r10 = 0x00000000
keg_fetch_slab() at keg_fetch_slab+324
         pc = 0xc0525f78  lr = 0xc0525768 (zone_fetch_slab+92)
         sp = 0xc2dbbb30  fp = 0xc2dbbb48
         r4 = 0x00000001  r5 = 0xc0b4d000
         r6 = 0xc0b4d000  r7 = 0xc0b4a9a0
         r8 = 0xffffffff  r9 = 0xd8bd32c0
        r10 = 0x0000000d
zone_fetch_slab() at zone_fetch_slab+92
         pc = 0xc0525768  lr = 0xc0525818 (zone_import+76)
         sp = 0xc2dbbb50  fp = 0xc2dbbb88
         r4 = 0x0000000a  r5 = 0xc0b4d000
         r6 = 0xffffffff  r7 = 0xc052570c
         r8 = 0xc0b4a9a0 r10 = 0x0000000d
zone_import() at zone_import+76
         pc = 0xc0525818  lr = 0xc0521950 (uma_zalloc_arg+1008)
         sp = 0xc2dbbb90  fp = 0xc2dbbbc8
         r4 = 0x00000000  r5 = 0xc072937c
         r6 = 0xc05257cc  r7 = 0xc2e7114c
         r8 = 0xc0b4d000  r9 = 0xc2e71140
        r10 = 0xffffffff
uma_zalloc_arg() at uma_zalloc_arg+1008
         pc = 0xc0521950  lr = 0xc05218f0 (uma_zalloc_arg+912)
         sp = 0xc2dbbbd0  fp = 0xc2dbbc08
         r4 = 0x00000000  r5 = 0xc07293bc
         r6 = 0xc2e126e4  r7 = 0x00000000
         r8 = 0xc2e12600  r9 = 0xc062d1c2
        r10 = 0xffffffff
uma_zalloc_arg() at uma_zalloc_arg+912
         pc = 0xc05218f0  lr = 0xc023df04 (malloc+236)
         sp = 0xc2dbbc10  fp = 0xc2dbbc38
         r4 = 0x00000000  r5 = 0xc2e12600
         r6 = 0xc0710294  r7 = 0x00000030
         r8 = 0x00000000  r9 = 0x00000002
        r10 = 0x00000002
malloc() at malloc+236
         pc = 0xc023df04  lr = 0xc02b39c0 (sbuf_new+160)
         sp = 0xc2dbbc40  fp = 0xc2dbbc58
         r4 = 0x00000000  r5 = 0xc4a61000
         r6 = 0x00000001  r7 = 0x00000000
         r8 = 0x00000000  r9 = 0x00000001
        r10 = 0x00000000
sbuf_new() at sbuf_new+160
         pc = 0xc02b39c0  lr = 0xc0329b24 (vfs_mountroot+68)
         sp = 0xc2dbbc60  fp = 0xc2dbbdc0
         r4 = 0xc4a61000  r5 = 0xc4a61000
         r6 = 0xc01fb10c  r7 = 0x00000000
         r8 = 0xc4a6a000  r9 = 0x00000000
vfs_mountroot() at vfs_mountroot+68
         pc = 0xc0329b24  lr = 0xc01fb128 (start_init+28)
         sp = 0xc2dbbdc8  fp = 0xc2dbbe20
         r4 = 0xc4a61000  r5 = 0xc4a61000
         r6 = 0xc01fb10c  r7 = 0x00000000
         r8 = 0xc2dbbe48  r9 = 0x00000000
        r10 = 0x00000000
start_init() at start_init+28
         pc = 0xc01fb128  lr = 0xc0223064 (fork_exit+160)
         sp = 0xc2dbbe28  fp = 0xc2dbbe40
         r4 = 0xc4a6a000  r5 = 0xc4a61000
         r6 = 0xc01fb10c  r7 = 0x00000000
         r8 = 0xc2dbbe48  r9 = 0x00000000
        r10 = 0x00000000
fork_exit() at fork_exit+160
         pc = 0xc0223064  lr = 0xc055ed34 (swi_exit)
         sp = 0xc2dbbe48  fp = 0x00000000
         r4 = 0xc01fb10c  r5 = 0x00000000
         r6 = 0x00000000  r7 = 0x00000000
         r8 = 0x00000000 r10 = 0x00000000
swi_exit() at swi_exit
         pc = 0xc055ed34  lr = 0xc055ed34 (swi_exit)
         sp = 0xc2dbbe48  fp = 0x00000000
db> bt
Tracing pid 1 tid 100001 td 0xc4a6a000
cpu_switch() at cpu_switch+24
         pc = 0xc0579a8c  lr = 0xc028d544 (sched_switch+1672)
         sp = 0xc2dbb858  fp = 0xc2dbb8b0
sched_switch() at sched_switch+1672
         pc = 0xc028d544  lr = 0xc026a014 (mi_switch+572)
         sp = 0xc2dbb8b8  fp = 0xc2dbb8e0
         r4 = 0x00000008  r5 = 0x00000000
         r6 = 0xc4a6a000  r7 = 0x000014ff
         r8 = 0x00000000  r9 = 0x00000602
        r10 = 0xc072bef8
mi_switch() at mi_switch+572
         pc = 0xc026a014  lr = 0xc02664f8 (critical_exit_preempt+112)
         sp = 0xc2dbb8e8  fp = 0xc2dbb8f0
         r4 = 0xc4a6a000  r5 = 0xc05fc082
         r6 = 0xc4a6a000  r7 = 0x00000000
         r8 = 0xc072bef8  r9 = 0xd89df000
        r10 = 0xd89f5fff
critical_exit_preempt() at critical_exit_preempt+112
         pc = 0xc02664f8  lr = 0xc0560b1c (intr_irq_handler+152)
         sp = 0xc2dbb8f8  fp = 0xc2dbb910
         r4 = 0xc2dbb918  r5 = 0xc08bba60
intr_irq_handler() at intr_irq_handler+152
         pc = 0xc0560b1c  lr = 0xc055eda4 (exception_exit)
         sp = 0xc2dbb918  fp = 0xc2dbb9b8
         r4 = 0x00000133  r5 = 0xc0603ea3
         r6 = 0xc0b52010  r7 = 0x00000004
         r8 = 0xc0b52000 r10 = 0xd89f5fff
exception_exit() at exception_exit
         pc = 0xc055eda4  lr = 0xc02c6620 (bt_alloc+36)
         sp = 0xc2dbb9a8  fp = 0xc2dbb9b8
         r0 = 0x00000000  r1 = 0xc4a6a000
         r2 = 0xc0603ea3  r3 = 0x00000133
         r4 = 0x00000133  r5 = 0xc0603ea3
         r6 = 0xc0b52010  r7 = 0x00000004
         r8 = 0xc0b52000  r9 = 0xd89df000
        r10 = 0xd89f5fff r12 = 0x00000000
$a.2() at $a.2+24
         pc = 0xc0243a0c  lr = 0xc02c6620 (bt_alloc+36)
         sp = 0xc2dbb9c0  fp = 0xc2dbb9c8
         r4 = 0xc0b52000  r5 = 0xd89df000
         r6 = 0xd8764d20  r7 = 0xd89dffff
bt_alloc() at bt_alloc+36
         pc = 0xc02c6620  lr = 0xc02c4a78 (vmem_xalloc+1920)
         sp = 0xc2dbb9d0  fp = 0xc2dbba30
         r4 = 0x00001000  r5 = 0xd89df000
vmem_xalloc() at vmem_xalloc+1920
         pc = 0xc02c4a78  lr = 0xc02c428c (vmem_alloc+180)
         sp = 0xc2dbba38  fp = 0xc2dbba68
         r4 = 0x00003603  r5 = 0xc0b52000
         r6 = 0x00001000  r7 = 0x00002101
         r8 = 0xc2dbba74  r9 = 0x00000001
        r10 = 0xc05256b8
vmem_alloc() at vmem_alloc+180
         pc = 0xc02c428c  lr = 0xc052bc48 (kmem_malloc_domain+72)
         sp = 0xc2dbba70  fp = 0xc2dbba90
         r4 = 0xc0b52000  r5 = 0x00001000
         r6 = 0x00000101  r7 = 0x00000000
         r8 = 0x00001000 r10 = 0xc05256b8
kmem_malloc_domain() at kmem_malloc_domain+72
         pc = 0xc052bc48  lr = 0xc05233cc (keg_alloc_slab+224)
         sp = 0xc2dbba98  fp = 0xc2dbbad0
         r4 = 0x00000101  r5 = 0xc0b4a9a0
         r6 = 0xd8bd3318  r7 = 0xc0b4a9b0
         r8 = 0x00001000 r10 = 0xc05256b8
keg_alloc_slab() at keg_alloc_slab+224
         pc = 0xc05233cc  lr = 0xc0525f78 (keg_fetch_slab+324)
         sp = 0xc2dbbad8  fp = 0xc2dbbb28
         r4 = 0xc0b4a9a0  r5 = 0x00000000
         r6 = 0x00000001  r7 = 0xc0b4d000
         r8 = 0x00000000  r9 = 0xd8bd32c0
        r10 = 0x00000000
keg_fetch_slab() at keg_fetch_slab+324
         pc = 0xc0525f78  lr = 0xc0525768 (zone_fetch_slab+92)
         sp = 0xc2dbbb30  fp = 0xc2dbbb48
         r4 = 0x00000001  r5 = 0xc0b4d000
         r6 = 0xc0b4d000  r7 = 0xc0b4a9a0
         r8 = 0xffffffff  r9 = 0xd8bd32c0
        r10 = 0x0000000d
zone_fetch_slab() at zone_fetch_slab+92
         pc = 0xc0525768  lr = 0xc0525818 (zone_import+76)
         sp = 0xc2dbbb50  fp = 0xc2dbbb88
         r4 = 0x0000000a  r5 = 0xc0b4d000
         r6 = 0xffffffff  r7 = 0xc052570c
         r8 = 0xc0b4a9a0 r10 = 0x0000000d
zone_import() at zone_import+76
         pc = 0xc0525818  lr = 0xc0521950 (uma_zalloc_arg+1008)
         sp = 0xc2dbbb90  fp = 0xc2dbbbc8
         r4 = 0x00000000  r5 = 0xc072937c
         r6 = 0xc05257cc  r7 = 0xc2e7114c
         r8 = 0xc0b4d000  r9 = 0xc2e71140
        r10 = 0xffffffff
uma_zalloc_arg() at uma_zalloc_arg+1008
         pc = 0xc0521950  lr = 0xc05218f0 (uma_zalloc_arg+912)
         sp = 0xc2dbbbd0  fp = 0xc2dbbc08
         r4 = 0x00000000  r5 = 0xc07293bc
         r6 = 0xc2e126e4  r7 = 0x00000000
         r8 = 0xc2e12600  r9 = 0xc062d1c2
        r10 = 0xffffffff
uma_zalloc_arg() at uma_zalloc_arg+912
         pc = 0xc05218f0  lr = 0xc023df04 (malloc+236)
         sp = 0xc2dbbc10  fp = 0xc2dbbc38
         r4 = 0x00000000  r5 = 0xc2e12600
         r6 = 0xc0710294  r7 = 0x00000030
         r8 = 0x00000000  r9 = 0x00000002
        r10 = 0x00000002
malloc() at malloc+236
         pc = 0xc023df04  lr = 0xc02b39c0 (sbuf_new+160)
         sp = 0xc2dbbc40  fp = 0xc2dbbc58
         r4 = 0x00000000  r5 = 0xc4a61000
         r6 = 0x00000001  r7 = 0x00000000
         r8 = 0x00000000  r9 = 0x00000001
        r10 = 0x00000000
sbuf_new() at sbuf_new+160
         pc = 0xc02b39c0  lr = 0xc0329b24 (vfs_mountroot+68)
         sp = 0xc2dbbc60  fp = 0xc2dbbdc0
         r4 = 0xc4a61000  r5 = 0xc4a61000
         r6 = 0xc01fb10c  r7 = 0x00000000
         r8 = 0xc4a6a000  r9 = 0x00000000
vfs_mountroot() at vfs_mountroot+68
         pc = 0xc0329b24  lr = 0xc01fb128 (start_init+28)
         sp = 0xc2dbbdc8  fp = 0xc2dbbe20
         r4 = 0xc4a61000  r5 = 0xc4a61000
         r6 = 0xc01fb10c  r7 = 0x00000000
         r8 = 0xc2dbbe48  r9 = 0x00000000
        r10 = 0x00000000
start_init() at start_init+28
         pc = 0xc01fb128  lr = 0xc0223064 (fork_exit+160)
         sp = 0xc2dbbe28  fp = 0xc2dbbe40
         r4 = 0xc4a6a000  r5 = 0xc4a61000
         r6 = 0xc01fb10c  r7 = 0x00000000
         r8 = 0xc2dbbe48  r9 = 0x00000000
        r10 = 0x00000000
fork_exit() at fork_exit+160
         pc = 0xc0223064  lr = 0xc055ed34 (swi_exit)
         sp = 0xc2dbbe48  fp = 0x00000000
         r4 = 0xc01fb10c  r5 = 0x00000000
         r6 = 0x00000000  r7 = 0x00000000
         r8 = 0x00000000 r10 = 0x00000000
swi_exit() at swi_exit
         pc = 0xc055ed34  lr = 0xc055ed34 (swi_exit)
         sp = 0xc2dbbe48  fp = 0x00000000
Comment 7 Bob Frazier 2018-06-09 19:13:26 UTC
sorry if there's information overload here but I wanted to capture as much as I could get out of the debuger.

My conclusion is that a kernel process is calling vmem_xalloc(), which had an exception take place that involved bt_alloc(), or maybe something else that's erroneously being flagged as bt_alloc().

The problem is that vmem_xalloc() does not call bt_alloc directly, but it does call vmem_clip() which returns 'void' [so this might be messing up the stack frame?].  vmem_clip() then calls bt_alloc() in a couple of places, and returns after doing its magic.

The problem here is that it appears that the 'vmem_t' structure might either be incomplete, or somehow corrupted, which causes an exception (an actual interrupt?) of some kind to happen (page fault, or ?).  I can't really tell from the debug info I've posted here (someone more familiar with this might recognize it).

But here's a summary of what happened:

a) vmem_xalloc locks the VM via VM_LOCK
b) vmem_xalloc loops through structures looking for a page that's big enough
c) vmem_xalloc (apparently) calls 'vmem_clip()' to (apparently) shorten it to the correct size
d) "an exception happens" while the VM is still locked
e) the exception handler allows other threads to run, waiting for who knows what
f) another thread tries to lock the VM with the Giant lock already grabbed
g) deadlock

So it's important to fix whatever is causing this exception, as (I think) it has happened on the PPC platform as well, so it's not just me and looks like a 'ticking time bomb' waiting for a Murphy's Law to arm it.  I'm sure it will affect everyone at some point, although it appears to be ok if you change the bootup timing in a way that doesn't deadlock the VM (by enabling verbose logging, for one).  That's a bad workaround, but it keeps me able to work with it in the mean time.

I'll do whatever additional analysis anyone wants done. Just let me know by posting to the bug.
Comment 8 Bob Frazier 2018-06-09 20:30:17 UTC
some additional information:

# addr2line -e /usr/obj/e-drive/src.current/arm.armv7/sys/RPI2/kernel.full 0xc0243a0c
/e-drive/src.current/sys/kern/kern_mutex.c:1062

# addr2line -e /usr/obj/e-drive/src.current/arm.armv7/sys/RPI2/kernel.full 0xc02c4a78 
/e-drive/src.current/sys/kern/subr_vmem.c:973

# addr2line -e /usr/obj/e-drive/src.current/arm.armv7/sys/RPI2/kernel.full 0xc02c6620
/e-drive/src.current/sys/kern/subr_vmem.c:308
Comment 9 Bob Frazier 2018-06-09 20:46:25 UTC
from loader prompt, with 'verbose boot' off, did the following:

loader>  set hw.ncpu=1

result:  boot success!

(non-verbose) boot output looks like this:

/boot/dtb/bcm2836-rpi-2-b.dtb size=0x4c49
Loaded DTB from file 'bcm2836-rpi-2-b.dtb'.
Loading DTB overlays: 'spigen-rpi2.dtbo'
/boot/dtb/overlays/spigen-rpi2.dtbo size=0x3c6
applying DTB overlay '/boot/dtb/overlays/spigen-rpi2.dtbo'
Kernel entry at 0x1200100...
Kernel args: (null)
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #2 r334665M: Tue Jun  5 21:17:17 PDT 2018
    bobf@hack.SFT.local:/e-drive/obj.current/e-drive/src.current/arm.armv7/sys/RPI2 arm
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
CPU: ARM Cortex-A7 r0p5 (ECO: 0x00000000)
CPU Features: 
  Multiprocessing, Thumb2, Security, Virtualization, Generic Timer, VMSAv7,
  PXN, LPAE, Coherent Walk
Optional instructions: 
  SDIV/UDIV, UMULL, SMULL, SIMD(ext)
LoUU:2 LoC:3 LoUIS:2 
Cache level 1:
 32KB/64B 4-way data cache WB Read-Alloc Write-Alloc
 32KB/32B 2-way instruction cache Read-Alloc
Cache level 2:
 512KB/64B 8-way unified cache WB Read-Alloc Write-Alloc
real memory  = 0 (0 MB)
avail memory = 1008693248 (961 MB)
random: unblocking device.
random: entropy device external interface
kbd0 at kbdmux0
ofwbus0: <Open Firmware Device Tree>
simplebus0: <Flattened device tree simple bus> on ofwbus0
local_intc0: <BCM2836 Interrupt Controller> mem 0x40000000-0x400000ff on simplebus0
intc0: <BCM2835 Interrupt Controller> mem 0x7e00b200-0x7e00b3ff irq 25 on simplebus0
gpio0: <BCM2708/2835 GPIO controller> mem 0x7e200000-0x7e2000b3 irq 27,28,29,30 on simplebus0
gpio0: read-only pins: 46-53.
gpio0: reserved pins: 48-53.
gpiobus0: <OFW GPIO bus> on gpio0
generic_timer0: <ARMv7 Generic Timer> irq 1,2,3,4 on ofwbus0
Timecounter "ARM MPCore Timecounter" frequency 19200000 Hz quality 1000
Event timer "ARM MPCore Eventtimer" frequency 19200000 Hz quality 1000
bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007eff irq 9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24 on simplebus0
bcmwd0: <BCM2708/2835 Watchdog> mem 0x7e100000-0x7e100027 on simplebus0
bcm2835_clkman0: <BCM283x Clock Manager> mem 0x7e101000-0x7e102fff on simplebus0
bcmrng0: <Broadcom BCM2835 RNG> mem 0x7e104000-0x7e10400f on simplebus0
mbox0: <BCM2835 VideoCore Mailbox> mem 0x7e00b880-0x7e00b8bf irq 26 on simplebus0
gpioc0: <GPIO controller> on gpio0
uart0: <PrimeCell UART (PL011)> mem 0x7e201000-0x7e201fff irq 31 on simplebus0
uart0: console (115200,n,8,1)
spi0: <BCM2708/2835 SPI controller> mem 0x7e204000-0x7e204fff irq 33 on simplebus0
spibus0: <OFW SPI bus> on spi0
spigen0: <SPI Generic IO> at cs 0 mode 0 on spibus0
spigen1: <SPI Generic IO> at cs 1 mode 0 on spibus0
spigen2: <SPI Generic IO> at cs 2 mode 0 on spibus0
iichb0: <BCM2708/2835 BSC controller> mem 0x7e205000-0x7e205fff irq 34 on simplebus0
pwm0: <BCM2708/2835 PWM controller> mem 0x7e20c000-0x7e20c027 on simplebus0
sdhci_bcm0: <Broadcom 2708 SDHCI controller> mem 0x7e300000-0x7e3000ff irq 41 on simplebus0
mmc0: <MMC/SD bus> on sdhci_bcm0
iichb1: <BCM2708/2835 BSC controller> mem 0x7e804000-0x7e804fff irq 44 on simplebus0
iichb2: <BCM2708/2835 BSC controller> mem 0x7e805000-0x7e805fff irq 45 on simplebus0
bcm283x_dwcotg0: <DWC OTG 2.0 integrated USB controller (bcm283x)> mem 0x7e980000-0x7e98ffff irq 50 on simplebus0
usbus0 on bcm283x_dwcotg0
vchiq0: <BCM2835 VCHIQ> mem 0x7e00b840-0x7e00b84e irq 52 on simplebus0
vchiq: local ver 8 (min 3), remote ver 8.
pcm0: <VCHIQ audio> on vchiq0
fb0: <BCM2835 VT framebuffer driver> on simplebus0
fbd0 on fb0
VT: initialize with new VT driver "fb".
fb0: 656x416(656x416@0,0) 24bpp
fb0: fbswap: 0, pitch 1968, base rx3ef34000, screen_size 818688
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
bcm2835_cpufreq0: <CPU Frequency Control> on cpu0
cpu1: <Open Firmware CPU> on cpulist0
cpu2: <Open Firmware CPU> on cpulist0
cpu3: <Open Firmware CPU> on cpulist0
gpioled0: <GPIO LEDs> on ofwbus0
cryptosoft0: <software crypto>
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
iicbus0: <OFW I2C bus> on iichb0
iic0: <I2C generic I/O> on iicbus0
ugen0.1: <DWCOTG OTG Root HUB> at usbus0
uhub0: <DWCOTG OTG Root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
mmcsd0: 4GB <SDHC SS04G 8.0 SN 032B0249 MFG 11/2014 by 3 SD> at mmc0 50.0MHz/4bit/65535-block
iicbus1: <OFW I2C bus> on iichb1
iic1: <I2C generic I/O> on iicbus1
iicbus2: <OFW I2C bus> on iichb2
iic2: <I2C generic I/O> on iicbus2
bcm2835_cpufreq0: ARM 600MHz, Core 250MHz, SDRAM 400MHz, Turbo OFF
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/ufs/rootfs [rw]...
Warning: no time-of-day clock registered, system time will not be set accurately
uhub0: 1 port with 1 removable, self powered
ugen0.2: <vendor 0x0424 product 0x9514> at usbus0
uhub1 on uhub0
uhub1: <vendor 0x0424 product 0x9514, class 9/0, rev 2.00/2.00, addr 2> on usbus0
uhub1: MTT enabled
Setting hostuuid: 883f4aee-4e62-11e8-8d4c-b827eb35f155.
Setting hostid: 0xb55ef262.
No suitable dump device was found.
Starting file system checks:
/dev/ufs/rootfs: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ufs/rootfs: clean, 226368 free (2072 frags, 28037 blocks, 0.2% fragmentation)
uhub1: 5 ports with 4 removable, self powered
Mounting local filesystems:ugen0.3: <vendor 0x0424 product 0xec00> at usbus0
smsc0 on uhub1
smsc0: <vendor 0x0424 product 0xec00, rev 2.00/2.00, addr 3> on usbus0
smsc0: chip 0xec00, rev. 0002
miibus0: <MII bus> on smsc0
ukphy0: <Generic IEEE 802.3u media interface> PHY 1 on miibus0
ukphy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ue0: <USB Ethernet> on smsc0
ue0: Ethernet address: b8:27:eb:35:f1:55
.
Setting hostname: pi2c.
/etc/rc: WARNING: $ipv6_enable is obsolete.  Use $ipv6_activate_all_interfaces instead.
Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: .
/etc/rc: WARNING: $ipv6_enable is obsolete.  Use $ipv6_activate_all_interfaces instead.
/etc/rc: WARNING: $ipv6_enable is obsolete.  Use $ipv6_activate_all_interfaces instead.
lo0: link state changed to UP
smsc0: chip 0xec00, rev. 0002
ue0: link state changed to DOWN
ue0: link state changed to UP
Starting Network: lo0 ue0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
	inet 127.0.0.1 netmask 0xff000000 
	groups: lo 
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=80009<RXCSUM,VLAN_MTU,LINKSTATE>
	ether b8:27:eb:35:f1:55
	inet6 fe80::ba27:ebff:fe35:f155%ue0 prefixlen 64 scopeid 0x2 
	inet6 2001:470:d:c9c:ba27:ebff:fe35:f155 prefixlen 64 autoconf 
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
Soft Float compatibility ldconfig path:
Starting devd.
Starting dhclient.
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPACK from 192.168.122.1
bound to 192.168.122.32 -- renewal in 82839 seconds.
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Creating and/or trimming log files.
Starting syslogd.
installing gpioshutdown
kern.gpioshutdown.led_gpio: -1 -> 35
kern.gpioshutdown.led_gpio_set: 1 -> 1
gpio0: Stray irq 35 disabled
Clearing /tmp (X related).
Updating motd:.
Mounting late filesystems:.
Starting powerd.
Configuring vt: blanktime.
Performing sanity check on sshd configuration.
Starting sshd.
Starting cron.
Starting background file system checks in 60 seconds.

Fri Jun  8 19:21:30 UTC 2018

FreeBSD/arm (pi2c) (ttyu0)

login:
Comment 10 Bob Frazier 2018-06-09 21:34:43 UTC
Created attachment 194119 [details]
unedited console output for VERBOSE_SYSINIT

this was captured from serial console output using a different kernel with VERBOSE_SYSINIT enabled.
Comment 11 Bob Frazier 2018-06-09 21:36:17 UTC
other notes:

when booting the original kernel (not the 'VERBOSE_SYSINIT' one), and entering 'set hw.ncpu=4' from the loader prompt, it sometimes worked, and sometimes did not work.  however I have not yet observed 'set hw.ncpu=2' not working.
Comment 12 Bob Frazier 2018-06-13 09:25:32 UTC
attempted to reproduce in r335007  - could not reproduce the bug

not sure if it's fixed or being worked around.  In any case it's not happening in this revision.
Comment 13 Bob Frazier 2018-06-23 21:38:02 UTC
this bug re-surfaced in r335576 .  will attempt to narrow it down if possible.
Comment 14 Bob Frazier 2018-08-24 18:24:21 UTC
does not appear to happen in r338208 nor the previous revision that I'd built a few weeks ago.  Will continue to test for this.