using rev 334665 on an RPi 2 using the older bcm2836 CPU. I have been able to consistently demonstrate this problem more than half of the time when I boot up from a power up, or a 'reboot', or by pressing a key in the serial console after 'poweroff'. The hang appears to happen immediately after the following text displays on the console Releasing APs Consistently, whenever I enable verbose output as a boot option in the 'beastie' menu, the reboot takes place normally, i.e. without hanging. uname output: FreeBSD pi2c 12.0-CURRENT FreeBSD 12.0-CURRENT #2 r334665M: Tue Jun 5 21:17:17 PDT 2018 bobf@hack.SFT.local:/e-drive/obj.current/e-drive/src.current/arm.armv7/sys/RPI2 arm The problem first appeared when I obtained source yesterday and built a fresh version. Prior to this I had been using code that was around 3 or 4 weeks old. This problem was reproduced several times with unmodified code and no additional overlays or kernel modules being loaded. Typical console output demonstrating problem: /boot/dtb/bcm2836-rpi-2-b.dtb size=0x4c49 Loaded DTB from file 'bcm2836-rpi-2-b.dtb'. Kernel entry at 0x1200100... Kernel args: (null) KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #2 r334665M: Tue Jun 5 21:17:17 PDT 2018 bobf@hack.SFT.local:/e-drive/obj.current/e-drive/src.current/arm.armv7/sys/RPI2 arm FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) WARNING: WITNESS option enabled, expect reduced performance. VT: init without driver. CPU: ARM Cortex-A7 r0p5 (ECO: 0x00000000) CPU Features: Multiprocessing, Thumb2, Security, Virtualization, Generic Timer, VMSAv7, PXN, LPAE, Coherent Walk Optional instructions: SDIV/UDIV, UMULL, SMULL, SIMD(ext) LoUU:2 LoC:3 LoUIS:2 Cache level 1: 32KB/64B 4-way data cache WB Read-Alloc Write-Alloc 32KB/32B 2-way instruction cache Read-Alloc Cache level 2: 512KB/64B 8-way unified cache WB Read-Alloc Write-Alloc real memory = 0 (0 MB) avail memory = 1008672768 (961 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs random: unblocking device. random: entropy device external interface kbd0 at kbdmux0 ofwbus0: <Open Firmware Device Tree> simplebus0: <Flattened device tree simple bus> on ofwbus0 local_intc0: <BCM2836 Interrupt Controller> mem 0x40000000-0x400000ff on simplebus0 intc0: <BCM2835 Interrupt Controller> mem 0x7e00b200-0x7e00b3ff irq 25 on simplebus0 gpio0: <BCM2708/2835 GPIO controller> mem 0x7e200000-0x7e2000b3 irq 27,28,29,30 on simplebus0 gpio0: read-only pins: 46-53. gpio0: reserved pins: 48-53. gpiobus0: <OFW GPIO bus> on gpio0 generic_timer0: <ARMv7 Generic Timer> irq 1,2,3,4 on ofwbus0 Timecounter "ARM MPCore Timecounter" frequency 19200000 Hz quality 1000 Event timer "ARM MPCore Eventtimer" frequency 19200000 Hz quality 1000 bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007eff irq 9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24 on simplebus0 bcmwd0: <BCM2708/2835 Watchdog> mem 0x7e100000-0x7e100027 on simplebus0 bcmrng0: <Broadcom BCM2835 RNG> mem 0x7e104000-0x7e10400f on simplebus0 mbox0: <BCM2835 VideoCore Mailbox> mem 0x7e00b880-0x7e00b8bf irq 26 on simplebus0 gpioc0: <GPIO controller> on gpio0 uart0: <PrimeCell UART (PL011)> mem 0x7e201000-0x7e201fff irq 31 on simplebus0 uart0: console (115200,n,8,1) spi0: <BCM2708/2835 SPI controller> mem 0x7e204000-0x7e204fff irq 33 on simplebus0 spibus0: <OFW SPI bus> on spi0 iichb0: <BCM2708/2835 BSC controller> mem 0x7e205000-0x7e205fff irq 34 on simplebus0 sdhci_bcm0: <Broadcom 2708 SDHCI controller> mem 0x7e300000-0x7e3000ff irq 41 on simplebus0 mmc0: <MMC/SD bus> on sdhci_bcm0 iichb1: <BCM2708/2835 BSC controller> mem 0x7e804000-0x7e804fff irq 44 on simplebus0 iichb2: <BCM2708/2835 BSC controller> mem 0x7e805000-0x7e805fff irq 45 on simplebus0 bcm283x_dwcotg0: <DWC OTG 2.0 integrated USB controller (bcm283x)> mem 0x7e980000-0x7e98ffff irq 50 on simplebus0 usbus0 on bcm283x_dwcotg0 vchiq0: <BCM2835 VCHIQ> mem 0x7e00b840-0x7e00b84e irq 52 on simplebus0 vchiq: local ver 8 (min 3), remote ver 8. pcm0: <VCHIQ audio> on vchiq0 fb0: <BCM2835 VT framebuffer driver> on simplebus0 fbd0 on fb0 VT: initialize with new VT driver "fb". fb0: 656x416(656x416@0,0) 24bpp fb0: fbswap: 0, pitch 1968, base rx3ef34000, screen_size 818688 cpulist0: <Open Firmware CPU Group> on ofwbus0 cpu0: <Open Firmware CPU> on cpulist0 bcm2835_cpufreq0: <CPU Frequency Control> on cpu0 cpu1: <Open Firmware CPU> on cpulist0 cpu2: <Open Firmware CPU> on cpulist0 cpu3: <Open Firmware CPU> on cpulist0 gpioled0: <GPIO LEDs> on ofwbus0 cryptosoft0: <software crypto> Timecounters tick every 1.000 msec usbus0: 480Mbps High Speed USB v2.0 iicbus0: <OFW I2C bus> on iichb0 iic0: <I2C generic I/O> on iicbus0 ugen0.1: <DWCOTG OTG Root HUB> at usbus0 uhub0: <DWCOTG OTG Root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0 mmcsd0: 4GB <SDHC SS04G 8.0 SN 032B0249 MFG 11/2014 by 3 SD> at mmc0 50.0MHz/4bit/65535-block iicbus1: <OFW I2C bus> on iichb1 iic1: <I2C generic I/O> on iicbus1 iicbus2: <OFW I2C bus> on iichb2 iic2: <I2C generic I/O> on iicbus2 bcm2835_cpufreq0: ARM 600MHz, Core 250MHz, SDRAM 400MHz, Turbo OFF Release APs (this is where the hang takes place; no response to keyboard input on the serial console at this point. power cycle is only option)
was able to type ~ then ctrl+b and dropped into the debugger KDB: enter: Break to debugger [ thread pid 10 tid 100002 ] Stopped at $d.5: ldrb r15, [r15, r15, ror r15]! did 'bt' got the following: Tracing pid 10 tid 100002 td 0xc4a69b40 db_trace_self() at db_trace_self pc = 0xc055c420 lr = 0xc00c6150 (db_stack_trace+0x10c) sp = 0xc2dbe870 fp = 0xc2dbe888 db_stack_trace() at db_stack_trace+0x10c pc = 0xc00c6150 lr = 0xc00c5da0 (db_command+0x258) sp = 0xc2dbe890 fp = 0xc2dbe930 r4 = 0x00000001 r5 = 0x00000000 r6 = 0xc060444a r10 = 0xc08c0efc db_command() at db_command+0x258 pc = 0xc00c5da0 lr = 0xc00c5b38 (db_command_loop+0x74) sp = 0xc2dbe938 fp = 0xc2dbe948 r4 = 0xc05d6759 r5 = 0xc05fb38a r6 = 0xc08c0ee8 r7 = 0xc08589e0 r8 = 0xc0734288 r9 = 0xc08589c0 r10 = 0xc06dd208 db_command_loop() at db_command_loop+0x74 pc = 0xc00c5b38 lr = 0xc00c9530 (db_trap+0x12c) sp = 0xc2dbe950 fp = 0xc2dbea68 r4 = 0x00000000 r5 = 0xc08c0ef4 r6 = 0xc2dbeb20 r10 = 0xc06dd208 db_trap() at db_trap+0x12c pc = 0xc00c9530 lr = 0xc02a8c2c (kdb_trap+0x19c) sp = 0xc2dbea70 fp = 0xc2dbea98 r4 = 0x00000000 r5 = 0x00000001 r6 = 0xc2dbeb20 r7 = 0xc08589e0 kdb_trap() at kdb_trap+0x19c pc = 0xc02a8c2c lr = 0xc057b5e4 (undefinedinstruction+0x214) sp = 0xc2dbeaa0 fp = 0xc2dbeb18 r4 = 0x00000000 r5 = 0x00000000 r6 = 0xc057b320 r7 = 0xe7ffffff r8 = 0xc2dbeb20 r9 = 0x00000001 r10 = 0xc02a8534 undefinedinstruction() at undefinedinstruction+0x214 pc = 0xc057b5e4 lr = 0xc055eda4 (exception_exit) sp = 0xc2dbeb20 fp = 0xc2dbebc0 r4 = 0x60000193 r5 = 0xd87ad490 r6 = 0x00000001 r7 = 0x00040000 r8 = 0x00000000 r9 = 0x00000000 r10 = 0xc06ea12c exception_exit() at exception_exit pc = 0xc055eda4 lr = 0xc02a851c (kdb_alt_break_internal+0x1b0) sp = 0xc2dbebb0 fp = 0xc2dbebc0 r0 = 0xc08589d0 r1 = 0x00000000 r2 = 0xc2dbeaec r3 = 0x60000193 r4 = 0xd87ad400 r5 = 0xd87ad490 r6 = 0x00000001 r7 = 0x00040000 r8 = 0x00000000 r9 = 0x00000000 r10 = 0xc06ea12c r12 = 0x00000001 $a.6() at $a.6 pc = 0xc02a8538 lr = 0xc02a8364 (kdb_alt_break+0x10) sp = 0xc2dbebc8 fp = 0xc2dbebc8 r4 = 0xd87ad400 r5 = 0xd87ad490 r6 = 0x00000001 r7 = 0x00040000 kdb_alt_break() at kdb_alt_break+0x10 pc = 0xc02a8364 lr = 0xc012c1b4 (uart_intr_rxready+0x8c) sp = 0xc2dbebd0 fp = 0xc2dbebe0 uart_intr_rxready() at uart_intr_rxready+0x8c pc = 0xc012c1b4 lr = 0xc012cff0 (uart_intr+0x11c) sp = 0xc2dbebe8 fp = 0xc2dbec20 r4 = 0xc4aaf800 r5 = 0xd87ad400 r6 = 0xd87ad59c r10 = 0xc06ea12c uart_intr() at uart_intr+0x11c pc = 0xc012cff0 lr = 0xc0226054 (intr_event_handle+0xa4) sp = 0xc2dbec28 fp = 0xc2dbec48 r4 = 0xc4aaf800 r5 = 0xc2dbed28 r6 = 0xd879d058 r7 = 0xc4a69b40 r8 = 0xc2e80600 r9 = 0x00000000 r10 = 0xc2dbed28 intr_event_handle() at intr_event_handle+0xa4 pc = 0xc0226054 lr = 0xc0560cb4 (intr_isrc_dispatch+0x68) sp = 0xc2dbec50 fp = 0xc2dbec58 r4 = 0xd879d058 r5 = 0xc2dbed28 r6 = 0xd879d058 r7 = 0x00000041 r8 = 0xc0638409 r9 = 0xd879c018 r10 = 0x001fffff intr_isrc_dispatch() at intr_isrc_dispatch+0x68 pc = 0xc0560cb4 lr = 0xc05892e4 (bcm2835_intc_intr+0x88) sp = 0xc2dbec60 fp = 0xc2dbec80 r4 = 0xd879c000 r5 = 0x00000000 bcm2835_intc_intr() at bcm2835_intc_intr+0x88 pc = 0xc05892e4 lr = 0xc0226054 (intr_event_handle+0xa4) sp = 0xc2dbec88 fp = 0xc2dbeca8 r4 = 0xc4ab0600 r5 = 0xc2dbed28 r6 = 0x00000000 r7 = 0xc4a69b40 r8 = 0xc2e80980 r9 = 0x00000000 r10 = 0xc2dbed28 intr_event_handle() at intr_event_handle+0xa4 pc = 0xc0226054 lr = 0xc0560cb4 (intr_isrc_dispatch+0x68) sp = 0xc2dbecb0 fp = 0xc2dbecb8 r4 = 0xd879a204 r5 = 0xc2dbed28 r6 = 0x00000000 r7 = 0x00000000 r8 = 0x00000100 r9 = 0xc0856cc4 r10 = 0xc2dbed28 intr_isrc_dispatch() at intr_isrc_dispatch+0x68 pc = 0xc0560cb4 lr = 0xc0580dac (bcm_lintc_intr+0x268) sp = 0xc2dbecc0 fp = 0xc2dbed00 r4 = 0xd879a000 r5 = 0x000000c0 bcm_lintc_intr() at bcm_lintc_intr+0x268 pc = 0xc0580dac lr = 0xc0560b14 (intr_irq_handler+0x90) sp = 0xc2dbed08 fp = 0xc2dbed20 r4 = 0xc2dbed28 r5 = 0xc08bba60 r6 = 0xc4a69b40 r7 = 0x00000000 r8 = 0xc072bef8 r9 = 0xc0856cc4 r10 = 0x00000000 intr_irq_handler() at intr_irq_handler+0x90 pc = 0xc0560b14 lr = 0xc055eda4 (exception_exit) sp = 0xc2dbed28 fp = 0xc2dbedc0 r4 = 0xc4a69b40 r5 = 0x00000000 r6 = 0xc0856cd4 r7 = 0x00000000 r8 = 0xffffffff r10 = 0x00000000 exception_exit() at exception_exit pc = 0xc055eda4 lr = 0xc0563b68 (cpu_idle+0x74) sp = 0xc2dbedb8 fp = 0xc2dbedc0 r0 = 0x00000000 r1 = 0x000001c0 r2 = 0x600000d3 r3 = 0x60000013 r4 = 0xc4a69b40 r5 = 0x00000000 r6 = 0xc0856cd4 r7 = 0x00000000 r8 = 0xffffffff r9 = 0xc0856cc4 r10 = 0x00000000 r12 = 0xd874f180 cpu_idle() at cpu_idle+0xa0 pc = 0xc0563b94 lr = 0xc0290564 (sched_idletd+0x510) sp = 0xc2dbedc8 fp = 0xc2dbee20 r4 = 0xc0856cc8 r10 = 0x00000000 sched_idletd() at sched_idletd+0x510 pc = 0xc0290564 lr = 0xc0223064 (fork_exit+0xa0) sp = 0xc2dbee28 fp = 0xc2dbee40 r4 = 0xc4a69b40 r5 = 0xc4a60ac8 r6 = 0xc0290054 r7 = 0x00000000 r8 = 0xc2dbee48 r9 = 0x00000000 r10 = 0x00000000 fork_exit() at fork_exit+0xa0 pc = 0xc0223064 lr = 0xc055ed34 (swi_exit) sp = 0xc2dbee48 fp = 0x00000000 r4 = 0xc0290054 r5 = 0x00000000 r6 = 0x00000000 r7 = 0x00000000 r8 = 0x00000000 r10 = 0x00000000 swi_exit() at swi_exit pc = 0xc055ed34 lr = 0xc055ed34 (swi_exit) sp = 0xc2dbee48 fp = 0x00000000
'show witness' output: Sleep locks: mmcsd disk (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/mmc/mmcsd.c:1446 USB control transfer SX lock (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/usb/usb_device.c:2970 usb_def_mtx (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/usb/usb_request.c:488 USB config SX lock (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/usb/usb_device.c:2884 Giant (type: sleep mutex, depth: 1, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_module.c:116 pipe mutex (type: sleep mutex, depth: 2, active refs: 1) -- never acquired sigio lock (type: sleep mutex, depth: 3, active refs: 2) -- never acquired process group (type: sleep mutex, depth: 4, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_fork.c:589 pmc-sleep (type: sleep mutex, depth: 5, active refs: 1) -- never acquired process lock (type: sleep mutex, depth: 6, active refs: 42) -- last acquired @ /e-drive/src.current/sys/kern/kern_thread.c:1230 session (type: sleep mutex, depth: 7, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_fork.c:599 uidinfo hash (type: rw, depth: 8, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_resource.c:1281 time lock (type: sleep mutex, depth: 9, active refs: 2) -- never acquired jail mutex (type: sleep mutex, depth: 7, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_jail.c:2680 osd_object (type: rm, depth: 8, active refs: 3) -- last acquired @ /e-drive/src.current/sys/kern/kern_osd.c:227 osd_list (type: sleep mutex, depth: 9, active refs: 3) -- last acquired @ /e-drive/src.current/sys/kern/kern_osd.c:270 sigacts (type: sleep mutex, depth: 7, active refs: 29) -- last acquired @ /e-drive/src.current/sys/kern/kern_kthread.c:112 process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed) process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed) uidinfo hash (type: rw, depth: 8, active refs: 2) -- (already displayed) igmp_mtx (type: sleep mutex, depth: 3, active refs: 2) -- last acquired @ /e-drive/src.current/sys/netinet/igmp.c:2165 ifnet_rw (type: rw, depth: 4, active refs: 3) -- last acquired @ /e-drive/src.current/sys/net/if.c:1420 if_addr_lock (type: sleep mutex, depth: 5, active refs: 3) -- last acquired @ /e-drive/src.current/sys/net/if.c:1463 mld_mtx (type: sleep mutex, depth: 3, active refs: 2) -- last acquired @ /e-drive/src.current/sys/netinet6/mld6.c:1751 ifnet_rw (type: rw, depth: 4, active refs: 3) -- (already displayed) vm map (system) (type: sleep mutex, depth: 2, active refs: 146) -- last acquired @ /e-drive/src.current/sys/vm/vm_map.c:1577 vnode interlock (type: sleep mutex, depth: 3, active refs: 42) -- never acquired cdev (type: sleep mutex, depth: 4, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_conf.c:74 devfs dirlist lock (type: sleep mutex, depth: 5, active refs: 1) -- last acquired @ /e-drive/src.current/sys/fs/devfs/devfs_dir.c:66 ncneg (type: sleep mutex, depth: 4, active refs: 1) -- never acquired vm page free queue (type: sleep mutex, depth: 7, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/vm_page.c:1841 vm pagequeue (type: sleep mutex, depth: 8, active refs: 5) -- never acquired SYSMAPS (type: sleep mutex, depth: 4, active refs: 4) -- last acquired @ /e-drive/src.current/sys/arm/arm/pmap-v6.c:1633 128 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:3856 kernel vm object (type: rw, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/vm_kern.c:437 pmap pv global (type: rw, depth: 4, active refs: 2) -- last acquired @ /e-drive/src.current/sys/arm/arm/pmap-v6.c:3865 pmap (type: sleep mutex, depth: 5, active refs: 67) -- last acquired @ /e-drive/src.current/sys/arm/arm/pmap-v6.c:1942 pmap pv list (type: rw, depth: 6, active refs: 1) -- never acquired vm page free queue (type: sleep mutex, depth: 7, active refs: 2) -- (already displayed) vm page free queue (type: sleep mutex, depth: 7, active refs: 2) -- (already displayed) SYSMAPS (type: sleep mutex, depth: 4, active refs: 4) -- (already displayed) UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:2323 16 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:2323 256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:2565 vm reserv (type: sleep mutex, depth: 4, active refs: 963) -- last acquired @ /e-drive/src.current/sys/vm/vm_reserv.c:869 VM reserv domain (type: sleep mutex, depth: 5, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/vm_reserv.c:539 resv obj lock (type: sleep mutex, depth: 5, active refs: 4) -- last acquired @ /e-drive/src.current/sys/vm/vm_reserv.c:394 resv obj lock (type: sleep mutex, depth: 5, active refs: 4) -- (already displayed) KMAP ENTRY (type: sleep mutex, depth: 3, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:3856 kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_vmem.c:1180 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 16 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) 256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) cdev (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) vm object (type: rw, depth: 2, active refs: 139) -- last acquired @ /e-drive/src.current/sys/vm/vm_glue.c:387 vm page (type: sleep mutex, depth: 3, active refs: 33) -- never acquired pmap pv global (type: rw, depth: 4, active refs: 2) -- (already displayed) vm page free queue (type: sleep mutex, depth: 7, active refs: 2) -- (already displayed) SYSMAPS (type: sleep mutex, depth: 4, active refs: 4) -- (already displayed) UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) pmap pv global (type: rw, depth: 4, active refs: 2) -- (already displayed) pmap (type: sleep mutex, depth: 5, active refs: 67) -- (already displayed) vm page free queue (type: sleep mutex, depth: 7, active refs: 2) -- (already displayed) SYSMAPS (type: sleep mutex, depth: 4, active refs: 4) -- (already displayed) UMA lock (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:1559 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 4 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:3856 8 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:3856 12 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:3856 16 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) 32 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:2565 64 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:3856 128 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed) 256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) vmem list lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_vmem.c:1067 btag lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_vmem.c:634 kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) vm object_list (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/vm_object.c:229 kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_vmem.c:883 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 16 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) 256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) swapdev (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/swap_pager.c:208 malloc (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_malloc.c:1001 OF xreflist lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/ofw/openfirm.c:164 vtdev (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_mutex.c:186 kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_environment.c:506 kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed) USB quirk (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/usb/quirk/usb_quirk.c:671 protect sysfilt_ops (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_event.c:1204 encapmtx (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet/ip_encap.c:369 domain list (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/uipc_domain.c:264 intr config (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_autoconf.c:159 devstat (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_devstat.c:97 intr event list (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_intr.c:267 jail mutex (type: sleep mutex, depth: 7, active refs: 1) -- (already displayed) root_holds (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/vfs_mountroot.c:178 XPT topology lock (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/cam/cam_xpt.c:5360 kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed) CAM bus lock (type: sleep mutex, depth: 4, active refs: 1) -- last acquired @ /e-drive/src.current/sys/cam/cam_xpt.c:4861 CAM queue lock (type: sleep mutex, depth: 5, active refs: 1) -- last acquired @ /e-drive/src.current/sys/cam/cam_xpt.c:4913 kstkch (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/vm_glue.c:330 if_cloners lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/net/if_clone.c:367 et_mtx (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_clocksource.c:672 loginclasses lock (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_loginclass.c:158 lltable_list_lock (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/net/if_llatbl.c:642 chacha20_mtx (type: sleep mutex, depth: 2, active refs: 4) -- last acquired @ /e-drive/src.current/sys/libkern/arc4random.c:172 kobj (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_kobj.c:288 bounce pages lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/arm/arm/busdma_machdep-v6.c:1593 eventhandler (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_eventhandler.c:118 eventhandler list (type: sleep mutex, depth: 2, active refs: 32) -- last acquired @ /e-drive/src.current/sys/kern/subr_eventhandler.c:129 vnet_rwlock (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet6/mld6.c:1313 igmp_mtx (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed) mld_mtx (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed) IP reassembly (type: sleep mutex, depth: 3, active refs: 64) -- last acquired @ /e-drive/src.current/sys/netinet/ip_reass.c:492 tcptw (type: rw, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet/tcp_timewait.c:697 ip6qlock (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet6/frag6.c:752 USB ref mutex (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/usb/usb_hub.c:2243 buffer arena (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_vmem.c:1020 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) transient arena (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_vmem.c:1344 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) bufq empty lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/vfs_bio.c:1958 reseed mutex (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/random/fortuna.c:223 TID lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_unit.c:644 tidhash (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_thread.c:1223 process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed) ui_vmsize (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/swap_pager.c:225 intr map table (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_intr.c:1521 intr pic list (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_intr.c:729 intr isrc table (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_intr.c:1205 taskqgroup (type: sleep mutex, depth: 2, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/subr_gtaskqueue.c:929 intr event (type: sleep mutex, depth: 2, active refs: 29) -- last acquired @ /e-drive/src.current/sys/kern/kern_intr.c:581 thread (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_taskqueue.c:467 GEOM orphanage (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/geom/geom_event.c:402 devd (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_bus.c:617 crypto driver table (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/opencrypto/crypto.c:744 LED mtx (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/led/led.c:295 XPT lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/cam/cam_xpt.c:936 XPT topology lock (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) CAM XPT task (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_taskqueue.c:467 CAM queue lock (type: sleep mutex, depth: 5, active refs: 1) -- (already displayed) CAM bus lock (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed) CAM bus lock (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed) CAM device lock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/cam/cam_xpt.c:2338 XPT topology lock (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) CAM bus lock (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed) rman head (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_rman.c:150 rman (type: sleep mutex, depth: 2, active refs: 4) -- last acquired @ /e-drive/src.current/sys/kern/subr_rman.c:670 bcmdma (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/arm/broadcom/bcm2835/bcm2835_dma.c:331 vcio mbox (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/arm/broadcom/bcm2835/bcm2835_mbox.c:243 ttymtx (type: sleep mutex, depth: 2, active refs: 13) -- last acquired @ /e-drive/src.current/sys/kern/subr_terminal.c:274 bcm_bsc (type: sleep mutex, depth: 2, active refs: 3) -- last acquired @ /e-drive/src.current/sys/arm/broadcom/bcm2835/bcm2835_bsc.c:679 sdhci (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/sdhci/sdhci.c:2006 kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed) kobj (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) mmc (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/mmc/mmc.c:427 mmc (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) usb_def_mtx (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) VCHI spinlock &service_spinlock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_core.c:308 VCHI rwlock (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_arm.c:2741 condvar (type: sleep mutex, depth: 3, active refs: 5) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/compat/vchi_bsd.c:271 condvar (type: sleep mutex, depth: 3, active refs: 5) -- (already displayed) bcm_audio_lock (type: sleep mutex, depth: 3, active refs: 1) -- last acquired @ /e-drive/src.current/sys/arm/broadcom/bcm2835/bcm2835_audio.c:487 vnode_free_list (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/vfs_subr.c:1417 newnfsd_mtx (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/fs/nfsclient/nfs_clkrpc.c:296 ncl_iod_mutex (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/fs/nfsclient/nfs_clnfsiod.c:220 if_clone lock (type: sleep mutex, depth: 2, active refs: 3) -- last acquired @ /e-drive/src.current/sys/net/if_clone.c:354 if_afdata (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/net/if.c:943 IPsec transforms list (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netipsec/key.c:8628 cc_list (type: rw, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet/cc/cc.c:264 sound cdev (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/sound/pcm/sound.c:784 pcm play channel (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/sound/pcm/channel.c:1813 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 8 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed) kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) kobj (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) bcm_audio_lock (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) pcm virtual play channel (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/sound/pcm/channel.c:1226 kobj (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) sctp_it_wq (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet/sctp_bsd_addr.c:88 shpsx (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_sharedpage.c:88 soaio jobs (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/sys_socket.c:462 Syncer mtx (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/vfs_subr.c:2199 &g_connected_mutex (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_connected.c:108 &state->recycle_mutex (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_core.c:1124 &state->mutex (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_core.c:2614 proctree (type: sx, depth: 1, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_fork.c:588 allproc (type: sx, depth: 2, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/kern_fork.c:959 allprison (type: sx, depth: 3, active refs: 2) -- last acquired @ /e-drive/src.current/sys/kern/sysv_msg.c:289 process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed) process group (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed) process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed) vm object (type: rw, depth: 2, active refs: 139) -- (already displayed) UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 4 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed) 64 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed) vm object_list (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) kstkch (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) eventhandler list (type: sleep mutex, depth: 2, active refs: 32) -- (already displayed) TID lock (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) tidhash (type: rw, depth: 2, active refs: 1) -- (already displayed) filedesc structure (type: sx, depth: 1, active refs: 29) -- last acquired @ /e-drive/src.current/sys/kern/kern_descrip.c:1956 VCHI spinlock &service_spinlock (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) &state->slot_mutex (type: sx, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_core.c:785 VCHI spinlock "a_spinlock (type: sleep mutex, depth: 2, active refs: 1) -- last acquired @ /e-drive/src.current/sys/contrib/vchiq/interface/vchiq_arm/vchiq_core.c:919 VCHI rwlock (type: rw, depth: 2, active refs: 1) -- (already displayed) mboxprop (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/arm/broadcom/bcm2835/bcm2835_mbox.c:402 Giant (type: sleep mutex, depth: 1, active refs: 2) -- (already displayed) UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 64 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed) 256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) vcio mbox (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) CAM doneq (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/cam/cam_xpt.c:5510 LED sx (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/led/led.c:286 cdev (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) unit# allocation (type: sleep mutex, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_unit.c:644 crypto return queues (type: sleep mutex, depth: 0, active refs: 4) -- last acquired @ /e-drive/src.current/sys/opencrypto/crypto.c:1584 crypto (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_taskqueue.c:754 crypto op queues (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/opencrypto/crypto.c:1438 GEOM topology (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/geom/geom_event.c:290 cdev (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 32 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed) 64 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed) kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) sysctl lock (type: sleepable rm, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_sysctl.c:824 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed) kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed) sleep mtxpool (type: sleep mutex, depth: 1, active refs: 128) -- last acquired @ /e-drive/src.current/sys/kern/vfs_bio.c:4364 devstat (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) root_holds (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) bio queue (type: sleep mutex, depth: 1, active refs: 2) -- last acquired @ /e-drive/src.current/sys/geom/geom_io.c:103 GEOM orphanage (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) devd (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) sndstat lock (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/sound/pcm/sndstat.c:264 kqueue_ctx (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_taskqueue.c:754 aiod_kick (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_taskqueue.c:754 user map (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/vm_map.c:3429 pbuf mutex (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/swap_pager.c:512 kernel linker (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_linker.c:592 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 4 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed) kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) kobj (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) module subsystem sx lock (type: sx, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_module.c:133 UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 128 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed) kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) USB sym mutex (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/dev/usb/usb_dev.c:2320 vnet_sysinit_sxlock (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/net/vnet.c:482 proctree (type: sx, depth: 1, active refs: 2) -- (already displayed) Giant (type: sleep mutex, depth: 1, active refs: 2) -- (already displayed) process lock (type: sleep mutex, depth: 6, active refs: 42) -- (already displayed) bpf global lock (type: sx, depth: 1, active refs: 2) -- last acquired @ /e-drive/src.current/sys/net/bpf.c:2598 bpf interface lock (type: rw, depth: 2, active refs: 2) -- never acquired bpf cdev lock (type: sleep mutex, depth: 3, active refs: 1) -- never acquired vm object (type: rw, depth: 2, active refs: 139) -- (already displayed) UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 4 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed) 8 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed) 16 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) 32 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed) 128 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed) 256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) umadrain (type: sx, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/uma_core.c:2033 UMA lock (type: rw, depth: 2, active refs: 1) -- (already displayed) UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) kernel environment (type: sleep mutex, depth: 4, active refs: 1) -- (already displayed) hhook_head list lock (type: sleep mutex, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_hhook.c:320 helper list lock (type: rw, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/kern_khelp.c:296 kstkch (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) ifnet_sx (type: sx, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet/sctp_bsd_addr.c:209 ifnet_rw (type: rw, depth: 4, active refs: 3) -- (already displayed) if_cloners lock (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) pfil_head_list lock (type: sleep mutex, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/net/pfil.c:211 chacha20_mtx (type: sleep mutex, depth: 2, active refs: 4) -- (already displayed) eventhandler (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) eventhandler list (type: sleep mutex, depth: 2, active refs: 32) -- (already displayed) vnet_rwlock (type: rw, depth: 2, active refs: 1) -- (already displayed) reseed mutex (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) TID lock (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) tidhash (type: rw, depth: 2, active refs: 1) -- (already displayed) filedesc structure (type: sx, depth: 1, active refs: 29) -- (already displayed) devd (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) if_clone lock (type: sleep mutex, depth: 2, active refs: 3) -- (already displayed) tcp_func_lock (type: rw, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/netinet/tcp_subr.c:825 rib head lock (type: rw, depth: 1, active refs: 2) -- last acquired @ /e-drive/src.current/sys/net/route.c:989 vnet_sxlock (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/net/vnet.c:260 vnet_rwlock (type: rw, depth: 2, active refs: 1) -- (already displayed) osd_module (type: sx, depth: 0, active refs: 3) -- last acquired @ /e-drive/src.current/sys/kern/kern_osd.c:110 osd_object (type: rm, depth: 8, active refs: 3) -- (already displayed) vm daemon (type: sleep mutex, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/vm/vm_swapout.c:391 vfsconf (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/vfs_init.c:190 cdev (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 4 Bucket (type: sleep mutex, depth: 2, active refs: 2) -- (already displayed) 256 Bucket (type: sleep mutex, depth: 4, active refs: 2) -- (already displayed) kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) umadrain (type: sx, depth: 1, active refs: 1) -- (already displayed) sysctl lock (type: sleepable rm, depth: 1, active refs: 1) -- (already displayed) jail mutex (type: sleep mutex, depth: 7, active refs: 1) -- (already displayed) eventhandler (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) eventhandler list (type: sleep mutex, depth: 2, active refs: 32) -- (already displayed) devd (type: sleep mutex, depth: 2, active refs: 1) -- (already displayed) pfs_fileno (type: sleep mutex, depth: 1, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/subr_unit.c:644 pfs_node (type: sleep mutex, depth: 1, active refs: 20) -- last acquired @ /e-drive/src.current/sys/fs/pseudofs/pseudofs_internal.h:105 tty list (type: sx, depth: 0, active refs: 1) -- last acquired @ /e-drive/src.current/sys/kern/tty.c:1357 TCP ID tree (type: rw, depth: 0, active refs: 1) -- never acquired tcp log id bucket (type: sleep mutex, depth: 1, active refs: 1) -- never acquired tcpinp (type: rw, depth: 2, active refs: 2) -- never acquired so_snd (type: sleep mutex, depth: 3, active refs: 5) -- never acquired so_rcv (type: sleep mutex, depth: 4, active refs: 2) -- never acquired sellck (type: sleep mutex, depth: 5, active refs: 1) -- never acquired radix node head (type: rw, depth: 5, active refs: 1) -- never acquired rtentry (type: sleep mutex, depth: 6, active refs: 1) -- never acquired ifaddr (type: sleep mutex, depth: 7, active refs: 1) -- never acquired TCP log expireq (type: sleep mutex, depth: 3, active refs: 1) -- never acquired dn->dn_mtx (type: sx, depth: 0, active refs: 1) -- never acquired dr->dt.di.dr_mtx (type: sx, depth: 1, active refs: 1) -- never acquired db->db_mtx (type: sx, depth: 2, active refs: 1) -- never acquired ncvn (type: sleep mutex, depth: 0, active refs: 513) -- never acquired ncbuc (type: rw, depth: 1, active refs: 513) -- never acquired vnode interlock (type: sleep mutex, depth: 3, active refs: 42) -- (already displayed) kqueue (type: sleep mutex, depth: 0, active refs: 1) -- never acquired struct mount mtx (type: sleep mutex, depth: 1, active refs: 1) -- never acquired vnode interlock (type: sleep mutex, depth: 3, active refs: 42) -- (already displayed) vm map (user) (type: sx, depth: 0, active refs: 146) -- last acquired @ /e-drive/src.current/sys/vm/vm_kern.c:576 vm object (type: rw, depth: 2, active refs: 139) -- (already displayed) UMA zone (type: sleep mutex, depth: 4, active refs: 408) -- (already displayed) 128 Bucket (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed) kernel vm object (type: rw, depth: 3, active refs: 1) -- (already displayed) kernel arena domain (type: sleep mutex, depth: 3, active refs: 1) -- (already displayed) ng_node (type: sleep mutex, depth: 0, active refs: 1) -- never acquired ng_worklist (type: sleep mutex, depth: 1, active refs: 1) -- never acquired network driver (type: sleep mutex, depth: 0, active refs: 1) -- never acquired 802.11 com lock (type: sleep mutex, depth: 0, active refs: 1) -- never acquired nfsd_mtx (type: sleep mutex, depth: 0, active refs: 1) -- never acquired so_snd (type: sleep mutex, depth: 3, active refs: 5) -- (already displayed) tcp (type: rw, depth: 0, active refs: 2) -- never acquired tcpinp (type: rw, depth: 2, active refs: 2) -- (already displayed) udp (type: rw, depth: 0, active refs: 2) -- never acquired udpinp (type: rw, depth: 1, active refs: 3) -- never acquired so_snd (type: sleep mutex, depth: 3, active refs: 5) -- (already displayed) in_multi_list_mtx (type: sleep mutex, depth: 2, active refs: 2) -- never acquired igmp_mtx (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed) in6_multi_list_mtx (type: sleep mutex, depth: 2, active refs: 2) -- never acquired mld_mtx (type: sleep mutex, depth: 3, active refs: 2) -- (already displayed) unp_link_rwlock (type: rw, depth: 0, active refs: 2) -- never acquired unp_list_lock (type: sleep mutex, depth: 1, active refs: 1) -- never acquired unp (type: sleep mutex, depth: 2, active refs: 1) -- never acquired so_snd (type: sleep mutex, depth: 3, active refs: 5) -- (already displayed) in6_multi_sx (type: sx, depth: 0, active refs: 2) -- never acquired udpinp (type: rw, depth: 1, active refs: 3) -- (already displayed) in_multi_sx (type: sx, depth: 0, active refs: 2) -- never acquired udpinp (type: rw, depth: 1, active refs: 3) -- (already displayed) accept (type: sleep mutex, depth: 0, active refs: 2) -- never acquired so_snd (type: sleep mutex, depth: 3, active refs: 5) -- (already displayed) umtx lock (type: sleep mutex, depth: 0, active refs: 2) -- never acquired Spin locks: Locks which were never acquired: select mtxpool (type: sleep mutex, depth: 0) g_dev (type: sleep mutex, depth: 0) g_disk_done (type: sleep mutex, depth: 0) g_disk_start (type: sleep mutex, depth: 0) primary pcm mixer (type: sleep mutex, depth: 0) &service->bulk_mutex (type: sx, depth: 0) &instance->bulk_waiter_list_mutex (type: sx, depth: 0) mmcsd IOCTL (type: sleep mutex, depth: 0) USB HUB mutex (type: sleep mutex, depth: 0) iicbus (type: sleep mutex, depth: 0) USB device mutex (type: sleep mutex, depth: 0) MD config lock (type: sx, depth: 0) ip_id_mtx (type: sleep mutex, depth: 0) unp_defer (type: sleep mutex, depth: 0) nd6 DAD queue (type: rw, depth: 0) nd6 list (type: rw, depth: 0) nd6 onlink (type: sleep mutex, depth: 0) addrsel_sxlock (type: sx, depth: 0) addrsel_lock (type: sleep mutex, depth: 0) scope6_lock (type: sleep mutex, depth: 0) fast ipsec security policy acquire list (type: sleep mutex, depth: 0) ipsec SA acquiring list (type: sleep mutex, depth: 0) sahtree (type: rm, depth: 0) fast ipsec regtree (type: sleep mutex, depth: 0) sptree (type: rm, depth: 0) rip (type: rw, depth: 0) udplite (type: rw, depth: 0) iterator (type: sleep mutex, depth: 0) sctp_addr_wq (type: sleep mutex, depth: 0) sctp-addr (type: rw, depth: 0) sctp-info (type: rw, depth: 0) isn_mtx (type: sleep mutex, depth: 0) tcp_hc_entry (type: sleep mutex, depth: 0) tcp_sc_head (type: sleep mutex, depth: 0) pcbinfolist (type: rw, depth: 0) pcbinfohash (type: rw, depth: 0) hhook_head rm lock (type: rm, depth: 0) PFil shared rmlock (type: rm, depth: 0) exec args kva (type: sleep mutex, depth: 0) pfs_vncache (type: sleep mutex, depth: 0) if send queue (type: sleep mutex, depth: 0) msq (type: sleep mutex, depth: 0) semu (type: sleep mutex, depth: 0) sem (type: sleep mutex, depth: 0) semid (type: sleep mutex, depth: 0) shm dictionary (type: sx, depth: 0) shm timestamps (type: sleep mutex, depth: 0) sysvshmsx (type: sx, depth: 0) nlm_global_lock (type: sleep mutex, depth: 0) aio_job (type: sleep mutex, depth: 0) nfsuserd (type: sleep mutex, depth: 0) nfs_req_mutex (type: sleep mutex, depth: 0) nfs_slock_mutex (type: sleep mutex, depth: 0) nfs_sockl_mutex (type: sleep mutex, depth: 0) nfs_nameid_mutex (type: sleep mutex, depth: 0) dirhash list (type: sleep mutex, depth: 0) dqhlock (type: sleep mutex, depth: 0) ffs_susp (type: sx, depth: 0) ncnegs (type: sleep mutex, depth: 0) ncneglh (type: sleep mutex, depth: 0) ncnegl (type: sleep mutex, depth: 0) crossmp (type: lockmgr, depth: 0) bufobj interlock (type: rw, depth: 0) vnode (type: lockmgr, depth: 0) vfs hash (type: rw, depth: 0) sg_lock (type: sleep mutex, depth: 0) sp_lock (type: sleep mutex, depth: 0) nfs_clstate_mutex (type: sleep mutex, depth: 0) nfs_state_mutex (type: sleep mutex, depth: 0) mntid (type: sleep mutex, depth: 0) swcr_sessions_lock (type: rw, depth: 0) cpufreq lock (type: sx, depth: 0) &state->bulk_transfer_mutex (type: sx, depth: 0) &state->sync_mutex (type: sx, depth: 0) VCHI spinlock &msg_queue_spinlock (type: sleep mutex, depth: 0) vchiq (type: sleep mutex, depth: 0) VCHI spinlock &bulk_waiter_spinlock (type: sleep mutex, depth: 0) spigen2 (type: sleep mutex, depth: 0) spigen1 (type: sleep mutex, depth: 0) spigen0 (type: sleep mutex, depth: 0) bcm_spi (type: sleep mutex, depth: 0) bcmwd (type: sleep mutex, depth: 0) gpiobus (type: sleep mutex, depth: 0) CAM LUNs lock (type: sleep mutex, depth: 0) XPT highpower lock (type: sleep mutex, depth: 0) nfslock (type: sleep mutex, depth: 0) sysmouse (type: sleep mutex, depth: 0) midistat lock (type: sleep mutex, depth: 0) netisr_mtx (type: sleep mutex, depth: 0) swi_giant (type: sleep mutex, depth: 0) swi (type: sleep mutex, depth: 0) sf_buf (type: sleep mutex, depth: 0) ktrace_sx (type: sx, depth: 0) ktrace (type: sleep mutex, depth: 0) system map (type: sleep mutex, depth: 0) p_peers (type: sleep mutex, depth: 0) bufspace daemon run lock (type: sleep mutex, depth: 0) bufq clean subqueue lock (type: sleep mutex, depth: 0) bufq dirty lock (type: sleep mutex, depth: 0) bufq clean lock (type: sleep mutex, depth: 0) bufwait (type: lockmgr, depth: 0) dirty buf lock (type: sleep mutex, depth: 0) buffer daemon lock (type: sleep mutex, depth: 0) runningbufspace lock (type: sleep mutex, depth: 0) vnet_data alloc lock (type: sx, depth: 0) dpcpu alloc lock (type: sx, depth: 0) umtxshm (type: sleep mutex, depth: 0) umtxql (type: sleep mutex, depth: 0) ipsec_ioctl (type: sx, depth: 0) if_ipsec softc list (type: rm, depth: 0) CAM SIM free lock (type: sleep mutex, depth: 0) NFS create verifier mutex (type: sleep mutex, depth: 0) mem4map (type: sx, depth: 0) PCM channel sync group lock (type: sleep mutex, depth: 0) in6_ifaddr_lock (type: rm, depth: 0) in6_multi_free_mtx (type: sleep mutex, depth: 0) ifnet descr (type: sx, depth: 0) DEVFS ruleset lock (type: sx, depth: 0) rtc list (type: sx, depth: 0) Global Softdep Lock (type: sleep mutex, depth: 0) mountlist (type: sleep mutex, depth: 0) db_script_mtx (type: sleep mutex, depth: 0) msgbuf lock (type: sleep mutex, depth: 0) pmc-sx (type: sx, depth: 0) tty snoop registration (type: sx, depth: 0) db_capture_sx (type: sx, depth: 0) so_glabel (type: sleep mutex, depth: 0) cdevpriv lock (type: sleep mutex, depth: 0) fail point set sx (type: sx, depth: 0) in_ifaddr_lock (type: rm, depth: 0) acct_sx (type: sx, depth: 0) in_multi_free_mtx (type: sleep mutex, depth: 0) accept_filter_mtx (type: sleep mutex, depth: 0) UUID generator mutex lock (type: sleep mutex, depth: 0) snapdata free list (type: sleep mutex, depth: 0) clone events drain lock (type: sx, depth: 0) knlist lock for lockless objects (type: sleep mutex, depth: 0) in_control (type: sx, depth: 0) kqueue order (type: sleep mutex, depth: 0) rtsock route_cb lock (type: sleep mutex, depth: 0) devfs interlock (type: sleep mutex, depth: 0) rawcb (type: sleep mutex, depth: 0) NLM svid lock (type: sleep mutex, depth: 0) owner graph lock (type: sx, depth: 0) lock owners lock (type: sx, depth: 0) lock states lock (type: sx, depth: 0) auth_unix_lock (type: sx, depth: 0) sysctl mem (type: sx, depth: 0) phys_pager list (type: sleep mutex, depth: 0) dev_pager list (type: sleep mutex, depth: 0) swsysc (type: sx, depth: 0) swspsx (type: sx, depth: 0) vm map sleep mutex (type: sleep mutex, depth: 0) vmfctr (type: rw, depth: 0) 6 Bucket (type: sleep mutex, depth: 0) vm pageout lock (type: sleep mutex, depth: 0) vm domainset lock (type: sleep mutex, depth: 0) PMAP2 (type: sleep mutex, depth: 0)
ps output: pid ppid pgrp uid state wmesg wchan cmd 28 0 0 0 DL syncer 0xc08b7520 [syncer] 27 0 0 0 LL *kernel 0xc2e77880 [vnlru] 26 0 0 0 LL *kernel 0xc2e77880 [bufdaemon] 25 0 0 0 DL psleep 0xc08bb8ec [vmdaemon] 24 0 0 0 LL *kernel 0xc2e77880 [dom0] 23 0 0 0 SL worker_ 0xc2e99b90 [bcm2835_audio_worke] 22 0 0 0 SL VCHI co 0xd87df80c [VCHIQka-0] 21 0 0 0 DL mmcsd d 0xc2e99a00 [mmcsd0: mmc/sd card] 20 0 0 0 DL - 0xc08b6aa4 [soaiod4] 19 0 0 0 DL - 0xc08b6aa4 [soaiod3] 18 0 0 0 DL - 0xc08b6aa4 [soaiod2] 17 0 0 0 DL - 0xc08b6aa4 [soaiod1] 16 0 0 0 DL - 0xc0745120 [rand_harvestq] 15 0 0 0 DL waiting 0xc08c4fa8 [sctp_iterator] 14 0 0 0 SL sema cv 0xc08dceb0 [VCHIQs-0] 9 0 0 0 SL sema cv 0xc08dce8c [VCHIQr-0] 8 0 0 0 SL sema cv 0xc08dce68 [VCHIQ-0] 13 0 0 0 LL (threaded) [usb] 100043 D - 0xd87b302c [usbus0] 100044 D - 0xd87b305c [usbus0] 100045 D - 0xd87b308c [usbus0] 100046 L *Giant 0xc2e7a800 [usbus0] 100047 D - 0xd87b30ec [usbus0] 7 0 0 0 DL (threaded) [cam] 100030 D - 0xc0742140 [doneq0] 100055 D - 0xc074206c [scanner] 6 0 0 0 DL crypto_ 0xc2ee52c4 [crypto returns 3] 5 0 0 0 DL crypto_ 0xc2ee528c [crypto returns 2] 4 0 0 0 DL crypto_ 0xc2ee5254 [crypto returns 1] 3 0 0 0 DL crypto_ 0xc2ee521c [crypto returns 0] 2 0 0 0 DL crypto_ 0xc08bb07c [crypto] 12 0 0 0 LL (threaded) [geom] 100018 L *kernel 0xc2e77880 [g_event] 100019 D - 0xc08c3704 [g_up] 100020 D - 0xc08c3708 [g_down] 11 0 0 0 LL (threaded) [intr] 100006 I [swi5: fast taskq] 100009 I [swi6: task queue] 100010 I [swi6: Giant taskq] 100012 I [swi1: netisr 0] 100013 I [swi3: vm] 100014 L *Giant 0xc2e7a800 [swi4: clock (0)] 100015 I [swi4: clock (1)] 100016 I [swi4: clock (2)] 100017 I [swi4: clock (3)] 100031 I [intc0,28: bcm_dma0] 100032 I [intc0,29: bcm_dma0] 100033 I [intc0,32: bcm_dma0] 100034 I [intc0,33: bcm_dma0] 100035 I [intc0,34: bcm_dma0] 100036 I [intc0,35: bcm_dma0] 100037 I [intc0,1: mbox0] 100038 I [swi0: uart] 100039 I [intc0,62: spi0] 100040 I [intc0,61: iichb0++] 100041 I [intc0,70: +] 100042 I [intc0,17: +] 100048 I [intc0,2: vchiq0] 10 0 0 0 RL (threaded) [idle] 100002 Run CPU 0 [idle: cpu0] 100003 Run CPU 1 [idle: cpu1] 100004 Run CPU 2 [idle: cpu2] 100005 CanRun [idle: cpu3] 1 0 0 0 RL [kernel] 0 0 0 0 LLs (threaded) [kernel] 100000 L *kernel 0xc2e77880 [swapper] 100007 D - 0xc2ea3a00 [aiod_kick taskq] 100008 D - 0xc2ea3980 [kqueue_ctx taskq] 100011 D - 0xc2ea3800 [thread taskq] 100021 D - 0xc2ea3680 [crypto_0] 100022 D - 0xc2ea3680 [crypto_1] 100023 D - 0xc2ea3680 [crypto_2] 100024 D - 0xc2ea3680 [crypto_3] 100054 D - 0xc2ea3580 [CAM taskq] 100068 L *kernel 0xc2e77880 [softirq_0] 100069 L *kernel 0xc2e77880 [softirq_1] 100070 L *kernel 0xc2e77880 [softirq_2] 100071 D - 0xc2ea3200 [softirq_3]
'show alllocks' output: Process 13 (usb) thread 0xd87c63c0 (100046) exclusive sx USB config SX lock (USB config SX lock) r = 0 (0xd8b8c028) locked @ /e-drive/src.current/sys/dev/usb/usb_device.c:2884 Process 1 (kernel) thread 0xc4a6a000 (100001) exclusive sleep mutex kernel arena domain (kernel arena domain) r = 0 (0xc0b52000) locked @ /e-drive/src.current/sys/kern/subr_vmem.c:1180 Process 0 (kernel) thread 0xc08c3ac0 (100000) exclusive sleep mutex Giant (Giant) r = 0 (0xc0735140) locked @ /e-drive/src.current/sys/kern/kern_module.c:116
stack back trace for thread ID 100000 (which has giant lock) Tracing pid 0 tid 100000 td 0xc08c3ac0 cpu_switch() at cpu_switch+24 pc = 0xc0579a8c lr = 0xc028d544 (sched_switch+1672) sp = 0xc0b13a60 fp = 0xc0b13ab8 sched_switch() at sched_switch+1672 pc = 0xc028d544 lr = 0xc026a014 (mi_switch+572) sp = 0xc0b13ac0 fp = 0xc0b13ae8 r4 = 0x0000000a r5 = 0x00000000 r6 = 0xc08c3ac0 r7 = 0x0000f248 r8 = 0x00000000 r9 = 0x00000103 r10 = 0xc072bef8 mi_switch() at mi_switch+572 pc = 0xc026a014 lr = 0xc02bf8f4 (turnstile_wait+924) sp = 0xc0b13af0 fp = 0xc0b13b18 r4 = 0xc08c3ac0 r5 = 0xc2e77880 r6 = 0xc06035f4 r7 = 0x00000000 r8 = 0xc085d0d8 r9 = 0x00000000 r10 = 0xc4a6a000 turnstile_wait() at turnstile_wait+924 pc = 0xc02bf8f4 lr = 0xc0243398 (__mtx_lock_sleep+572) sp = 0xc0b13b20 fp = 0xc0b13b70 r4 = 0x00000000 r5 = 0xc0b52010 r6 = 0x00000000 r7 = 0xc0b13b48 r8 = 0xc0b52000 r9 = 0xc2e77880 r10 = 0xc4a6a000 __mtx_lock_sleep() at __mtx_lock_sleep+572 pc = 0xc0243398 lr = 0xc0243060 (__mtx_lock_flags+236) sp = 0xc0b13b78 fp = 0xc0b13ba0 r4 = 0x00000000 r5 = 0xc0603ea3 r6 = 0xc0b52010 r7 = 0x00000000 r8 = 0xc0b52000 r9 = 0x00000000 r10 = 0x0000049c __mtx_lock_flags() at __mtx_lock_flags+236 pc = 0xc0243060 lr = 0xc02c4464 (vmem_xalloc+364) sp = 0xc0b13ba8 fp = 0xc0b13c08 r4 = 0x00000000 r5 = 0x00002000 r6 = 0x00002001 r7 = 0x00001000 r8 = 0xc0b13c4c r9 = 0x00000001 r10 = 0xc0b52000 vmem_xalloc() at vmem_xalloc+364 pc = 0xc02c4464 lr = 0xc02c428c (vmem_alloc+180) sp = 0xc0b13c10 fp = 0xc0b13c40 r4 = 0x00003603 r5 = 0xc0b52000 r6 = 0x00001000 r7 = 0x00002101 r8 = 0xc0b13c4c r9 = 0x00000001 r10 = 0xc05256b8 vmem_alloc() at vmem_alloc+180 pc = 0xc02c428c lr = 0xc052bc48 (kmem_malloc_domain+72) sp = 0xc0b13c48 fp = 0xc0b13c68 r4 = 0xc0b52000 r5 = 0x00001000 r6 = 0x00000101 r7 = 0x00000000 r8 = 0x00001000 r10 = 0xc05256b8 kmem_malloc_domain() at kmem_malloc_domain+72 pc = 0xc052bc48 lr = 0xc05233cc (keg_alloc_slab+224) sp = 0xc0b13c70 fp = 0xc0b13ca8 r4 = 0x00000101 r5 = 0xc0b4a9a0 r6 = 0xd8bd3420 r7 = 0xc0b4a9b0 r8 = 0x00001000 r10 = 0xc05256b8 keg_alloc_slab() at keg_alloc_slab+224 pc = 0xc05233cc lr = 0xc0525f78 (keg_fetch_slab+324) sp = 0xc0b13cb0 fp = 0xc0b13d00 r4 = 0xc0b4a9a0 r5 = 0x00000000 r6 = 0x00000001 r7 = 0xc0b4d000 r8 = 0x00000000 r9 = 0xc2e25100 r10 = 0x00000000 keg_fetch_slab() at keg_fetch_slab+324 pc = 0xc0525f78 lr = 0xc0525768 (zone_fetch_slab+92) sp = 0xc0b13d08 fp = 0xc0b13d20 r4 = 0x00000001 r5 = 0xc0b4d000 r6 = 0xc0b4d000 r7 = 0xc0b4a9a0 r8 = 0xffffffff r9 = 0xc2e25100 r10 = 0x0000000e zone_fetch_slab() at zone_fetch_slab+92 pc = 0xc0525768 lr = 0xc0525818 (zone_import+76) sp = 0xc0b13d28 fp = 0xc0b13d60 r4 = 0x00000000 r5 = 0xc0b4d000 r6 = 0xffffffff r7 = 0xc052570c r8 = 0x00000000 r10 = 0x0000000e zone_import() at zone_import+76 pc = 0xc0525818 lr = 0xc0521950 (uma_zalloc_arg+1008) sp = 0xc0b13d68 fp = 0xc0b13da0 r4 = 0x00000000 r5 = 0xc072938c r6 = 0xc05257cc r7 = 0xc2e2510c r8 = 0xc0b4d000 r9 = 0xc2e25100 r10 = 0xffffffff uma_zalloc_arg() at uma_zalloc_arg+1008 pc = 0xc0521950 lr = 0xc05218f0 (uma_zalloc_arg+912) sp = 0xc0b13da8 fp = 0xc0b13de0 r4 = 0x00000000 r5 = 0xc07293bc r6 = 0xc2e10864 r7 = 0x00000000 r8 = 0xc2e10780 r9 = 0xc062d1c2 r10 = 0xffffffff uma_zalloc_arg() at uma_zalloc_arg+912 pc = 0xc05218f0 lr = 0xc023df04 (malloc+236) sp = 0xc0b13de8 fp = 0xc0b13e10 r4 = 0xc0707a70 r5 = 0xc2e10780 r6 = 0xc0707a70 r7 = 0x00000020 r8 = 0x00000000 r9 = 0x00000001 r10 = 0x00000002 malloc() at malloc+236 pc = 0xc023df04 lr = 0xc02a6790 (taskqgroup_adjust+2248) sp = 0xc0b13e18 fp = 0xc0b13e68 r4 = 0xc0707a70 r5 = 0xc2ea3b80 r6 = 0x00000003 r7 = 0xc02a71dc r8 = 0x00000000 r9 = 0xc05c79a2 r10 = 0xc06005af taskqgroup_adjust() at taskqgroup_adjust+2248 pc = 0xc02a6790 lr = 0xc01fa1d4 (mi_startup+396) sp = 0xc0b13e70 fp = 0xc0b13e90 r4 = 0xc08c3f9c r5 = 0x00000001 r6 = 0xc0732ac8 r7 = 0x00000000 r8 = 0xc08c3f98 r9 = 0xc2ea5e6c r10 = 0xc08c3ea0 mi_startup() at mi_startup+396 pc = 0xc01fa1d4 lr = 0xc0000244 (_start+324) sp = 0xc0b13e98 fp = 0x00000000 r4 = 0xc0000378 r5 = 0xc0914000 r6 = 0x01090e00 r7 = 0x00c52078 r8 = 0xc0a6a000 r9 = 0x3effb800 r10 = 0x00000040 _start() at _start+324 pc = 0xc0000244 lr = 0xc0000244 (_start+324) sp = 0xc0b13e98 fp = 0x00000000
back trace for thread 100001 (which has a locked at sys/kern/subr_vmem.c:1180 ) Tracing pid 1 tid 100001 td 0xc4a6a000 cpu_switch() at cpu_switch+24 pc = 0xc0579a8c lr = 0xc028d544 (sched_switch+1672) sp = 0xc2dbb858 fp = 0xc2dbb8b0 sched_switch() at sched_switch+1672 pc = 0xc028d544 lr = 0xc026a014 (mi_switch+572) sp = 0xc2dbb8b8 fp = 0xc2dbb8e0 r4 = 0x00000008 r5 = 0x00000000 r6 = 0xc4a6a000 r7 = 0x000014ff r8 = 0x00000000 r9 = 0x00000602 r10 = 0xc072bef8 mi_switch() at mi_switch+572 pc = 0xc026a014 lr = 0xc02664f8 (critical_exit_preempt+112) sp = 0xc2dbb8e8 fp = 0xc2dbb8f0 r4 = 0xc4a6a000 r5 = 0xc05fc082 r6 = 0xc4a6a000 r7 = 0x00000000 r8 = 0xc072bef8 r9 = 0xd89df000 r10 = 0xd89f5fff critical_exit_preempt() at critical_exit_preempt+112 pc = 0xc02664f8 lr = 0xc0560b1c (intr_irq_handler+152) sp = 0xc2dbb8f8 fp = 0xc2dbb910 r4 = 0xc2dbb918 r5 = 0xc08bba60 intr_irq_handler() at intr_irq_handler+152 pc = 0xc0560b1c lr = 0xc055eda4 (exception_exit) sp = 0xc2dbb918 fp = 0xc2dbb9b8 r4 = 0x00000133 r5 = 0xc0603ea3 r6 = 0xc0b52010 r7 = 0x00000004 r8 = 0xc0b52000 r10 = 0xd89f5fff exception_exit() at exception_exit pc = 0xc055eda4 lr = 0xc02c6620 (bt_alloc+36) sp = 0xc2dbb9a8 fp = 0xc2dbb9b8 r0 = 0x00000000 r1 = 0xc4a6a000 r2 = 0xc0603ea3 r3 = 0x00000133 r4 = 0x00000133 r5 = 0xc0603ea3 r6 = 0xc0b52010 r7 = 0x00000004 r8 = 0xc0b52000 r9 = 0xd89df000 r10 = 0xd89f5fff r12 = 0x00000000 $a.2() at $a.2+24 pc = 0xc0243a0c lr = 0xc02c6620 (bt_alloc+36) sp = 0xc2dbb9c0 fp = 0xc2dbb9c8 r4 = 0xc0b52000 r5 = 0xd89df000 r6 = 0xd8764d20 r7 = 0xd89dffff bt_alloc() at bt_alloc+36 pc = 0xc02c6620 lr = 0xc02c4a78 (vmem_xalloc+1920) sp = 0xc2dbb9d0 fp = 0xc2dbba30 r4 = 0x00001000 r5 = 0xd89df000 vmem_xalloc() at vmem_xalloc+1920 pc = 0xc02c4a78 lr = 0xc02c428c (vmem_alloc+180) sp = 0xc2dbba38 fp = 0xc2dbba68 r4 = 0x00003603 r5 = 0xc0b52000 r6 = 0x00001000 r7 = 0x00002101 r8 = 0xc2dbba74 r9 = 0x00000001 r10 = 0xc05256b8 vmem_alloc() at vmem_alloc+180 pc = 0xc02c428c lr = 0xc052bc48 (kmem_malloc_domain+72) sp = 0xc2dbba70 fp = 0xc2dbba90 r4 = 0xc0b52000 r5 = 0x00001000 r6 = 0x00000101 r7 = 0x00000000 r8 = 0x00001000 r10 = 0xc05256b8 kmem_malloc_domain() at kmem_malloc_domain+72 pc = 0xc052bc48 lr = 0xc05233cc (keg_alloc_slab+224) sp = 0xc2dbba98 fp = 0xc2dbbad0 r4 = 0x00000101 r5 = 0xc0b4a9a0 r6 = 0xd8bd3318 r7 = 0xc0b4a9b0 r8 = 0x00001000 r10 = 0xc05256b8 keg_alloc_slab() at keg_alloc_slab+224 pc = 0xc05233cc lr = 0xc0525f78 (keg_fetch_slab+324) sp = 0xc2dbbad8 fp = 0xc2dbbb28 r4 = 0xc0b4a9a0 r5 = 0x00000000 r6 = 0x00000001 r7 = 0xc0b4d000 r8 = 0x00000000 r9 = 0xd8bd32c0 r10 = 0x00000000 keg_fetch_slab() at keg_fetch_slab+324 pc = 0xc0525f78 lr = 0xc0525768 (zone_fetch_slab+92) sp = 0xc2dbbb30 fp = 0xc2dbbb48 r4 = 0x00000001 r5 = 0xc0b4d000 r6 = 0xc0b4d000 r7 = 0xc0b4a9a0 r8 = 0xffffffff r9 = 0xd8bd32c0 r10 = 0x0000000d zone_fetch_slab() at zone_fetch_slab+92 pc = 0xc0525768 lr = 0xc0525818 (zone_import+76) sp = 0xc2dbbb50 fp = 0xc2dbbb88 r4 = 0x0000000a r5 = 0xc0b4d000 r6 = 0xffffffff r7 = 0xc052570c r8 = 0xc0b4a9a0 r10 = 0x0000000d zone_import() at zone_import+76 pc = 0xc0525818 lr = 0xc0521950 (uma_zalloc_arg+1008) sp = 0xc2dbbb90 fp = 0xc2dbbbc8 r4 = 0x00000000 r5 = 0xc072937c r6 = 0xc05257cc r7 = 0xc2e7114c r8 = 0xc0b4d000 r9 = 0xc2e71140 r10 = 0xffffffff uma_zalloc_arg() at uma_zalloc_arg+1008 pc = 0xc0521950 lr = 0xc05218f0 (uma_zalloc_arg+912) sp = 0xc2dbbbd0 fp = 0xc2dbbc08 r4 = 0x00000000 r5 = 0xc07293bc r6 = 0xc2e126e4 r7 = 0x00000000 r8 = 0xc2e12600 r9 = 0xc062d1c2 r10 = 0xffffffff uma_zalloc_arg() at uma_zalloc_arg+912 pc = 0xc05218f0 lr = 0xc023df04 (malloc+236) sp = 0xc2dbbc10 fp = 0xc2dbbc38 r4 = 0x00000000 r5 = 0xc2e12600 r6 = 0xc0710294 r7 = 0x00000030 r8 = 0x00000000 r9 = 0x00000002 r10 = 0x00000002 malloc() at malloc+236 pc = 0xc023df04 lr = 0xc02b39c0 (sbuf_new+160) sp = 0xc2dbbc40 fp = 0xc2dbbc58 r4 = 0x00000000 r5 = 0xc4a61000 r6 = 0x00000001 r7 = 0x00000000 r8 = 0x00000000 r9 = 0x00000001 r10 = 0x00000000 sbuf_new() at sbuf_new+160 pc = 0xc02b39c0 lr = 0xc0329b24 (vfs_mountroot+68) sp = 0xc2dbbc60 fp = 0xc2dbbdc0 r4 = 0xc4a61000 r5 = 0xc4a61000 r6 = 0xc01fb10c r7 = 0x00000000 r8 = 0xc4a6a000 r9 = 0x00000000 vfs_mountroot() at vfs_mountroot+68 pc = 0xc0329b24 lr = 0xc01fb128 (start_init+28) sp = 0xc2dbbdc8 fp = 0xc2dbbe20 r4 = 0xc4a61000 r5 = 0xc4a61000 r6 = 0xc01fb10c r7 = 0x00000000 r8 = 0xc2dbbe48 r9 = 0x00000000 r10 = 0x00000000 start_init() at start_init+28 pc = 0xc01fb128 lr = 0xc0223064 (fork_exit+160) sp = 0xc2dbbe28 fp = 0xc2dbbe40 r4 = 0xc4a6a000 r5 = 0xc4a61000 r6 = 0xc01fb10c r7 = 0x00000000 r8 = 0xc2dbbe48 r9 = 0x00000000 r10 = 0x00000000 fork_exit() at fork_exit+160 pc = 0xc0223064 lr = 0xc055ed34 (swi_exit) sp = 0xc2dbbe48 fp = 0x00000000 r4 = 0xc01fb10c r5 = 0x00000000 r6 = 0x00000000 r7 = 0x00000000 r8 = 0x00000000 r10 = 0x00000000 swi_exit() at swi_exit pc = 0xc055ed34 lr = 0xc055ed34 (swi_exit) sp = 0xc2dbbe48 fp = 0x00000000 db> bt Tracing pid 1 tid 100001 td 0xc4a6a000 cpu_switch() at cpu_switch+24 pc = 0xc0579a8c lr = 0xc028d544 (sched_switch+1672) sp = 0xc2dbb858 fp = 0xc2dbb8b0 sched_switch() at sched_switch+1672 pc = 0xc028d544 lr = 0xc026a014 (mi_switch+572) sp = 0xc2dbb8b8 fp = 0xc2dbb8e0 r4 = 0x00000008 r5 = 0x00000000 r6 = 0xc4a6a000 r7 = 0x000014ff r8 = 0x00000000 r9 = 0x00000602 r10 = 0xc072bef8 mi_switch() at mi_switch+572 pc = 0xc026a014 lr = 0xc02664f8 (critical_exit_preempt+112) sp = 0xc2dbb8e8 fp = 0xc2dbb8f0 r4 = 0xc4a6a000 r5 = 0xc05fc082 r6 = 0xc4a6a000 r7 = 0x00000000 r8 = 0xc072bef8 r9 = 0xd89df000 r10 = 0xd89f5fff critical_exit_preempt() at critical_exit_preempt+112 pc = 0xc02664f8 lr = 0xc0560b1c (intr_irq_handler+152) sp = 0xc2dbb8f8 fp = 0xc2dbb910 r4 = 0xc2dbb918 r5 = 0xc08bba60 intr_irq_handler() at intr_irq_handler+152 pc = 0xc0560b1c lr = 0xc055eda4 (exception_exit) sp = 0xc2dbb918 fp = 0xc2dbb9b8 r4 = 0x00000133 r5 = 0xc0603ea3 r6 = 0xc0b52010 r7 = 0x00000004 r8 = 0xc0b52000 r10 = 0xd89f5fff exception_exit() at exception_exit pc = 0xc055eda4 lr = 0xc02c6620 (bt_alloc+36) sp = 0xc2dbb9a8 fp = 0xc2dbb9b8 r0 = 0x00000000 r1 = 0xc4a6a000 r2 = 0xc0603ea3 r3 = 0x00000133 r4 = 0x00000133 r5 = 0xc0603ea3 r6 = 0xc0b52010 r7 = 0x00000004 r8 = 0xc0b52000 r9 = 0xd89df000 r10 = 0xd89f5fff r12 = 0x00000000 $a.2() at $a.2+24 pc = 0xc0243a0c lr = 0xc02c6620 (bt_alloc+36) sp = 0xc2dbb9c0 fp = 0xc2dbb9c8 r4 = 0xc0b52000 r5 = 0xd89df000 r6 = 0xd8764d20 r7 = 0xd89dffff bt_alloc() at bt_alloc+36 pc = 0xc02c6620 lr = 0xc02c4a78 (vmem_xalloc+1920) sp = 0xc2dbb9d0 fp = 0xc2dbba30 r4 = 0x00001000 r5 = 0xd89df000 vmem_xalloc() at vmem_xalloc+1920 pc = 0xc02c4a78 lr = 0xc02c428c (vmem_alloc+180) sp = 0xc2dbba38 fp = 0xc2dbba68 r4 = 0x00003603 r5 = 0xc0b52000 r6 = 0x00001000 r7 = 0x00002101 r8 = 0xc2dbba74 r9 = 0x00000001 r10 = 0xc05256b8 vmem_alloc() at vmem_alloc+180 pc = 0xc02c428c lr = 0xc052bc48 (kmem_malloc_domain+72) sp = 0xc2dbba70 fp = 0xc2dbba90 r4 = 0xc0b52000 r5 = 0x00001000 r6 = 0x00000101 r7 = 0x00000000 r8 = 0x00001000 r10 = 0xc05256b8 kmem_malloc_domain() at kmem_malloc_domain+72 pc = 0xc052bc48 lr = 0xc05233cc (keg_alloc_slab+224) sp = 0xc2dbba98 fp = 0xc2dbbad0 r4 = 0x00000101 r5 = 0xc0b4a9a0 r6 = 0xd8bd3318 r7 = 0xc0b4a9b0 r8 = 0x00001000 r10 = 0xc05256b8 keg_alloc_slab() at keg_alloc_slab+224 pc = 0xc05233cc lr = 0xc0525f78 (keg_fetch_slab+324) sp = 0xc2dbbad8 fp = 0xc2dbbb28 r4 = 0xc0b4a9a0 r5 = 0x00000000 r6 = 0x00000001 r7 = 0xc0b4d000 r8 = 0x00000000 r9 = 0xd8bd32c0 r10 = 0x00000000 keg_fetch_slab() at keg_fetch_slab+324 pc = 0xc0525f78 lr = 0xc0525768 (zone_fetch_slab+92) sp = 0xc2dbbb30 fp = 0xc2dbbb48 r4 = 0x00000001 r5 = 0xc0b4d000 r6 = 0xc0b4d000 r7 = 0xc0b4a9a0 r8 = 0xffffffff r9 = 0xd8bd32c0 r10 = 0x0000000d zone_fetch_slab() at zone_fetch_slab+92 pc = 0xc0525768 lr = 0xc0525818 (zone_import+76) sp = 0xc2dbbb50 fp = 0xc2dbbb88 r4 = 0x0000000a r5 = 0xc0b4d000 r6 = 0xffffffff r7 = 0xc052570c r8 = 0xc0b4a9a0 r10 = 0x0000000d zone_import() at zone_import+76 pc = 0xc0525818 lr = 0xc0521950 (uma_zalloc_arg+1008) sp = 0xc2dbbb90 fp = 0xc2dbbbc8 r4 = 0x00000000 r5 = 0xc072937c r6 = 0xc05257cc r7 = 0xc2e7114c r8 = 0xc0b4d000 r9 = 0xc2e71140 r10 = 0xffffffff uma_zalloc_arg() at uma_zalloc_arg+1008 pc = 0xc0521950 lr = 0xc05218f0 (uma_zalloc_arg+912) sp = 0xc2dbbbd0 fp = 0xc2dbbc08 r4 = 0x00000000 r5 = 0xc07293bc r6 = 0xc2e126e4 r7 = 0x00000000 r8 = 0xc2e12600 r9 = 0xc062d1c2 r10 = 0xffffffff uma_zalloc_arg() at uma_zalloc_arg+912 pc = 0xc05218f0 lr = 0xc023df04 (malloc+236) sp = 0xc2dbbc10 fp = 0xc2dbbc38 r4 = 0x00000000 r5 = 0xc2e12600 r6 = 0xc0710294 r7 = 0x00000030 r8 = 0x00000000 r9 = 0x00000002 r10 = 0x00000002 malloc() at malloc+236 pc = 0xc023df04 lr = 0xc02b39c0 (sbuf_new+160) sp = 0xc2dbbc40 fp = 0xc2dbbc58 r4 = 0x00000000 r5 = 0xc4a61000 r6 = 0x00000001 r7 = 0x00000000 r8 = 0x00000000 r9 = 0x00000001 r10 = 0x00000000 sbuf_new() at sbuf_new+160 pc = 0xc02b39c0 lr = 0xc0329b24 (vfs_mountroot+68) sp = 0xc2dbbc60 fp = 0xc2dbbdc0 r4 = 0xc4a61000 r5 = 0xc4a61000 r6 = 0xc01fb10c r7 = 0x00000000 r8 = 0xc4a6a000 r9 = 0x00000000 vfs_mountroot() at vfs_mountroot+68 pc = 0xc0329b24 lr = 0xc01fb128 (start_init+28) sp = 0xc2dbbdc8 fp = 0xc2dbbe20 r4 = 0xc4a61000 r5 = 0xc4a61000 r6 = 0xc01fb10c r7 = 0x00000000 r8 = 0xc2dbbe48 r9 = 0x00000000 r10 = 0x00000000 start_init() at start_init+28 pc = 0xc01fb128 lr = 0xc0223064 (fork_exit+160) sp = 0xc2dbbe28 fp = 0xc2dbbe40 r4 = 0xc4a6a000 r5 = 0xc4a61000 r6 = 0xc01fb10c r7 = 0x00000000 r8 = 0xc2dbbe48 r9 = 0x00000000 r10 = 0x00000000 fork_exit() at fork_exit+160 pc = 0xc0223064 lr = 0xc055ed34 (swi_exit) sp = 0xc2dbbe48 fp = 0x00000000 r4 = 0xc01fb10c r5 = 0x00000000 r6 = 0x00000000 r7 = 0x00000000 r8 = 0x00000000 r10 = 0x00000000 swi_exit() at swi_exit pc = 0xc055ed34 lr = 0xc055ed34 (swi_exit) sp = 0xc2dbbe48 fp = 0x00000000
sorry if there's information overload here but I wanted to capture as much as I could get out of the debuger. My conclusion is that a kernel process is calling vmem_xalloc(), which had an exception take place that involved bt_alloc(), or maybe something else that's erroneously being flagged as bt_alloc(). The problem is that vmem_xalloc() does not call bt_alloc directly, but it does call vmem_clip() which returns 'void' [so this might be messing up the stack frame?]. vmem_clip() then calls bt_alloc() in a couple of places, and returns after doing its magic. The problem here is that it appears that the 'vmem_t' structure might either be incomplete, or somehow corrupted, which causes an exception (an actual interrupt?) of some kind to happen (page fault, or ?). I can't really tell from the debug info I've posted here (someone more familiar with this might recognize it). But here's a summary of what happened: a) vmem_xalloc locks the VM via VM_LOCK b) vmem_xalloc loops through structures looking for a page that's big enough c) vmem_xalloc (apparently) calls 'vmem_clip()' to (apparently) shorten it to the correct size d) "an exception happens" while the VM is still locked e) the exception handler allows other threads to run, waiting for who knows what f) another thread tries to lock the VM with the Giant lock already grabbed g) deadlock So it's important to fix whatever is causing this exception, as (I think) it has happened on the PPC platform as well, so it's not just me and looks like a 'ticking time bomb' waiting for a Murphy's Law to arm it. I'm sure it will affect everyone at some point, although it appears to be ok if you change the bootup timing in a way that doesn't deadlock the VM (by enabling verbose logging, for one). That's a bad workaround, but it keeps me able to work with it in the mean time. I'll do whatever additional analysis anyone wants done. Just let me know by posting to the bug.
some additional information: # addr2line -e /usr/obj/e-drive/src.current/arm.armv7/sys/RPI2/kernel.full 0xc0243a0c /e-drive/src.current/sys/kern/kern_mutex.c:1062 # addr2line -e /usr/obj/e-drive/src.current/arm.armv7/sys/RPI2/kernel.full 0xc02c4a78 /e-drive/src.current/sys/kern/subr_vmem.c:973 # addr2line -e /usr/obj/e-drive/src.current/arm.armv7/sys/RPI2/kernel.full 0xc02c6620 /e-drive/src.current/sys/kern/subr_vmem.c:308
from loader prompt, with 'verbose boot' off, did the following: loader> set hw.ncpu=1 result: boot success! (non-verbose) boot output looks like this: /boot/dtb/bcm2836-rpi-2-b.dtb size=0x4c49 Loaded DTB from file 'bcm2836-rpi-2-b.dtb'. Loading DTB overlays: 'spigen-rpi2.dtbo' /boot/dtb/overlays/spigen-rpi2.dtbo size=0x3c6 applying DTB overlay '/boot/dtb/overlays/spigen-rpi2.dtbo' Kernel entry at 0x1200100... Kernel args: (null) KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #2 r334665M: Tue Jun 5 21:17:17 PDT 2018 bobf@hack.SFT.local:/e-drive/obj.current/e-drive/src.current/arm.armv7/sys/RPI2 arm FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) WARNING: WITNESS option enabled, expect reduced performance. VT: init without driver. CPU: ARM Cortex-A7 r0p5 (ECO: 0x00000000) CPU Features: Multiprocessing, Thumb2, Security, Virtualization, Generic Timer, VMSAv7, PXN, LPAE, Coherent Walk Optional instructions: SDIV/UDIV, UMULL, SMULL, SIMD(ext) LoUU:2 LoC:3 LoUIS:2 Cache level 1: 32KB/64B 4-way data cache WB Read-Alloc Write-Alloc 32KB/32B 2-way instruction cache Read-Alloc Cache level 2: 512KB/64B 8-way unified cache WB Read-Alloc Write-Alloc real memory = 0 (0 MB) avail memory = 1008693248 (961 MB) random: unblocking device. random: entropy device external interface kbd0 at kbdmux0 ofwbus0: <Open Firmware Device Tree> simplebus0: <Flattened device tree simple bus> on ofwbus0 local_intc0: <BCM2836 Interrupt Controller> mem 0x40000000-0x400000ff on simplebus0 intc0: <BCM2835 Interrupt Controller> mem 0x7e00b200-0x7e00b3ff irq 25 on simplebus0 gpio0: <BCM2708/2835 GPIO controller> mem 0x7e200000-0x7e2000b3 irq 27,28,29,30 on simplebus0 gpio0: read-only pins: 46-53. gpio0: reserved pins: 48-53. gpiobus0: <OFW GPIO bus> on gpio0 generic_timer0: <ARMv7 Generic Timer> irq 1,2,3,4 on ofwbus0 Timecounter "ARM MPCore Timecounter" frequency 19200000 Hz quality 1000 Event timer "ARM MPCore Eventtimer" frequency 19200000 Hz quality 1000 bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007eff irq 9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24 on simplebus0 bcmwd0: <BCM2708/2835 Watchdog> mem 0x7e100000-0x7e100027 on simplebus0 bcm2835_clkman0: <BCM283x Clock Manager> mem 0x7e101000-0x7e102fff on simplebus0 bcmrng0: <Broadcom BCM2835 RNG> mem 0x7e104000-0x7e10400f on simplebus0 mbox0: <BCM2835 VideoCore Mailbox> mem 0x7e00b880-0x7e00b8bf irq 26 on simplebus0 gpioc0: <GPIO controller> on gpio0 uart0: <PrimeCell UART (PL011)> mem 0x7e201000-0x7e201fff irq 31 on simplebus0 uart0: console (115200,n,8,1) spi0: <BCM2708/2835 SPI controller> mem 0x7e204000-0x7e204fff irq 33 on simplebus0 spibus0: <OFW SPI bus> on spi0 spigen0: <SPI Generic IO> at cs 0 mode 0 on spibus0 spigen1: <SPI Generic IO> at cs 1 mode 0 on spibus0 spigen2: <SPI Generic IO> at cs 2 mode 0 on spibus0 iichb0: <BCM2708/2835 BSC controller> mem 0x7e205000-0x7e205fff irq 34 on simplebus0 pwm0: <BCM2708/2835 PWM controller> mem 0x7e20c000-0x7e20c027 on simplebus0 sdhci_bcm0: <Broadcom 2708 SDHCI controller> mem 0x7e300000-0x7e3000ff irq 41 on simplebus0 mmc0: <MMC/SD bus> on sdhci_bcm0 iichb1: <BCM2708/2835 BSC controller> mem 0x7e804000-0x7e804fff irq 44 on simplebus0 iichb2: <BCM2708/2835 BSC controller> mem 0x7e805000-0x7e805fff irq 45 on simplebus0 bcm283x_dwcotg0: <DWC OTG 2.0 integrated USB controller (bcm283x)> mem 0x7e980000-0x7e98ffff irq 50 on simplebus0 usbus0 on bcm283x_dwcotg0 vchiq0: <BCM2835 VCHIQ> mem 0x7e00b840-0x7e00b84e irq 52 on simplebus0 vchiq: local ver 8 (min 3), remote ver 8. pcm0: <VCHIQ audio> on vchiq0 fb0: <BCM2835 VT framebuffer driver> on simplebus0 fbd0 on fb0 VT: initialize with new VT driver "fb". fb0: 656x416(656x416@0,0) 24bpp fb0: fbswap: 0, pitch 1968, base rx3ef34000, screen_size 818688 cpulist0: <Open Firmware CPU Group> on ofwbus0 cpu0: <Open Firmware CPU> on cpulist0 bcm2835_cpufreq0: <CPU Frequency Control> on cpu0 cpu1: <Open Firmware CPU> on cpulist0 cpu2: <Open Firmware CPU> on cpulist0 cpu3: <Open Firmware CPU> on cpulist0 gpioled0: <GPIO LEDs> on ofwbus0 cryptosoft0: <software crypto> Timecounters tick every 1.000 msec usbus0: 480Mbps High Speed USB v2.0 iicbus0: <OFW I2C bus> on iichb0 iic0: <I2C generic I/O> on iicbus0 ugen0.1: <DWCOTG OTG Root HUB> at usbus0 uhub0: <DWCOTG OTG Root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0 mmcsd0: 4GB <SDHC SS04G 8.0 SN 032B0249 MFG 11/2014 by 3 SD> at mmc0 50.0MHz/4bit/65535-block iicbus1: <OFW I2C bus> on iichb1 iic1: <I2C generic I/O> on iicbus1 iicbus2: <OFW I2C bus> on iichb2 iic2: <I2C generic I/O> on iicbus2 bcm2835_cpufreq0: ARM 600MHz, Core 250MHz, SDRAM 400MHz, Turbo OFF WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/ufs/rootfs [rw]... Warning: no time-of-day clock registered, system time will not be set accurately uhub0: 1 port with 1 removable, self powered ugen0.2: <vendor 0x0424 product 0x9514> at usbus0 uhub1 on uhub0 uhub1: <vendor 0x0424 product 0x9514, class 9/0, rev 2.00/2.00, addr 2> on usbus0 uhub1: MTT enabled Setting hostuuid: 883f4aee-4e62-11e8-8d4c-b827eb35f155. Setting hostid: 0xb55ef262. No suitable dump device was found. Starting file system checks: /dev/ufs/rootfs: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/rootfs: clean, 226368 free (2072 frags, 28037 blocks, 0.2% fragmentation) uhub1: 5 ports with 4 removable, self powered Mounting local filesystems:ugen0.3: <vendor 0x0424 product 0xec00> at usbus0 smsc0 on uhub1 smsc0: <vendor 0x0424 product 0xec00, rev 2.00/2.00, addr 3> on usbus0 smsc0: chip 0xec00, rev. 0002 miibus0: <MII bus> on smsc0 ukphy0: <Generic IEEE 802.3u media interface> PHY 1 on miibus0 ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto ue0: <USB Ethernet> on smsc0 ue0: Ethernet address: b8:27:eb:35:f1:55 . Setting hostname: pi2c. /etc/rc: WARNING: $ipv6_enable is obsolete. Use $ipv6_activate_all_interfaces instead. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . /etc/rc: WARNING: $ipv6_enable is obsolete. Use $ipv6_activate_all_interfaces instead. /etc/rc: WARNING: $ipv6_enable is obsolete. Use $ipv6_activate_all_interfaces instead. lo0: link state changed to UP smsc0: chip 0xec00, rev. 0002 ue0: link state changed to DOWN ue0: link state changed to UP Starting Network: lo0 ue0. lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=80009<RXCSUM,VLAN_MTU,LINKSTATE> ether b8:27:eb:35:f1:55 inet6 fe80::ba27:ebff:fe35:f155%ue0 prefixlen 64 scopeid 0x2 inet6 2001:470:d:c9c:ba27:ebff:fe35:f155 prefixlen 64 autoconf media: Ethernet autoselect (100baseTX <full-duplex>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib Soft Float compatibility ldconfig path: Starting devd. Starting dhclient. DHCPREQUEST on ue0 to 255.255.255.255 port 67 DHCPACK from 192.168.122.1 bound to 192.168.122.32 -- renewal in 82839 seconds. add host 127.0.0.1: gateway lo0 fib 0: route already in table add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. installing gpioshutdown kern.gpioshutdown.led_gpio: -1 -> 35 kern.gpioshutdown.led_gpio_set: 1 -> 1 gpio0: Stray irq 35 disabled Clearing /tmp (X related). Updating motd:. Mounting late filesystems:. Starting powerd. Configuring vt: blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Fri Jun 8 19:21:30 UTC 2018 FreeBSD/arm (pi2c) (ttyu0) login:
Created attachment 194119 [details] unedited console output for VERBOSE_SYSINIT this was captured from serial console output using a different kernel with VERBOSE_SYSINIT enabled.
other notes: when booting the original kernel (not the 'VERBOSE_SYSINIT' one), and entering 'set hw.ncpu=4' from the loader prompt, it sometimes worked, and sometimes did not work. however I have not yet observed 'set hw.ncpu=2' not working.
attempted to reproduce in r335007 - could not reproduce the bug not sure if it's fixed or being worked around. In any case it's not happening in this revision.
this bug re-surfaced in r335576 . will attempt to narrow it down if possible.
does not appear to happen in r338208 nor the previous revision that I'd built a few weeks ago. Will continue to test for this.
^Triage: I'm sorry that this PR did not get addressed in a timely fashion. By now, the version that it was created against is long out of support. Please re-open if it is still a problem on a supported version.