Attachment 154224 Details for Bug 195458 – debugging session after freebsd-update and reboot

debugging session after freebsd-update and reboot

script.txt (text/plain), 47.15 KB, created by Glen Barber on 2015-03-11 20:27:30 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Glen Barber

Created: 2015-03-11 20:27:30 UTC

Size: 47.15 KB

patch

obsolete

>Script started on Wed Mar 11 16:18:32 2015
>command: cu -l /tmp/freebsdvm-pty -s 38400
>Connected
>
>root@release:~ # cat /var/run/dmesg.boot
>Copyright (c) 1992-2014 The FreeBSD Project.
>Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>        The Regents of the University of California. All rights reserved.
>FreeBSD is a registered trademark of The FreeBSD Foundation.
>FreeBSD 10.1-RELEASE-p6 #6 r279857M: Wed Mar 11 15:30:25 UTC 2015
>    root@release:/usr/obj/usr/src/sys/GENERIC amd64
>FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
>WARNING: WITNESS option enabled, expect reduced performance.
>CPU: Intel(R) Core(TM) i7-4800MQ CPU @ 2.70GHz (2695.47-MHz K8-class CPU)
>  Origin = "GenuineIntel"  Id = 0x306c3  Family = 0x6  Model = 0x3c  Stepping = 3
>  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
>  Features2=0x201<SSE3,SSSE3>
>  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
>  AMD Features2=0x1<LAHF>
>real memory  = 2147418112 (2047 MB)
>avail memory = 2046881792 (1952 MB)
>Event timer "LAPIC" quality 400
>ACPI APIC Table: <VBOX   VBOXAPIC>
>FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
>FreeBSD/SMP: 1 package(s) x 4 core(s)
> cpu0 (BSP): APIC ID:  0
> cpu1 (AP): APIC ID:  1
> cpu2 (AP): APIC ID:  2
> cpu3 (AP): APIC ID:  3
>ioapic0 <Version 1.1> irqs 0-23 on motherboard
>random: <Software, Yarrow> initialized
>kbd1 at kbdmux0
>acpi0: <VBOX VBOXXSDT> on motherboard
>acpi0: Power Button (fixed)
>acpi0: Sleep Button (fixed)
>cpu0: <ACPI CPU> on acpi0
>cpu1: <ACPI CPU> on acpi0
>cpu2: <ACPI CPU> on acpi0
>cpu3: <ACPI CPU> on acpi0
>attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
>Timecounter "i8254" frequency 1193182 Hz quality 0
>Event timer "i8254" frequency 1193182 Hz quality 100
>Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
>acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
>pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
>pci0: <ACPI PCI bus> on pcib0
>isab0: <PCI-ISA bridge> at device 1.0 on pci0
>isa0: <ISA bus> on isab0
>atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd000-0xd00f at device 1.1 on pci0
>ata0: <ATA channel> at channel 0 on atapci0
>ata1: <ATA channel> at channel 1 on atapci0
>vgapci0: <VGA-compatible display> mem 0xe0000000-0xe0ffffff irq 18 at device 2.0 on pci0
>vgapci0: Boot video device
>em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0xd010-0xd017 mem 0xf0000000-0xf001ffff irq 19 at device 3.0 on pci0
>em0: Ethernet address: 08:00:27:09:a4:5b
>pcm0: <Intel ICH (82801AA)> port 0xd100-0xd1ff,0xd200-0xd23f irq 21 at device 5.0 on pci0
>pcm0: <SigmaTel STAC9700/83/84 AC97 Codec>
>ohci0: <OHCI (generic) USB controller> mem 0xf0804000-0xf0804fff irq 22 at device 6.0 on pci0
>usbus0 on ohci0
>pci0: <bridge> at device 7.0 (no driver attached)
>uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
>uart0: console (38400,n,8,1)
>battery0: <ACPI Control Method Battery> on acpi0
>acpi_acad0: <AC Adapter> on acpi0
>atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
>atkbd0: <AT Keyboard> irq 1 on atkbdc0
>kbd0 at atkbd0
>atkbd0: [GIANT-LOCKED]
>psm0: <PS/2 Mouse> irq 12 on atkbdc0
>psm0: [GIANT-LOCKED]
>psm0: model IntelliMouse Explorer, device ID 4
>orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xe2000-0xeefff on isa0
>sc0: <System console> at flags 0x100 on isa0
>sc0: VGA <16 virtual consoles, flags=0x300>
>vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
>atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
>Event timer "RTC" frequency 32768 Hz quality 0
>ppc0: cannot reserve I/O port range
>Timecounters tick every 10.000 msec
>pcm0: measured ac97 link rate at 35622 Hz
>lock order reversal:
>lock order reversal:
>lock order reversal:
> 1st 0xffffffff817ae930 entropy harvest mutex (entropy harvest mutex) @ /usr/src/sys/dev/random/random_harvestq.c:198
> 2nd 0xfffff80002785838 uart_hwmtx (uart_hwmtx) @ /usr/src/sys/dev/uart/uart_cpu.h:94
>KDB: stack backtrace:
>db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe009371f130
>kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe009371f1e0
>witness_checkorder() at witness_checkorder+0xc04/frame 0xfffffe009371f260
>__mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x4e/frame 0xfffffe009371f2a0
>uart_cnputc() at uart_cnputc+0x3b/frame 0xfffffe009371f2c0
>cnputc() at cnputc+0x80/frame 0xfffffe009371f2f0
>cnputs() at cnputs+0x58/frame 0xfffffe009371f310
>putchar() at putchar+0x151/frame 0xfffffe009371f390
>kvprintf() at kvprintf+0xf6/frame 0xfffffe009371f4a0
>_vprintf() at _vprintf+0x8d/frame 0xfffffe009371f580
>printf() at printf+0x53/frame 0xfffffe009371f5e0
>witness_checkorder() at witness_checkorder+0x9e0/frame 0xfffffe009371f660
>__mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x4e/frame 0xfffffe009371f6a0
>sc_puts() at sc_puts+0xb0/frame 0xfffffe009371f6e0
>sc_cnputc() at sc_cnputc+0xe5/frame 0xfffffe009371f710
>cnputc() at cnputc+0x80/frame 0xfffffe009371f740
>cnputs() at cnputs+0x58/frame 0xfffffe009371f760
>putchar() at putchar+0x151/frame 0xfffffe009371f7e0
>kvprintf() at kvprintf+0xf6/frame 0xfffffe009371f8f0
>_vprintf() at _vprintf+0x8d/frame 0xfffffe009371f9d0
>printf() at printf+0x53/frame 0xfffffe009371fa30
>witness_checkorder() at witness_checkorder+0x9e0/frame 0xfffffe009371fab0
>__mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x4e/frame 0xfffffe009371faf0
>msleep_spin_sbt() at msleep_spin_sbt+0x5b/frame 0xfffffe009371fb70
>random_kthread() at random_kthread+0x78/frame 0xfffffe009371fbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe009371fbf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009371fbf0
>--- trap 0, rip = 0, rsp = 0xfffffe009371fcb0, rbp = 0 ---
> 1st 0xffffffff817ae930 entropy harvest mutex (entropy harvest mutex) @ /usr/src/sys/dev/random/random_harvestq.c:198
> 2nd 0xffffffff814a95a8 scrlock (scrlock) @ /usr/src/sys/dev/syscons/syscons.c:2694
>KDB: stack backtrace:
>db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe009371f530
>kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe009371f5e0
>witness_checkorder() at witness_checkorder+0xc04/frame 0xfffffe009371f660
>__mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x4e/frame 0xfffffe009371f6a0
>sc_puts() at sc_puts+0xb0/frame 0xfffffe009371f6e0
>sc_cnputc() at sc_cnputc+0xe5/frame 0xfffffe009371f710
>cnputc() at cnputc+0x80/frame 0xfffffe009371f740
>cnputs() at cnputs+0x58/frame 0xfffffe009371f760
>putchar() at putchar+0x151/frame 0xfffffe009371f7e0
>kvprintf() at kvprintf+0xf6/frame 0xfffffe009371f8f0
>_vprintf() at _vprintf+0x8d/frame 0xfffffe009371f9d0
>printf() at printf+0x53/frame 0xfffffe009371fa30
>witness_checkorder() at witness_checkorder+0x9e0/frame 0xfffffe009371fab0
>__mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x4e/frame 0xfffffe009371faf0
>msleep_spin_sbt() at msleep_spin_sbt+0x5b/frame 0xfffffe009371fb70
>random_kthread() at random_kthread+0x78/frame 0xfffffe009371fbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe009371fbf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009371fbf0
>--- trap 0, rip = 0, rsp = 0xfffffe009371fcb0, rbp = 0 ---
> 1st 0xffffffff817ae930 entropy harvest mutex (entropy harvest mutex) @ /usr/src/sys/dev/random/random_harvestq.c:198
> 2nd 0xffffffff8159afd8 sleepq chain (sleepq chain) @ /usr/src/sys/kern/subr_sleepqueue.c:237
>KDB: stack backtrace:
>db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe009371f980
>kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe009371fa30
>witness_checkorder() at witness_checkorder+0xc04/frame 0xfffffe009371fab0
>__mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x4e/frame 0xfffffe009371faf0
>msleep_spin_sbt() at msleep_spin_sbt+0x5b/frame 0xfffffe009371fb70
>random_kthread() at random_kthread+0x78/frame 0xfffffe009371fbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe009371fbf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009371fbf0
>--- trap 0, rip = 0, rsp = 0xfffffe009371fcb0, rbp = 0 ---
>usbus0: 12Mbps Full Speed USB v1.0
>ugen0.1: <Apple> at usbus0
>uhub0: <Apple OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
>ada0 at ata0 bus 0 scbus0 target 0 lun 0
>ada0: <VBOX HARDDISK 1.0> ATA-6 device
>ada0: Serial Number VB43127253-afd276f9
>ada0: 33.300MB/s transfers (UDMA2, PIO 65536bytes)
>ada0: 20480MB (41943040 512 byte sectors: 16H 63S/T 16383C)
>ada0: Previously was known as ad0
>cd0 at ata1 bus 0 scbus1 target 0 lun 0
>cd0: <VBOX CD-ROM 1.0> Removable CD-ROM SCSI-0 device 
>cd0: Serial Number VB2-01700376
>cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
>cd0: Attempt to query device size failed: NOT READY, Medium not present
>random: unblocking device.
>SMP: AP CPU #3 Launched!
>SMP: AP CPU #2 Launched!
>SMP: AP CPU #1 Launched!
>WARNING: WITNESS option enabled, expect reduced performance.
>Root mount waiting for: usbus0
>uhub0: 8 ports with 8 removable, self powered
>Trying to mount root from ufs:/dev/ada0p2 [rw]...
>root@release:~ # freebsd-update install
>Installing updates...lock order reversal:
> 1st 0xfffffe007b88caa8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3058
> 2nd 0xfffff80002fc2c00 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
>KDB: stack backtrace:
>db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe007b674420
>kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe007b6744d0
>witness_checkorder() at witness_checkorder+0xc04/frame 0xfffffe007b674550
>_sx_xlock() at _sx_xlock+0x47/frame 0xfffffe007b674590
>ufsdirhash_add() at ufsdirhash_add+0x4a/frame 0xfffffe007b6745d0
>ufs_direnter() at ufs_direnter+0x6a0/frame 0xfffffe007b674690
>ufs_mkdir() at ufs_mkdir+0x84d/frame 0xfffffe007b674880
>VOP_MKDIR_APV() at VOP_MKDIR_APV+0xc4/frame 0xfffffe007b6748b0
>kern_mkdirat() at kern_mkdirat+0x209/frame 0xfffffe007b674ae0
>amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe007b674bf0
>Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe007b674bf0
>--- syscall (136, FreeBSD ELF64, sys_mkdir), rip = 0x800b56bda, rsp = 0x7fffffffe938, rbp = 0x7fffffffeaf0 ---
>lock order reversal:
> 1st 0xfffff80029742068 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2137
> 2nd 0xfffffe007ba4d5e8 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:262
> 3rd 0xfffff80021514548 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2137
>KDB: stack backtrace:
>db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe007b650f40
>kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe007b650ff0
>witness_checkorder() at witness_checkorder+0xc04/frame 0xfffffe007b651070
>__lockmgr_args() at __lockmgr_args+0x73a/frame 0xfffffe007b6511a0
>ffs_lock() at ffs_lock+0x84/frame 0xfffffe007b6511f0
>VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xfffffe007b651220
>_vn_lock() at _vn_lock+0x43/frame 0xfffffe007b651280
>vget() at vget+0x5b/frame 0xfffffe007b6512c0
>vfs_hash_get() at vfs_hash_get+0xe1/frame 0xfffffe007b651310
>ffs_vgetf() at ffs_vgetf+0x40/frame 0xfffffe007b6513a0
>softdep_sync_buf() at softdep_sync_buf+0x929/frame 0xfffffe007b651480
>ffs_syncvnode() at ffs_syncvnode+0x226/frame 0xfffffe007b651500
>ffs_truncate() at ffs_truncate+0x715/frame 0xfffffe007b651660
>ufs_direnter() at ufs_direnter+0x7f9/frame 0xfffffe007b651720
>ufs_link() at ufs_link+0x157/frame 0xfffffe007b651870
>VOP_LINK_APV() at VOP_LINK_APV+0xe3/frame 0xfffffe007b6518a0
>kern_linkat() at kern_linkat+0x324/frame 0xfffffe007b651ac0
>sys_linkat() at sys_linkat+0x3b/frame 0xfffffe007b651ae0
>amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe007b651bf0
>Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe007b651bf0
>--- syscall (495, FreeBSD ELF64, sys_linkat), rip = 0x80088b08a, rsp = 0x7fffffffd8b8, rbp = 0x7fffffffea60 ---
>
>Completing this upgrade requires removing old shared object files.
>Please rebuild all installed 3rd party software (e.g., programs
>installed from the ports tree) and then run "/usr/sbin/freebsd-update install"
>again to finish installing updates.
>root@release:~ # shutdown -r now
>Shutdown NOW!
>shutdown: [pid 60157]
>                                                                               
>*** FINAL System shutdown message from root@release ***                      
>
>System going down IMMEDIATELY                                                  
>
>                                                                               
>
>System shutdown time has arrived
>Waiting (max 60 seconds) for system process `vnlru' to stop...done
>Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
>Waiting (max 60 seconds) for system process `syncer' to stop...
>Syncing disks, vnodes remaining...77 56 1 0 done
>All buffers synced.
>lock order reversal:
> 1st 0xfffff80002a567b8 ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:1223
> 2nd 0xfffff80002a572d8 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2137
>KDB: stack backtrace:
>db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00936d45e0
>kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe00936d4690
>witness_checkorder() at witness_checkorder+0xc04/frame 0xfffffe00936d4710
>__lockmgr_args() at __lockmgr_args+0x73a/frame 0xfffffe00936d4840
>vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe00936d4860
>VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xfffffe00936d4890
>_vn_lock() at _vn_lock+0x43/frame 0xfffffe00936d48f0
>vget() at vget+0x5b/frame 0xfffffe00936d4930
>devfs_allocv() at devfs_allocv+0xfd/frame 0xfffffe00936d4980
>devfs_root() at devfs_root+0x43/frame 0xfffffe00936d49b0
>dounmount() at dounmount+0x328/frame 0xfffffe00936d4a30
>vfs_unmountall() at vfs_unmountall+0x61/frame 0xfffffe00936d4a60
>kern_reboot() at kern_reboot+0x4b0/frame 0xfffffe00936d4ac0
>sys_reboot() at sys_reboot+0x5a/frame 0xfffffe00936d4ae0
>amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe00936d4bf0
>Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00936d4bf0
>--- syscall (55, FreeBSD ELF64, sys_reboot), rip = 0x40ecbc, rsp = 0x7fffffffe6d8, rbp = 0x7fffffffe7c0 ---
>KDB: enter: Break to debugger
>[ thread pid 11 tid 100005 ]
>Stopped at      kdb_alt_break_internal+0x17f:   movq    $0,kdb_why
>db> ps
>  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
>    9     0     0     0  DL      kpsusp   0xfffff80002a1c128 [syncer]
>    8     0     0     0  DL      kpsusp   0xfffff80002a1c5e8 [vnlru]
>    7     0     0     0  DL      (threaded)                  [bufdaemon]
>100055                   D       sdflush  0xfffff80002a83ae8 [/ worker]
>100046                   D       kpsusp   0xfffff80002a1caa8 [bufdaemon]
>    6     0     0     0  DL      pgzero   0xffffffff81749c8c [pagezero]
>    5     0     0     0  DL      psleep   0xffffffff817493c0 [vmdaemon]
>    4     0     0     0  DL      psleep   0xffffffff817d1b04 [pagedaemon]
>    3     0     0     0  DL      waiting_ 0xffffffff817c5878 [sctp_iterator]
>   15     0     0     0  DL      (threaded)                  [usb]
>100036                   D       -        0xfffffe0000979460 [usbus0]
>100035                   D       -        0xfffffe0000979408 [usbus0]
>100034                   D       -        0xfffffe00009793b0 [usbus0]
>100033                   D       -        0xfffffe0000979358 [usbus0]
>    2     0     0     0  DL      (threaded)                  [cam]
>100042                   D       -        0xffffffff8148b740 [scanner]
>100022                   D       -        0xffffffff8148b900 [doneq0]
>   14     0     0     0  DL      -        0xffffffff814a8a90 [rand_harvestq]
>   13     0     0     0  DL      (threaded)                  [geom]
>--More--        100015                   D       -        0xffffffff817ba0e0 [g_down]
>100014                   D       -        0xffffffff817ba0d8 [g_up]
>100013                   D       -        0xffffffff817ba0d0 [g_event]
>   12     0     0     0  RL      (threaded)                  [intr]
>100039                   I                                   [irq12: psm0]
>100038                   I                                   [irq1: atkbd0]
>100037                   I                                   [swi0: uart]
>100032                   I                                   [irq22: ohci0]
>100031                   I                                   [irq21: pcm0]
>100029                   I                                   [irq15: ata1]
>100028                   I                                   [irq14: ata0]
>100027                   I                                   [swi6: task queue]
>100020                   I                                   [swi5: fast taskq]
>100018                   I                                   [swi6: Giant taskq]
>100012                   I                                   [swi3: vm]
>100011                   I                                   [swi4: clock]
>100010                   I                                   [swi4: clock]
>100009                   I                                   [swi4: clock]
>100008                   Run     CPU 3                       [swi4: clock]
>100007                   I                                   [swi1: netisr 0]
>--More--           11     0     0     0  RL      (threaded)                  [idle]
>100006                   CanRun                              [idle: cpu3]
>100005                   Run     CPU 2                       [idle: cpu2]
>100004                   Run     CPU 1                       [idle: cpu1]
>100003                   CanRun                              [idle: cpu0]
>    1     0     1     0  RLs     CPU 0                       [init]
>   10     0     0     0  DL      audit_wo 0xffffffff817c90b8 [audit]
>    0     0     0     0  DLs     (threaded)                  [kernel]
>100041                   D       -        0xfffff80002559700 [CAM taskq]
>100030                   D       -        0xfffff800026fd700 [em0 taskq]
>100026                   D       -        0xfffff80002558e00 [kqueue taskq]
>100025                   D       -        0xfffff80002559300 [acpi_task_2]
>100024                   D       -        0xfffff80002559300 [acpi_task_1]
>100023                   D       -        0xfffff80002559300 [acpi_task_0]
>100021                   D       -        0xfffff80002559900 [ffs_trim taskq]
>100019                   D       -        0xfffff80002559e00 [thread taskq]
>100016                   D       -        0xfffff80002508500 [firmware taskq]
>100000                   D       swapin   0xffffffff817ba1d8 [swapper]
>db> show pcpu
>cpuid        = 2
>dynamic pcpu = 0xfffffe0112236a80
>curthread    = 0xfffff80002512490: pid 11 "idle: cpu2"
>curpcb       = 0xfffffe00936e3cc0
>fpcurthread  = none
>idlethread   = 0xfffff80002512490: tid 100005 "idle: cpu2"
>curpmap      = 0xffffffff817bac48
>tssp         = 0xffffffff817d2860
>commontssp   = 0xffffffff817d2860
>rsp0         = 0xfffffe00936e3cc0
>gs32p        = 0xffffffff817d42b8
>ldt          = 0xffffffff817d42f8
>tss          = 0xffffffff817d42e8
>spin locks held:
>db> show allpcpu
>Current CPU: 2
>
>cpuid        = 0
>dynamic pcpu = 0x65aa80
>curthread    = 0xfffff80002513490: pid 1 "init"
>curpcb       = 0xfffffe00936d4cc0
>fpcurthread  = none
>idlethread   = 0xfffff80002513000: tid 100003 "idle: cpu0"
>curpmap      = 0xfffff80002aa9678
>tssp         = 0xffffffff817d2790
>commontssp   = 0xffffffff817d2790
>rsp0         = 0xfffffe00936d4cc0
>gs32p        = 0xffffffff817d41e8
>ldt          = 0xffffffff817d4228
>tss          = 0xffffffff817d4218
>spin locks held:
>
>cpuid        = 1
>dynamic pcpu = 0xfffffe00fad6ea80
>curthread    = 0xfffff80002512920: pid 11 "idle: cpu1"
>--More--        curpcb       = 0xfffffe00936decc0
>fpcurthread  = none
>idlethread   = 0xfffff80002512920: tid 100004 "idle: cpu1"
>curpmap      = 0xffffffff817bac48
>tssp         = 0xffffffff817d27f8
>commontssp   = 0xffffffff817d27f8
>rsp0         = 0xfffffe00936decc0
>gs32p        = 0xffffffff817d4250
>ldt          = 0xffffffff817d4290
>tss          = 0xffffffff817d4280
>spin locks held:
>
>cpuid        = 2
>dynamic pcpu = 0xfffffe0112236a80
>curthread    = 0xfffff80002512490: pid 11 "idle: cpu2"
>curpcb       = 0xfffffe00936e3cc0
>fpcurthread  = none
>idlethread   = 0xfffff80002512490: tid 100005 "idle: cpu2"
>curpmap      = 0xffffffff817bac48
>tssp         = 0xffffffff817d2860
>--More--        commontssp   = 0xffffffff817d2860
>rsp0         = 0xfffffe00936e3cc0
>gs32p        = 0xffffffff817d42b8
>ldt          = 0xffffffff817d42f8
>tss          = 0xffffffff817d42e8
>spin locks held:
>
>cpuid        = 3
>dynamic pcpu = 0xfffffe011223ea80
>curthread    = 0xfffff80002511490: pid 12 "swi4: clock"
>curpcb       = 0xfffffe00936f2cc0
>fpcurthread  = none
>idlethread   = 0xfffff80002512000: tid 100006 "idle: cpu3"
>curpmap      = 0xffffffff817bac48
>tssp         = 0xffffffff817d28c8
>commontssp   = 0xffffffff817d28c8
>rsp0         = 0xfffffe00936f2cc0
>gs32p        = 0xffffffff817d4320
>ldt          = 0xffffffff817d4360
>tss          = 0xffffffff817d4350
>--More--        spin locks held:
>
>db> show locks
>db> show alllocks
>Process 1 (init) thread 0xfffff80002513490 (100002)
>exclusive sleep mutex struct mount mtx (struct mount mtx) r = 0 (0xfffff80002a45b58) locked @ /usr/src/sys/kern/vfs_subr.c:4653
>exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff817bb0b8) locked @ /usr/src/sys/kern/kern_shutdown.c:195
>db> show lockedvnods
>Locked vnodes
>db> alltrace
>
>Tracing command syncer pid 9 tid 100048 td 0xfffff80002a22000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe007b61a9d0
>mi_switch() at mi_switch+0xe1/frame 0xfffffe007b61aa10
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe007b61aa40
>_sleep() at _sleep+0x2cd/frame 0xfffffe007b61aae0
>kproc_suspend_check() at kproc_suspend_check+0x7c/frame 0xfffffe007b61ab10
>sched_sync() at sched_sync+0xd2/frame 0xfffffe007b61abb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe007b61abf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007b61abf0
>--- trap 0, rip = 0, rsp = 0xfffffe007b61acb0, rbp = 0 ---
>
>Tracing command vnlru pid 8 tid 100047 td 0xfffff80002a22490
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe007b6159e0
>mi_switch() at mi_switch+0xe1/frame 0xfffffe007b615a20
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe007b615a50
>_sleep() at _sleep+0x2cd/frame 0xfffffe007b615af0
>kproc_suspend_check() at kproc_suspend_check+0x7c/frame 0xfffffe007b615b20
>vnlru_proc() at vnlru_proc+0x50/frame 0xfffffe007b615bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe007b615bf0
>--More--        fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007b615bf0
>--- trap 0, rip = 0, rsp = 0xfffffe007b615cb0, rbp = 0 ---
>
>Tracing command bufdaemon pid 7 tid 100055 td 0xfffff80002ac4000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe007b63da70
>mi_switch() at mi_switch+0xe1/frame 0xfffffe007b63dab0
>sleepq_timedwait() at sleepq_timedwait+0x3a/frame 0xfffffe007b63dae0
>_sleep() at _sleep+0x2aa/frame 0xfffffe007b63db80
>softdep_flush() at softdep_flush+0x132/frame 0xfffffe007b63dbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe007b63dbf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007b63dbf0
>--- trap 0, rip = 0, rsp = 0xfffffe007b63dcb0, rbp = 0 ---
>
>Tracing command bufdaemon pid 7 tid 100046 td 0xfffff80002a22920
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe007b610a40
>mi_switch() at mi_switch+0xe1/frame 0xfffffe007b610a80
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe007b610ab0
>_sleep() at _sleep+0x2cd/frame 0xfffffe007b610b50
>kproc_suspend_check() at kproc_suspend_check+0x7c/frame 0xfffffe007b610b80
>buf_daemon() at buf_daemon+0x9a/frame 0xfffffe007b610bb0
>--More--        fork_exit() at fork_exit+0x71/frame 0xfffffe007b610bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007b610bf0
>--- trap 0, rip = 0, rsp = 0xfffffe007b610cb0, rbp = 0 ---
>
>Tracing command pagezero pid 6 tid 100045 td 0xfffff80002786000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937ffa70
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937ffab0
>sleepq_timedwait() at sleepq_timedwait+0x3a/frame 0xfffffe00937ffae0
>_sleep() at _sleep+0x2aa/frame 0xfffffe00937ffb80
>vm_pagezero() at vm_pagezero+0x88/frame 0xfffffe00937ffbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00937ffbf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937ffbf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937ffcb0, rbp = 0 ---
>
>Tracing command vmdaemon pid 5 tid 100044 td 0xfffff80002786490
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937faa30
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937faa70
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00937faaa0
>_sleep() at _sleep+0x2cd/frame 0xfffffe00937fab40
>vm_daemon() at vm_daemon+0x7e/frame 0xfffffe00937fabb0
>--More--        fork_exit() at fork_exit+0x71/frame 0xfffffe00937fabf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937fabf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937facb0, rbp = 0 ---
>
>Tracing command pagedaemon pid 4 tid 100043 td 0xfffff80002786920
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937f5a30
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937f5a70
>sleepq_timedwait() at sleepq_timedwait+0x3a/frame 0xfffffe00937f5aa0
>_sleep() at _sleep+0x2aa/frame 0xfffffe00937f5b40
>vm_pageout() at vm_pageout+0x229/frame 0xfffffe00937f5bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00937f5bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937f5bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937f5cb0, rbp = 0 ---
>
>Tracing command sctp_iterator pid 3 tid 100040 td 0xfffff80002787920
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937e6a80
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937e6ac0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00937e6af0
>_sleep() at _sleep+0x2cd/frame 0xfffffe00937e6b90
>sctp_iterator_thread() at sctp_iterator_thread+0x59/frame 0xfffffe00937e6bb0
>--More--        fork_exit() at fork_exit+0x71/frame 0xfffffe00937e6bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937e6bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937e6cb0, rbp = 0 ---
>
>Tracing command usb pid 15 tid 100036 td 0xfffff80002740000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937cdaa0
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937cdae0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00937cdb10
>_cv_wait() at _cv_wait+0x187/frame 0xfffffe00937cdb80
>usb_process() at usb_process+0xd4/frame 0xfffffe00937cdbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00937cdbf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937cdbf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937cdcb0, rbp = 0 ---
>
>Tracing command usb pid 15 tid 100035 td 0xfffff80002740490
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937c8aa0
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937c8ae0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00937c8b10
>_cv_wait() at _cv_wait+0x187/frame 0xfffffe00937c8b80
>usb_process() at usb_process+0xd4/frame 0xfffffe00937c8bb0
>--More--        fork_exit() at fork_exit+0x71/frame 0xfffffe00937c8bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937c8bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937c8cb0, rbp = 0 ---
>
>Tracing command usb pid 15 tid 100034 td 0xfffff80002740920
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937c3aa0
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937c3ae0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00937c3b10
>_cv_wait() at _cv_wait+0x187/frame 0xfffffe00937c3b80
>usb_process() at usb_process+0xd4/frame 0xfffffe00937c3bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00937c3bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937c3bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937c3cb0, rbp = 0 ---
>
>Tracing command usb pid 15 tid 100033 td 0xfffff80002741000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937beaa0
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937beae0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00937beb10
>_cv_wait() at _cv_wait+0x187/frame 0xfffffe00937beb80
>usb_process() at usb_process+0xd4/frame 0xfffffe00937bebb0
>--More--        fork_exit() at fork_exit+0x71/frame 0xfffffe00937bebf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937bebf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937becb0, rbp = 0 ---
>
>Tracing command cam pid 2 tid 100042 td 0xfffff80002787000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937f0a40
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937f0a80
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00937f0ab0
>_sleep() at _sleep+0x2cd/frame 0xfffffe00937f0b50
>xpt_scanner_thread() at xpt_scanner_thread+0x5c/frame 0xfffffe00937f0bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00937f0bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937f0bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937f0cb0, rbp = 0 ---
>
>Tracing command cam pid 2 tid 100022 td 0xfffff800026ac920
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe0093739a50
>mi_switch() at mi_switch+0xe1/frame 0xfffffe0093739a90
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe0093739ac0
>_sleep() at _sleep+0x2cd/frame 0xfffffe0093739b60
>xpt_done_td() at xpt_done_td+0xa5/frame 0xfffffe0093739bb0
>--More--        fork_exit() at fork_exit+0x71/frame 0xfffffe0093739bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0093739bf0
>--- trap 0, rip = 0, rsp = 0xfffffe0093739cb0, rbp = 0 ---
>
>Tracing command rand_harvestq pid 14 tid 100017 td 0xfffff8000252d490
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe009371fa80
>mi_switch() at mi_switch+0xe1/frame 0xfffffe009371fac0
>sleepq_timedwait() at sleepq_timedwait+0x3a/frame 0xfffffe009371faf0
>msleep_spin_sbt() at msleep_spin_sbt+0x19e/frame 0xfffffe009371fb70
>random_kthread() at random_kthread+0x78/frame 0xfffffe009371fbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe009371fbf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009371fbf0
>--- trap 0, rip = 0, rsp = 0xfffffe009371fcb0, rbp = 0 ---
>
>Tracing command geom pid 13 tid 100015 td 0xfffff8000252e000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe0093715a60
>mi_switch() at mi_switch+0xe1/frame 0xfffffe0093715aa0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe0093715ad0
>_sleep() at _sleep+0x2cd/frame 0xfffffe0093715b70
>g_io_schedule_down() at g_io_schedule_down+0x3c/frame 0xfffffe0093715ba0
>--More--        g_down_procbody() at g_down_procbody+0x6d/frame 0xfffffe0093715bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe0093715bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0093715bf0
>--- trap 0, rip = 0, rsp = 0xfffffe0093715cb0, rbp = 0 ---
>
>Tracing command geom pid 13 tid 100014 td 0xfffff8000252e490
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe0093710a60
>mi_switch() at mi_switch+0xe1/frame 0xfffffe0093710aa0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe0093710ad0
>_sleep() at _sleep+0x2cd/frame 0xfffffe0093710b70
>g_io_schedule_up() at g_io_schedule_up+0xc3/frame 0xfffffe0093710ba0
>g_up_procbody() at g_up_procbody+0x6d/frame 0xfffffe0093710bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe0093710bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0093710bf0
>--- trap 0, rip = 0, rsp = 0xfffffe0093710cb0, rbp = 0 ---
>
>Tracing command geom pid 13 tid 100013 td 0xfffff8000252e920
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe009370ba70
>mi_switch() at mi_switch+0xe1/frame 0xfffffe009370bab0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe009370bae0
>--More--        _sleep() at _sleep+0x2cd/frame 0xfffffe009370bb80
>g_run_events() at g_run_events+0x35a/frame 0xfffffe009370bbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe009370bbf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009370bbf0
>--- trap 0, rip = 0, rsp = 0xfffffe009370bcb0, rbp = 0 ---
>
>Tracing command intr pid 12 tid 100039 td 0xfffff80002788000
>fork_trampoline() at fork_trampoline
>
>Tracing command intr pid 12 tid 100038 td 0xfffff80002788490
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937d7b30
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937d7b70
>ithread_loop() at ithread_loop+0x1a5/frame 0xfffffe00937d7bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00937d7bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937d7bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937d7cb0, rbp = 0 ---
>
>Tracing command intr pid 12 tid 100037 td 0xfffff80002788920
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937d2b30
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937d2b70
>--More--        ithread_loop() at ithread_loop+0x1a5/frame 0xfffffe00937d2bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00937d2bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937d2bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937d2cb0, rbp = 0 ---
>
>Tracing command intr pid 12 tid 100032 td 0xfffff80002741490
>fork_trampoline() at fork_trampoline
>
>Tracing command intr pid 12 tid 100031 td 0xfffff80002741920
>fork_trampoline() at fork_trampoline
>
>Tracing command intr pid 12 tid 100029 td 0xfffff80002742490
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937a2b30
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937a2b70
>ithread_loop() at ithread_loop+0x1a5/frame 0xfffffe00937a2bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00937a2bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937a2bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937a2cb0, rbp = 0 ---
>
>Tracing command intr pid 12 tid 100028 td 0xfffff80002742920
>--More--        sched_switch() at sched_switch+0x2a5/frame 0xfffffe009377ab30
>mi_switch() at mi_switch+0xe1/frame 0xfffffe009377ab70
>ithread_loop() at ithread_loop+0x1a5/frame 0xfffffe009377abb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe009377abf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009377abf0
>--- trap 0, rip = 0, rsp = 0xfffffe009377acb0, rbp = 0 ---
>
>Tracing command intr pid 12 tid 100027 td 0xfffff800026ab000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe0093752b30
>mi_switch() at mi_switch+0xe1/frame 0xfffffe0093752b70
>ithread_loop() at ithread_loop+0x1a5/frame 0xfffffe0093752bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe0093752bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0093752bf0
>--- trap 0, rip = 0, rsp = 0xfffffe0093752cb0, rbp = 0 ---
>
>Tracing command intr pid 12 tid 100020 td 0xfffff800026ad490
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe009372fb30
>mi_switch() at mi_switch+0xe1/frame 0xfffffe009372fb70
>ithread_loop() at ithread_loop+0x1a5/frame 0xfffffe009372fbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe009372fbf0
>--More--        fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009372fbf0
>--- trap 0, rip = 0, rsp = 0xfffffe009372fcb0, rbp = 0 ---
>
>Tracing command intr pid 12 tid 100018 td 0xfffff8000252d000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe0093725b30
>mi_switch() at mi_switch+0xe1/frame 0xfffffe0093725b70
>ithread_loop() at ithread_loop+0x1a5/frame 0xfffffe0093725bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe0093725bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0093725bf0
>--- trap 0, rip = 0, rsp = 0xfffffe0093725cb0, rbp = 0 ---
>
>Tracing command intr pid 12 tid 100012 td 0xfffff8000252f000
>fork_trampoline() at fork_trampoline
>
>Tracing command intr pid 12 tid 100011 td 0xfffff8000252f490
>fork_trampoline() at fork_trampoline
>
>Tracing command intr pid 12 tid 100010 td 0xfffff8000252f920
>fork_trampoline() at fork_trampoline
>
>--More--        Tracing command intr pid 12 tid 100009 td 0xfffff80002511000
>fork_trampoline() at fork_trampoline
>
>Tracing command intr pid 12 tid 100008 td 0xfffff80002511490
>cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00936c7cf0
>ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe00936c7d10
>trap() at trap+0x3a/frame 0xfffffe00936c7f20
>nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00936c7f20
>--- trap 0x13, rip = 0xffffffff808f379d, rsp = 0xfffffe00936c7fe0, rbp = 0xfffffe00936f2a00 ---
>__mtx_lock_sleep() at __mtx_lock_sleep+0x20d/frame 0xfffffe00936f2a00
>__mtx_lock_flags() at __mtx_lock_flags+0x85/frame 0xfffffe00936f2a40
>softclock_call_cc() at softclock_call_cc+0x102/frame 0xfffffe00936f2b20
>softclock() at softclock+0x44/frame 0xfffffe00936f2b40
>intr_event_execute_handlers() at intr_event_execute_handlers+0x83/frame 0xfffffe00936f2b70
>ithread_loop() at ithread_loop+0x96/frame 0xfffffe00936f2bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00936f2bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00936f2bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00936f2cb0, rbp = 0 ---
>
>Tracing command intr pid 12 tid 100007 td 0xfffff80002511920
>--More--        sched_switch() at sched_switch+0x2a5/frame 0xfffffe00936edb30
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00936edb70
>ithread_loop() at ithread_loop+0x1a5/frame 0xfffffe00936edbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00936edbf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00936edbf0
>--- trap 0, rip = 0, rsp = 0xfffffe00936edcb0, rbp = 0 ---
>
>Tracing command idle pid 11 tid 100006 td 0xfffff80002512000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00936e8aa0
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00936e8ae0
>critical_exit() at critical_exit+0x7e/frame 0xfffffe00936e8b00
>sched_idletd() at sched_idletd+0x409/frame 0xfffffe00936e8bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00936e8bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00936e8bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00936e8cb0, rbp = 0 ---
>
>Tracing command idle pid 11 tid 100005 td 0xfffff80002512490
>kdb_alt_break_internal() at kdb_alt_break_internal+0x17f/frame 0xfffffe00936e3880
>kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe00936e3890
>uart_intr_rxready() at uart_intr_rxready+0x99/frame 0xfffffe00936e38c0
>--More--        uart_intr() at uart_intr+0x111/frame 0xfffffe00936e3900
>intr_event_handle() at intr_event_handle+0x9b/frame 0xfffffe00936e3950
>intr_execute_handlers() at intr_execute_handlers+0x48/frame 0xfffffe00936e3980
>lapic_handle_intr() at lapic_handle_intr+0x3f/frame 0xfffffe00936e39a0
>Xapic_isr1() at Xapic_isr1+0xa4/frame 0xfffffe00936e39a0
>--- interrupt, rip = 0xffffffff80cc6bf6, rsp = 0xfffffe00936e3a60, rbp = 0xfffffe00936e3a70 ---
>acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00936e3a70
>acpi_cpu_idle() at acpi_cpu_idle+0x13a/frame 0xfffffe00936e3ac0
>cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe00936e3ae0
>cpu_idle() at cpu_idle+0x90/frame 0xfffffe00936e3b00
>sched_idletd() at sched_idletd+0x409/frame 0xfffffe00936e3bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00936e3bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00936e3bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00936e3cb0, rbp = 0 ---
>
>Tracing command idle pid 11 tid 100004 td 0xfffff80002512920
>cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe007c1f7cf0
>ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe007c1f7d10
>trap() at trap+0x3a/frame 0xfffffe007c1f7f20
>nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe007c1f7f20
>--More--        --- trap 0x13, rip = 0xffffffff80cc6bf6, rsp = 0xfffffe007c1f7fe0, rbp = 0xfffffe00936dea70 ---
>acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00936dea70
>acpi_cpu_idle() at acpi_cpu_idle+0x13a/frame 0xfffffe00936deac0
>cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe00936deae0
>cpu_idle() at cpu_idle+0x90/frame 0xfffffe00936deb00
>sched_idletd() at sched_idletd+0x409/frame 0xfffffe00936debb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00936debf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00936debf0
>--- trap 0, rip = 0, rsp = 0xfffffe00936decb0, rbp = 0 ---
>
>Tracing command idle pid 11 tid 100003 td 0xfffff80002513000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00936d9aa0
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00936d9ae0
>critical_exit() at critical_exit+0x7e/frame 0xfffffe00936d9b00
>sched_idletd() at sched_idletd+0x409/frame 0xfffffe00936d9bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00936d9bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00936d9bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00936d9cb0, rbp = 0 ---
>
>Tracing command init pid 1 tid 100002 td 0xfffff80002513490
>--More--        cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff8174e130
>ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffffff8174e150
>trap() at trap+0x3a/frame 0xffffffff8174e360
>nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff8174e360
>--- trap 0x13, rip = 0xffffffff809521b5, rsp = 0xffffffff8174e420, rbp = 0xfffffe00936d47b0 ---
>witness_checkorder() at witness_checkorder+0x45/frame 0xfffffe00936d47b0
>__mtx_lock_flags() at __mtx_lock_flags+0x48/frame 0xfffffe00936d47f0
>__mnt_vnode_next_all() at __mnt_vnode_next_all+0xf2/frame 0xfffffe00936d4820
>ffs_sync() at ffs_sync+0x229/frame 0xfffffe00936d48e0
>vfs_write_suspend() at vfs_write_suspend+0x17a/frame 0xfffffe00936d4920
>vfs_write_suspend_umnt() at vfs_write_suspend_umnt+0x1c/frame 0xfffffe00936d4950
>ffs_unmount() at ffs_unmount+0x58/frame 0xfffffe00936d49b0
>dounmount() at dounmount+0x407/frame 0xfffffe00936d4a30
>vfs_unmountall() at vfs_unmountall+0x61/frame 0xfffffe00936d4a60
>kern_reboot() at kern_reboot+0x4b0/frame 0xfffffe00936d4ac0
>sys_reboot() at sys_reboot+0x5a/frame 0xfffffe00936d4ae0
>amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe00936d4bf0
>Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00936d4bf0
>--- syscall (55, FreeBSD ELF64, sys_reboot), rip = 0x40ecbc, rsp = 0x7fffffffe6d8, rbp = 0x7fffffffe7c0 ---
>
>--More--        Tracing command audit pid 10 tid 100001 td 0xfffff80002513920
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00936cfa70
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00936cfab0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00936cfae0
>_cv_wait() at _cv_wait+0x187/frame 0xfffffe00936cfb50
>audit_worker() at audit_worker+0x73/frame 0xfffffe00936cfbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00936cfbf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00936cfbf0
>--- trap 0, rip = 0, rsp = 0xfffffe00936cfcb0, rbp = 0 ---
>
>Tracing command kernel pid 0 tid 100041 td 0xfffff80002787490
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937eba70
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937ebab0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00937ebae0
>_sleep() at _sleep+0x2cd/frame 0xfffffe00937ebb80
>taskqueue_thread_loop() at taskqueue_thread_loop+0xb8/frame 0xfffffe00937ebbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00937ebbf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937ebbf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937ebcb0, rbp = 0 ---
>
>--More--        Tracing command kernel pid 0 tid 100030 td 0xfffff80002742000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe00937a7a90
>mi_switch() at mi_switch+0xe1/frame 0xfffffe00937a7ad0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00937a7b00
>msleep_spin_sbt() at msleep_spin_sbt+0x1ad/frame 0xfffffe00937a7b80
>taskqueue_thread_loop() at taskqueue_thread_loop+0x7d/frame 0xfffffe00937a7bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe00937a7bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00937a7bf0
>--- trap 0, rip = 0, rsp = 0xfffffe00937a7cb0, rbp = 0 ---
>
>Tracing command kernel pid 0 tid 100026 td 0xfffff800026ab490
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe009374da70
>mi_switch() at mi_switch+0xe1/frame 0xfffffe009374dab0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe009374dae0
>_sleep() at _sleep+0x2cd/frame 0xfffffe009374db80
>taskqueue_thread_loop() at taskqueue_thread_loop+0xb8/frame 0xfffffe009374dbb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe009374dbf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009374dbf0
>--- trap 0, rip = 0, rsp = 0xfffffe009374dcb0, rbp = 0 ---
>
>--More--        Tracing command kernel pid 0 tid 100025 td 0xfffff800026ab920
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe0093748a90
>mi_switch() at mi_switch+0xe1/frame 0xfffffe0093748ad0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe0093748b00
>msleep_spin_sbt() at msleep_spin_sbt+0x1ad/frame 0xfffffe0093748b80
>taskqueue_thread_loop() at taskqueue_thread_loop+0x7d/frame 0xfffffe0093748bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe0093748bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0093748bf0
>--- trap 0, rip = 0, rsp = 0xfffffe0093748cb0, rbp = 0 ---
>
>Tracing command kernel pid 0 tid 100024 td 0xfffff800026ac000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe0093743a90
>mi_switch() at mi_switch+0xe1/frame 0xfffffe0093743ad0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe0093743b00
>msleep_spin_sbt() at msleep_spin_sbt+0x1ad/frame 0xfffffe0093743b80
>taskqueue_thread_loop() at taskqueue_thread_loop+0x7d/frame 0xfffffe0093743bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe0093743bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0093743bf0
>--- trap 0, rip = 0, rsp = 0xfffffe0093743cb0, rbp = 0 ---
>
>--More--        Tracing command kernel pid 0 tid 100023 td 0xfffff800026ac490
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe009373ea90
>mi_switch() at mi_switch+0xe1/frame 0xfffffe009373ead0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe009373eb00
>msleep_spin_sbt() at msleep_spin_sbt+0x1ad/frame 0xfffffe009373eb80
>taskqueue_thread_loop() at taskqueue_thread_loop+0x7d/frame 0xfffffe009373ebb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe009373ebf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009373ebf0
>--- trap 0, rip = 0, rsp = 0xfffffe009373ecb0, rbp = 0 ---
>
>Tracing command kernel pid 0 tid 100021 td 0xfffff800026ad000
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe0093734a70
>mi_switch() at mi_switch+0xe1/frame 0xfffffe0093734ab0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe0093734ae0
>_sleep() at _sleep+0x2cd/frame 0xfffffe0093734b80
>taskqueue_thread_loop() at taskqueue_thread_loop+0xb8/frame 0xfffffe0093734bb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe0093734bf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0093734bf0
>--- trap 0, rip = 0, rsp = 0xfffffe0093734cb0, rbp = 0 ---
>
>--More--        Tracing command kernel pid 0 tid 100019 td 0xfffff800026ad920
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe009372aa70
>mi_switch() at mi_switch+0xe1/frame 0xfffffe009372aab0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe009372aae0
>_sleep() at _sleep+0x2cd/frame 0xfffffe009372ab80
>taskqueue_thread_loop() at taskqueue_thread_loop+0xb8/frame 0xfffffe009372abb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe009372abf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009372abf0
>--- trap 0, rip = 0, rsp = 0xfffffe009372acb0, rbp = 0 ---
>
>Tracing command kernel pid 0 tid 100016 td 0xfffff8000252d920
>sched_switch() at sched_switch+0x2a5/frame 0xfffffe009371aa70
>mi_switch() at mi_switch+0xe1/frame 0xfffffe009371aab0
>sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe009371aae0
>_sleep() at _sleep+0x2cd/frame 0xfffffe009371ab80
>taskqueue_thread_loop() at taskqueue_thread_loop+0xb8/frame 0xfffffe009371abb0
>fork_exit() at fork_exit+0x71/frame 0xfffffe009371abf0
>fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009371abf0
>--- trap 0, rip = 0, rsp = 0xfffffe009371acb0, rbp = 0 ---
>
>--More--        Tracing command kernel pid 0 tid 100000 td 0xffffffff817ba6a0
>sched_switch() at sched_switch+0x2a5/frame 0xffffffff81ae3b50
>mi_switch() at mi_switch+0xe1/frame 0xffffffff81ae3b90
>sleepq_timedwait() at sleepq_timedwait+0x3a/frame 0xffffffff81ae3bc0
>_sleep() at _sleep+0x2aa/frame 0xffffffff81ae3c60
>swapper() at swapper+0x208/frame 0xffffffff81ae3cb0
>btext() at btext+0x2c
>db> thread 100002
>[ thread pid 1 tid 100002 ]
>cpustop_handler+0x28:   movl    $0x1,%rax
>db> where
>Tracing pid 1 tid 100002 td 0xfffff80002513490
>cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff8174e130
>ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffffff8174e150
>trap() at trap+0x3a/frame 0xffffffff8174e360
>nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff8174e360
>--- trap 0x13, rip = 0xffffffff809521b5, rsp = 0xffffffff8174e420, rbp = 0xfffffe00936d47b0 ---
>witness_checkorder() at witness_checkorder+0x45/frame 0xfffffe00936d47b0
>__mtx_lock_flags() at __mtx_lock_flags+0x48/frame 0xfffffe00936d47f0
>__mnt_vnode_next_all() at __mnt_vnode_next_all+0xf2/frame 0xfffffe00936d4820
>ffs_sync() at ffs_sync+0x229/frame 0xfffffe00936d48e0
>vfs_write_suspend() at vfs_write_suspend+0x17a/frame 0xfffffe00936d4920
>vfs_write_suspend_umnt() at vfs_write_suspend_umnt+0x1c/frame 0xfffffe00936d4950
>ffs_unmount() at ffs_unmount+0x58/frame 0xfffffe00936d49b0
>dounmount() at dounmount+0x407/frame 0xfffffe00936d4a30
>vfs_unmountall() at vfs_unmountall+0x61/frame 0xfffffe00936d4a60
>kern_reboot() at kern_reboot+0x4b0/frame 0xfffffe00936d4ac0
>sys_reboot() at sys_reboot+0x5a/frame 0xfffffe00936d4ae0
>amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe00936d4bf0
>Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00936d4bf0
>--- syscall (55, FreeBSD ELF64, sys_reboot), rip = 0x40ecbc, rsp = 0x7fffffffe6d8, rbp = 0x7fffffffe7c0 ---
>--More--
>db> ~
>[EOT]
>
>Script done on Wed Mar 11 16:22:36 2015

Actions: View

Attachments on bug 195458: 149945 | 154206 | 154207 | 154208 | 154211 | 154224