Bug 187527 - [panic] sparc64 11.0-CURRENT #1 r263096: Panic String: vm_page_alloc: page 0xfffff8003e961f10 has unexpected queue 1
Summary: [panic] sparc64 11.0-CURRENT #1 r263096: Panic String: vm_page_alloc: page 0x...
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-03-13 11:10 UTC by Anton Shterenlikht
Modified: 2015-12-06 15:55 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Anton Shterenlikht 2014-03-13 11:10:01 UTC
with textdump setup I got these files:

# cat info.last
Dump header from device /dev/ada1a
  Architecture: sparc64
  Architecture Version: 1
  Dump Length: 66560B (0 MB)
  Blocksize: 512
  Dumptime: Thu Mar 13 10:33:45 2014
  Hostname: mech-anton240.men.bris.ac.uk
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 11.0-CURRENT #1 r263096: Thu Mar 13 09:42:34 GMT 2014
    root@mech-anton240.men.bris.ac.uk:/usr/obj/usr/src/sys/QOF
  Panic String: vm_page_alloc: page 0xfffff8003e961f10 has unexpected queue 1
  Dump Parity: 873070248
  Bounds: 0
  Dump Status: good
#

# cat panic.txt
vm_page_alloc: page 0xfffff8003e961f10 has unexpected queue 1
#

# cat version.txt
FreeBSD 11.0-CURRENT #1 r263096: Thu Mar 13 09:42:34 GMT 2014
    root@mech-anton240.men.bris.ac.uk:/usr/obj/usr/src/sys/QOF
#

**************************************
config.txt
**************************************

options	CONFIG_AUTOGENERATED
ident	QOF
machine	sparc64
cpu	SUN4U
makeoptions	MODULES_OVERRIDE=
makeoptions	DEBUG=-gdwarf-2
options	USB_DEBUG
options	KBD_INSTALL_CDEV
options	TEXTDUMP_VERBOSE
options	TEXTDUMP_PREFERRED
options	DEBUG_VFS_LOCKS
options	DEBUG_LOCKS
options	MALLOC_DEBUG_MAXZONES=8
options	WITNESS_SKIPSPIN
options	WITNESS
options	INVARIANT_SUPPORT
options	INVARIANTS
options	DEADLKRES
options	GDB
options	DDB
options	KDB_TRACE
options	KDB
options	INCLUDE_CONFIG_FILE
options	MAC
options	CAPABILITIES
options	CAPABILITY_MODE
options	AUDIT
options	HWPMC_HOOKS
options	PRINTF_BUFR_SIZE=128
options	_KPOSIX_PRIORITY_SCHEDULING
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD7
options	COMPAT_FREEBSD6
options	COMPAT_FREEBSD5
options	GEOM_LABEL
options	GEOM_PART_GPT
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSD
options	NFSCL
options	MD_ROOT
options	QUOTA
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SOFTUPDATES
options	FFS
options	SCTP
options	INET6
options	INET
options	PREEMPTION
options	SCHED_ULE
options	NEW_PCIB
options	SUNKBD_EMULATE_ATKBD
options	GEOM_PART_VTOC8
options	GEOM_PART_BSD
device	mem
device	uart_ns8250
device	uart_sab82532
device	uart_z8530
device	ebus
device	isa
device	pci
device	sbus
device	central
device	fhc
device	ahci
device	ata
device	scbus
device	da
device	cd
device	pass
device	atkbdc
device	atkbd
device	sc
device	machfb
device	auxio
device	eeprom
device	mk48txx
device	rtc
device	mc146818
device	epic
device	sbbc
device	puc
device	scc
device	uart
device	ppbus
device	ppc
device	lpt
device	miibus
device	bge
device	loop
device	random
device	ether
device	vlan
device	tun
device	md
device	gif
device	faith
device	firmware
device	uhci
device	ohci
device	ehci
device	usb
device	ukbd
device	umass
device	ums
device	sound
device	snd_t4dwave

****************************************
msgbuf.txt
****************************************

GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #1 r263096: Thu Mar 13 09:42:34 GMT 2014
    root@mech-anton240.men.bris.ac.uk:/usr/obj/usr/src/sys/QOF sparc64
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
real memory  = 2147483648 (2048 MB)
avail memory = 2078556160 (1982 MB)
cpu0: Sun Microsystems UltraSparc-IIIi Processor (1503.00 MHz CPU)
random: <Software, Yarrow> initialized
nexus0: <Open Firmware Nexus device>
pcib0: <Sun Host-PCI bridge> mem 0x4000f600000-0x4000f60afff,0x4000f410000-0x4000f41701f,0x7fe00000000-0x7fe000000ff,0x4000f780000-0x4000f78ffff irq 1970,1968,1969,1972,1953 on nexus0
pcib0: Tomatillo, version 4, IGN 0x1e, bus A, PCI mode, 33MHz
pcib0: DVMA map: 0xc0000000 to 0xdfffffff 65536 entries
pci0: <OFW PCI bus> on pcib0
pcib1: <Sun Host-PCI bridge> mem 0x4000ff00000-0x4000ff0afff,0x4000fc10000-0x4000fc1701f,0x7f600000000-0x7f6000000ff,0x4000ff80000-0x4000ff8ffff irq 2035,2032,2033,2036,2019 on nexus0
pcib1: Tomatillo, version 4, IGN 0x1f, bus B, PCI mode, 66MHz
pcib1: DVMA map: 0xc0000000 to 0xdfffffff 65536 entries
pci1: <OFW PCI bus> on pcib1
nexus0: <memory-controller> mem 0x40000000000-0x40000000007 type memory-controller (no driver attached)
isab0: <PCI-ISA bridge> at device 7.0 on pci0
pcib0: could not route pin 1 for device 7.0
isa0: could not map ISA interrupt 1 for node 0xf0097238: parallel
isa0: <ISA bus> on isab0
pci0: <old, non-VGA display device> at device 6.0 (no driver attached)
pcm0: <Acer Labs M5451> port 0x900-0x9ff mem 0x100000-0x100fff at device 8.0 on pci0
pcm0: <Analog Devices AD1881A AC97 Codec>
pcm0: [GIANT-LOCKED]
ohci0: <AcerLabs M5237 (Aladdin-V) USB controller> mem 0x1000000-0x1000fff at device 10.0 on pci0
usbus0 on ohci0
ohci1: <AcerLabs M5237 (Aladdin-V) USB controller> mem 0x2000000-0x2000fff at device 11.0 on pci0
usbus1 on ohci1
atapci0: <AcerLabs M5229 UDMA100 controller> port 0xa00-0xa07,0xa18-0xa1b,0xa10-0xa17,0xa08-0xa0b,0xa20-0xa2f at device 13.0 on pci0
atapci0: using PIO transfers above 137GB as workaround for 48bit DMA access bug, expect reduced performance
ata2: <ATA channel> at channel 0 on atapci0
ata3: <ATA channel> at channel 1 on atapci0
machfb0: <ATI Rage XL> port 0xb00-0xbff mem 0x3000000-0x3ffffff,0x102000-0x102fff at device 2.0 on pci0
machfb0: console
machfb0: 16 MB aperture at 0xfddcc000 not swapped
machfb0: 8188 KB SGRAM 114.992 MHz, maximum RAMDAC clock 230 MHz, DSP
machfb0: resolution 1280x1024 at 8 bpp
jbusppm0: <JBus power management> mem 0x4000f000000-0x4000f000007,0x4000f410050-0x4000f41005f on nexus0
bge0: <Broadcom BCM5703 A2, ASIC rev. 0x001002> mem 0x200000-0x20ffff at device 2.0 on pci1
bge0: CHIP ID 0x00001002; ASIC REV 0x01; CHIP REV 0x10; PCI on PCI-X 66 MHz; 64bit
miibus0: <MII bus> on bge0
brgphy0: <BCM5703 1000BASE-T media interface> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Ethernet address: 00:03:ba:d9:a0:07
nexus0: <i2c> mem 0x4000fc64000-0x4000fc6400f type i2c (no driver attached)
syscons0: <System console> on nexus0
syscons0: Unknown <16 virtual consoles, flags=0x300>
rtc0: <Real-Time Clock> at port 0x70-0x71 on isa0
uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 44 on isa0
uart1: <16550 or compatible> at port 0x2e8-0x2ef irq 44 on isa0
ppc0: <ECP parallel printer port> at port 0x378-0x37f,0-0x4ff drq 1 on isa0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/1 bytes threshold
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Polled port
Timecounter "tick" frequency 1503000000 Hz quality 1000
Timecounter "stick" frequency 12000000 Hz quality 1000
Event timer "tick" frequency 1503000000 Hz quality 1000
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
ugen0.1: <AcerLabs> at usbus0
uhub0: <AcerLabs OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <AcerLabs> at usbus1
uhub1: <AcerLabs OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
ada0 at ata2 bus 0 scbus0 target 0 lun 0
cd0 at ata3 bus 0 scbus1 target 0 lun 0
cd0: <TSSTcorp CDW/DVD TS-H492C SI00> Removable CD-ROM SCSI-0 device 
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
ada0: <ST3120026A 8.01> ATA-6 device
ada0: Serial Number 4JT0KVXQ
ada0: 100.000MB/s transfers (UDMA5, PIO 8192bytes)
ada0: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad0
ada1 at ata2 bus 0 scbus0 target 1 lun 0
ada1: <ST380011A 3.04> ATA-6 device
ada1: Serial Number 3JVA6XG5
ada1: 100.000MB/s transfers (UDMA5, PIO 8192bytes)
ada1: 76319MB (156301488 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad1
GEOM_PART: integrity check failed (ada0, VTOC8)
GEOM_PART: integrity check failed (diskid/DISK-4JT0KVXQ, VTOC8)
random: unblocking device.
WARNING: WITNESS option enabled, expect reduced performance.
Root mount waiting for: usbus1
ugen1.2: <vendor 0x0430> at usbus1
ukbd0: <vendor 0x0430 product 0x0005, class 0/0, rev 1.10/2.00, addr 2> on usbus1
kbd0 at ukbd0
Root mount waiting for: usbus1
ugen1.3: <vendor 0x0430> at usbus1
ums0: <vendor 0x0430 product 0x0100, class 0/0, rev 2.00/1.07, addr 3> on usbus1
ums0: 3 buttons and [XY] coordinates ID=0
Trying to mount root from ufs:/dev/ada0 [rw]...
<118>Setting hostuuid: b65fa63e-a9f9-11e3-9948-0003bad9a007.
<118>Setting hostid: 0xf5f7c330.
<118>Starting ddb.
<118>Entropy harvesting: interrupts ethernet point_to_point swi.
<118>Starting file system checks:
<118>/dev/ada0: FILE SYSTEM CLEAN; SKIPPING CHECKS
<118>/dev/ada0: clean, 26456900 free (32652 frags, 3303031 blocks, 0.1% fragmentation)
GEOM_PART: integrity check failed (ada0, VTOC8)
GEOM_PART: integrity check failed (diskid/DISK-4JT0KVXQ, VTOC8)
<118>Mounting local file systems:.
<118>Writing entropy file:.
<118>Setting hostname: mech-anton240.men.bris.ac.uk.
<118>Starting Network: lo0 bge0.
<118>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
<118>	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
<118>	inet6 ::1 prefixlen 128 
<118>	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
<118>	inet 127.0.0.1 netmask 0xff000000 
<118>	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
<118>bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
<118>	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
<118>	ether 00:03:ba:d9:a0:07
<118>	inet 137.222.187.240 netmask 0xffffff00 broadcast 137.222.187.255 
<118>	inet6 fe80::203:baff:fed9:a007%bge0 prefixlen 64 scopeid 0x1 
<118>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<118>	media: Ethernet autoselect (none)
<118>	status: no carrier
<118>Starting devd.
<118>Configuring keyboard: keyrate.
<118>Starting ums0 moused.
<118>Starting pflogd: 
<118>add net default: gateway 137.222.187.250
<118>add net fe80::: gateway ::1
<118>add net ff02::: gateway ::1
<118>add net ::ffff:0.0.0.0: gateway ::1
<118>add net ::0.0.0.0: gateway ::1
<118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
<118>Creating and/or trimming log files.
<118>Starting syslogd.
<118>No core dumps found.
<118>Starting casperd.
<118>Clearing /tmp (X related).
<118>Starting lpd.
<118>Updating motd:.
<118>Mounting late file systems:.
<118>Starting ntpd.
<118>Starting powerd.
<118>powerd: no cpufreq(4) support -- aborting: No such file or directory
<118>/etc/rc: WARNING: failed to start powerd
<118>Configuring syscons: keyrate blanktime.
<118>Performing sanity check on sshd configuration.
<118>Starting sshd.
<118>Starting sendmail_submit.
<118>Starting sendmail_msp_queue.
<118>Starting cron.
<118>Starting background file system checks in 60 seconds.
<118>
<118>Thu Mar 13 09:48:17 GMT 2014
<118>Mar 13 09:48:17 mech-anton240 getty[796]: open /dev/ttyu2: No such file or directory
<118>Mar 13 09:48:33 mech-anton240 login: ROOT LOGIN (root) ON ttyv0
lock order reversal:
 1st 0xfffff800018dc2d8 ufs (ufs) @ /usr/src/sys/kern/vfs_lookup.c:518
 2nd 0xc149a9c8 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:262
 3rd 0xfffff800018f8a28 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2101
KDB: stack backtrace:
_witness_debugger() at _witness_debugger+0x38
witness_checkorder() at witness_checkorder+0xe80
__lockmgr_args() at __lockmgr_args+0x9e8
ffs_lock() at ffs_lock+0xbc
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x104
_vn_lock() at _vn_lock+0xd0
vget() at vget+0xac
vfs_hash_get() at vfs_hash_get+0xdc
ffs_vgetf() at ffs_vgetf+0x20
softdep_sync_buf() at softdep_sync_buf+0xd30
ffs_syncvnode() at ffs_syncvnode+0x294
ffs_truncate() at ffs_truncate+0x6f4
ufs_direnter() at ufs_direnter+0xa00
ufs_mkdir() at ufs_mkdir+0x958
VOP_MKDIR_APV() at VOP_MKDIR_APV+0x128
kern_mkdirat() at kern_mkdirat+0x20c
kern_mkdir() at kern_mkdir+0x14
sys_mkdir() at sys_mkdir+0x10
syscall() at syscall+0x304
-- syscall (136, FreeBSD ELF64, sys_mkdir) %o7=0x403de7ec --
userland() at 0x40476548
user trace: trap %o7=0x403de7ec
pc 0x40476548, sp 0x7fdffffd7b1
pc 0x403dea14, sp 0x7fdffffdd01
pc 0x100dc8, sp 0x7fdffffddc1
pc 0x100a24, sp 0x7fdffffde91
pc 0x402038f4, sp 0x7fdffffdf51
done
lock order reversal:
 1st 0xc14d2858 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3081
 2nd 0xfffff80001152c00 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
KDB: stack backtrace:
_witness_debugger() at _witness_debugger+0x38
witness_checkorder() at witness_checkorder+0xe80
_sx_xlock() at _sx_xlock+0xb8
ufsdirhash_acquire() at ufsdirhash_acquire+0x50
ufsdirhash_add() at ufsdirhash_add+0x4
ufs_direnter() at ufs_direnter+0x7f0
ufs_mkdir() at ufs_mkdir+0x958
VOP_MKDIR_APV() at VOP_MKDIR_APV+0x128
kern_mkdirat() at kern_mkdirat+0x20c
kern_mkdir() at kern_mkdir+0x14
sys_mkdir() at sys_mkdir+0x10
syscall() at syscall+0x304
-- syscall (136, FreeBSD ELF64, sys_mkdir) %o7=0x10b2bc --
userland() at 0x406ae548
user trace: trap %o7=0x10b2bc
pc 0x406ae548, sp 0x7fdffffd8f1
pc 0x10b0e8, sp 0x7fdffffda31
pc 0x10b0e8, sp 0x7fdffffdb71
pc 0x10b0e8, sp 0x7fdffffdcb1
pc 0x10b5ec, sp 0x7fdffffddf1
pc 0x1085d0, sp 0x7fdffffded1
pc 0x1025a4, sp 0x7fdffffdfc1
pc 0x402118f4, sp 0x7fdffffe081
done
<118>Mar 13 10:04:42 mech-anton240 login: ROOT LOGIN (root) ON ttyv1
panic: vm_page_alloc: page 0xfffff8003e961f10 has unexpected queue 1
KDB: stack backtrace:
vpanic() at vpanic+0xfc
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done
KDB: enter: panic

****************************************************
ddb.txt
****************************************************

db:0:kdb.enter.panic>  run lockinfo
db:1:lockinfo> show locks
exclusive sleep mutex vm page free queue (vm page free queue) r = 0 (0xc0ac7300) locked @ /usr/src/sys/vm/vm_page.c:1484
exclusive rw vm object (vm object) r = 0 (0xfffff8000bb13e00) locked @ /usr/src/sys/vm/vm_fault.c:299
shared sx vm map (user) (vm map (user)) r = 0 (0xfffff80001087778) locked @ /usr/src/sys/vm/vm_map.c:3843
db:1:locks>  show alllocks
Process 55018 (tr) thread 0xfffff80001e1adb0 (100095)
exclusive sleep mutex vm page free queue (vm page free queue) r = 0 (0xc0ac7300) locked @ /usr/src/sys/vm/vm_page.c:1484
exclusive rw vm object (vm object) r = 0 (0xfffff8000bb13e00) locked @ /usr/src/sys/vm/vm_fault.c:299
shared sx vm map (user) (vm map (user)) r = 0 (0xfffff80001087778) locked @ /usr/src/sys/vm/vm_map.c:3843
db:1:alllocks>  show lockedvnods
Locked vnodes
db:0:kdb.enter.panic>  show pcpu
cpuid        = 0
dynamic pcpu = 0xfffff7ff40076980
curthread    = 0xfffff80001e1adb0: pid 55018 "tr"
curpcb       = 0xc127d980
fpcurthread  = none
idlethread   = 0xfffff8000107bb60: tid 100003 "idle"
spin locks held:
db:0:kdb.enter.panic>  bt
Tracing pid 55018 tid 100095 td 0xfffff80001e1adb0
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done
db:0:kdb.enter.panic>  ps
  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
55020 55016 54936     0  S+      piperd   0xfffff80001502268 tr
55019 55016 54936     0  S+      piperd   0xfffff80001502c08 sed
55018 55016 54936     0  RL+     CPU 0                       tr
55016 54937 54936     0  S+      wait     0xfffff80001e192c0 sh
54937 54936 54936     0  S+      piperd   0xfffff800015029a0 sh
54936 53662 54936     0  S+      wait     0xfffff80001e18e10 sh
53662 53661 53661     0  S+      select   0xfffff8000172a240 make
53661 53653 53661     0  S+      wait     0xfffff8021bbb3770 sh
53653 53614 53614     0  S+      select   0xfffff80001729e40 make
53614 53613 53614     0  S+      wait     0xfffff800018b72c0 sh
53613 53612 53612     0  S+      select   0xfffff8000172a4c0 make
53612 53611 53612     0  S+      wait     0xfffff80001e18000 sh
53611 53610   819     0  S+      select   0xfffff80001729b40 make
53610   977   819     0  S+      wait     0xfffff8021bbb32c0 sh
38770 38748 38770  1001  S+      select   0xfffff8000172a3c0 ssh
38748   788 38748  1001  S+      pause    0xfffff8000154eeb8 tcsh
37571 37484 37571  1001  S+      select   0xfffff80001729f40 ssh
37484   787 37484  1001  S+      pause    0xfffff80001802eb8 tcsh
25973 10162 25973  1001  S+      select   0xfffff8000172a340 ssh
10162   786 10162  1001  S+      pause    0xfffff800018b8eb8 tcsh
 3559   785  3559     0  S+      ttyin    0xfffff800010f98a8 csh
  977   960   819     0  S+      wait     0xfffff800018524b0 make
  960   955   819     0  S+      wait     0xfffff80001852000 sh
  955   832   819     0  S+      wait     0xfffff80001853770 make
  832   828   819     0  S+      wait     0xfffff80001803770 sh
  828   819   819     0  S+      wait     0xfffff800018532c0 make
  819   797   819     0  S+      wait     0xfffff80001852e10 make
  797   784   797     0  S+      pause    0xfffff800018000a8 csh
  795     1   795     0  Ss+     ttyin    0xfffff800012100a8 getty
  794     1   794     0  Ss+     ttyin    0xfffff800012104a8 getty
  793     1   793     0  Ss+     ttyin    0xfffff800012120a8 getty
  792     1   792     0  Ss+     ttyin    0xfffff800012124a8 getty
  791     1   791     0  Ss+     ttyin    0xfffff800010f80a8 getty
  790     1   790     0  Ss+     ttyin    0xfffff800010f84a8 getty
  789     1   789     0  Ss+     ttyin    0xfffff800010f88a8 getty
  788     1   788     0  Ss+     wait     0xfffff800017a2000 login
  787     1   787     0  Ss+     wait     0xfffff800017a32c0 login
  786     1   786     0  Ss+     wait     0xfffff8000154f2c0 login
  785     1   785     0  Ss+     wait     0xfffff8000154e960 login
  784     1   784     0  Ss+     wait     0xfffff800010792c0 login
  743     1   743     0  Ss      nanslp   0xc07c1240 cron
  739     1   739    25  Ss      pause    0xfffff800017a2a08 sendmail
  736     1   736     0  Ss      select   0xfffff8000172b2c0 sendmail
  733     1   733     0  Ss      select   0xfffff8000172b4c0 sshd
  690     1   690     0  Ss      select   0xfffff800011afec0 ntpd
  667     1   667     0  Ss      select   0xfffff8000172b6c0 lpd
  625     1   625     0  Ss      select   0xfffff8000172b840 casperd
  624     1    17     0  S+      select   0xfffff8000172b940 casperd
  600     1   600     0  Ss      select   0xfffff8000172b9c0 syslogd
  477     1   477     0  Ss      select   0xfffff8000172bac0 devd
  460     1   460     0  Ss      select   0xfffff8000172bbc0 moused
  115     1   115     0  Ss      pause    0xfffff8000154e558 adjkerntz
   16     0     0     0  DL      sdflush  0xc0ac5e48 [softdepflush]
    9     0     0     0  DL      syncer   0xc0abdcf8 [syncer]
    8     0     0     0  DL      vlruwt   0xfffff8000149a4b0 [vnlru]
    7     0     0     0  DL      psleep   0xc0abd800 [bufdaemon]
    6     0     0     0  DL      pgzero   0xc0ac880c [pagezero]
    5     0     0     0  DL      psleep   0xc0ac749c [vmdaemon]
    4     0     0     0  DL      psleep   0xc0ac74ec [pagedaemon]
    3     0     0     0  DL      waiting_ 0xc0ac07f0 [sctp_iterator]
   15     0     0     0  DL      (threaded)                  [usb]
100029                   D       -        0x1001af460 [usbus1]
100028                   D       -        0x1001af408 [usbus1]
100027                   D       -        0x1001af3b0 [usbus1]
100026                   D       -        0x1001af358 [usbus1]
100024                   D       -        0x1001ab460 [usbus0]
100023                   D       -        0x1001ab408 [usbus0]
100022                   D       -        0x1001ab3b0 [usbus0]
100021                   D       -        0x1001ab358 [usbus0]
    2     0     0     0  DL      (threaded)                  [cam]
100036                   D       ccb_scan 0xc078a6c0 [scanner]
100018                   D       -        0xc078a880 [doneq0]
   14     0     0     0  DL      -        0xc078e2e0 [rand_harvestq]
   13     0     0     0  DL      (threaded)                  [geom]
100009                   D       -        0xc07bbf68 [g_down]
100008                   D       -        0xc07bbf60 [g_up]
100007                   D       -        0xc07bbf50 [g_event]
   12     0     0     0  WL      (threaded)                  [intr]
100032                   I                                   [swi0: uart uart]
100031                   I                                   [vec2012: bge0]
100030                   I                                   [vec1944: atapci0]
100025                   I                                   [vec1958: ohci1]
100020                   I                                   [vec1959: ohci0]
100019                   I                                   [vec1956: pcm0]
100017                   I                                   [swi6: task queue]
100016                   I                                   [swi6: Giant taskq]
100014                   I                                   [swi5: fast taskq]
100006                   I                                   [swi4: clock (0)]
100005                   I                                   [swi1: netisr 0]
100004                   I                                   [swi3: vm]
   11     0     0     0  RL                                  [idle]
    1     0     1     0  SLs     wait     0xfffff800010784b0 [init]
   10     0     0     0  DL      audit_wo 0xc0ac5218 [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100035                   D       -        0xfffff800010ab600 [CAM taskq]
100033                   D       -        0xc07bff28 [deadlkres]
100015                   D       -        0xfffff800010ab900 [thread taskq]
100013                   D       -        0xfffff800010abb00 [kqueue taskq]
100012                   D       -        0xfffff800010abc00 [ffs_trim taskq]
100010                   D       -        0xfffff800010abd00 [firmware taskq]
100000                   D       swapin   0xc07bc160 [swapper]
db:0:kdb.enter.panic>  alltrace

Tracing command tr pid 55020 tid 100080 td 0xfffff80001806db0
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
pipe_read() at pipe_read+0x484
dofileread() at dofileread+0x74
kern_readv() at kern_readv+0x58
sys_read() at sys_read+0x74
syscall() at syscall+0x304
-- syscall (3, FreeBSD ELF64, sys_read) %o7=0x403dcb90 --
userland() at 0x40479ac8
user trace: trap %o7=0x403dcb90
pc 0x40479ac8, sp 0x7fdffffd911
done

Tracing command sed pid 55019 tid 100105 td 0xfffff80001e1d240
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
pipe_read() at pipe_read+0x484
dofileread() at dofileread+0x74
kern_readv() at kern_readv+0x58
sys_read() at sys_read+0x74
syscall() at syscall+0x304
-- syscall (3, FreeBSD ELF64, sys_read) %o7=0x403e2b90 --
userland() at 0x4047fac8
user trace: trap %o7=0x403e2b90
pc 0x4047fac8, sp 0x7fdffffd761
done

Tracing command tr pid 55018 tid 100095 td 0xfffff80001e1adb0
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command sh pid 55016 tid 100092 td 0xfffff80001e1bb60
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x10ee78 --
userland() at 0x4071b9c8
user trace: trap %o7=0x10ee78
pc 0x4071b9c8, sp 0x7fdffffc211
done

Tracing command sh pid 54937 tid 100083 td 0xfffff800018bb240
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
pipe_read() at pipe_read+0x484
dofileread() at dofileread+0x74
kern_readv() at kern_readv+0x58
sys_read() at sys_read+0x74
syscall() at syscall+0x304
-- syscall (3, FreeBSD ELF64, sys_read) %o7=0x10ab20 --
userland() at 0x4071bac8
user trace: trap %o7=0x10ab20
pc 0x4071bac8, sp 0x7fdffffc631
pc 0x7fdffffd7f4, sp 0xc
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x1dc2 %o7=0xc0641b40 --
copyin() at copyin+0xac
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command sh pid 54936 tid 100093 td 0xfffff80001e1b6d0
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x10ee78 --
userland() at 0x4071b9c8
user trace: trap %o7=0x10ee78
pc 0x4071b9c8, sp 0x7fdffffd7a1
done

Tracing command make pid 53662 tid 100108 td 0xfffff80001e1c490
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x8
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x390
seltdwait() at seltdwait+0x84
sys_poll() at sys_poll+0x30c
syscall() at syscall+0x304
-- syscall (209, FreeBSD ELF64, sys_poll) %o7=0x10a928 --
userland() at 0x152288
user trace: trap %o7=0x10a928
pc 0x152288, sp 0x7fdffffc8c1
done

Tracing command sh pid 53661 tid 100103 td 0xfffff80001e1db60
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x10ee78 --
userland() at 0x4071b9c8
user trace: trap %o7=0x10ee78
pc 0x4071b9c8, sp 0x7fdffffd7a1
done

Tracing command make pid 53653 tid 100094 td 0xfffff80001e1b240
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x8
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x390
seltdwait() at seltdwait+0x84
sys_poll() at sys_poll+0x30c
syscall() at syscall+0x304
-- syscall (209, FreeBSD ELF64, sys_poll) %o7=0x10a928 --
userland() at 0x152288
user trace: trap %o7=0x10a928
pc 0x152288, sp 0x7fdffffc8c1
done

Tracing command sh pid 53614 tid 100086 td 0xfffff800018ba490
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x10ee78 --
userland() at 0x4071b9c8
user trace: trap %o7=0x10ee78
pc 0x4071b9c8, sp 0x7fdffffd581
done

Tracing command make pid 53613 tid 100102 td 0xfffff8021bbb6000
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x8
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x390
seltdwait() at seltdwait+0x84
sys_poll() at sys_poll+0x30c
syscall() at syscall+0x304
-- syscall (209, FreeBSD ELF64, sys_poll) %o7=0x10a928 --
userland() at 0x152288
user trace: trap %o7=0x10a928
pc 0x152288, sp 0x7fdffffc8d1
done

Tracing command sh pid 53612 tid 100096 td 0xfffff80001e1a920
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x10ee78 --
userland() at 0x4071b9c8
user trace: trap %o7=0x10ee78
pc 0x4071b9c8, sp 0x7fdffffd7b1
done

Tracing command make pid 53611 tid 100045 td 0xfffff800011a8490
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x8
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x390
seltdwait() at seltdwait+0x84
sys_poll() at sys_poll+0x30c
syscall() at syscall+0x304
-- syscall (209, FreeBSD ELF64, sys_poll) %o7=0x10a928 --
userland() at 0x152288
user trace: trap %o7=0x10a928
pc 0x152288, sp 0x7fdffffc8c1
done

Tracing command sh pid 53610 tid 100104 td 0xfffff80001e1d6d0
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x10ee78 --
userland() at 0x4071b9c8
user trace: trap %o7=0x10ee78
pc 0x4071b9c8, sp 0x7fdffffd341
pc 0xc0804000, sp 0x40596000
done

Tracing command ssh pid 38770 tid 100107 td 0xfffff80001e1c920
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_cv_wait_sig() at _cv_wait_sig+0x390
seltdwait() at seltdwait+0x94
kern_select() at kern_select+0x6f8
sys_select() at sys_select+0x44
syscall() at syscall+0x304
-- syscall (93, FreeBSD ELF64, sys_select) %o7=0x1119ac --
userland() at 0x40edeb48
user trace: trap %o7=0x1119ac
pc 0x40edeb48, sp 0x7fdffffa061
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x7fdffffbdc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command tcsh pid 38748 tid 100063 td 0xfffff800015516d0
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_sigsuspend() at kern_sigsuspend+0xc4
sys_sigsuspend() at sys_sigsuspend+0x28
syscall() at syscall+0x304
-- syscall (341, FreeBSD ELF64, sys_sigsuspend) %o7=0x123b50 --
userland() at 0x40744548
user trace: trap %o7=0x123b50
pc 0x40744548, sp 0x7fdffffd581
done

Tracing command ssh pid 37571 tid 100082 td 0xfffff800018bb6d0
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_cv_wait_sig() at _cv_wait_sig+0x390
seltdwait() at seltdwait+0x94
kern_select() at kern_select+0x6f8
sys_select() at sys_select+0x44
syscall() at syscall+0x304
-- syscall (93, FreeBSD ELF64, sys_select) %o7=0x1119ac --
userland() at 0x40edeb48
user trace: trap %o7=0x1119ac
pc 0x40edeb48, sp 0x7fdffffa061
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x7fdffffbdc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command tcsh pid 37484 tid 100081 td 0xfffff80001806920
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_sigsuspend() at kern_sigsuspend+0xc4
sys_sigsuspend() at sys_sigsuspend+0x28
syscall() at syscall+0x304
-- syscall (341, FreeBSD ELF64, sys_sigsuspend) %o7=0x123b50 --
userland() at 0x40744548
user trace: trap %o7=0x123b50
pc 0x40744548, sp 0x7fdffffd581
done

Tracing command ssh pid 25973 tid 100106 td 0xfffff80001e1cdb0
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_cv_wait_sig() at _cv_wait_sig+0x390
seltdwait() at seltdwait+0x94
kern_select() at kern_select+0x6f8
sys_select() at sys_select+0x44
syscall() at syscall+0x304
-- syscall (93, FreeBSD ELF64, sys_select) %o7=0x1119ac --
userland() at 0x40edeb48
user trace: trap %o7=0x1119ac
pc 0x40edeb48, sp 0x7fdffffa061
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x7fdffffbdc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command tcsh pid 10162 tid 100099 td 0xfffff800018bbb60
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_sigsuspend() at kern_sigsuspend+0xc4
sys_sigsuspend() at sys_sigsuspend+0x28
syscall() at syscall+0x304
-- syscall (341, FreeBSD ELF64, sys_sigsuspend) %o7=0x123b50 --
userland() at 0x40744548
user trace: trap %o7=0x123b50
pc 0x40744548, sp 0x7fdffffd581
done

Tracing command csh pid 3559 tid 100097 td 0xfffff80001e1a490
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_cv_wait_sig() at _cv_wait_sig+0x390
tty_wait() at tty_wait+0x84
ttydisc_read() at ttydisc_read+0x2a4
ttydev_read() at ttydev_read+0x84
devfs_read_f() at devfs_read_f+0xa8
dofileread() at dofileread+0x74
kern_readv() at kern_readv+0x58
sys_read() at sys_read+0x74
syscall() at syscall+0x304
-- syscall (3, FreeBSD ELF64, sys_read) %o7=0x11ee98 --
userland() at 0x40747ac8
user trace: trap %o7=0x11ee98
pc 0x40747ac8, sp 0x7fdffffd5d1
pc 0x6474e551, sp 0x7fdffffe760
done

Tracing command make pid 977 tid 100077 td 0xfffff80001807b60
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x1027e8 --
userland() at 0x152a88
user trace: trap %o7=0x1027e8
pc 0x152a88, sp 0x7fdffffc031
done

Tracing command sh pid 960 tid 100078 td 0xfffff800018076d0
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x10ee78 --
userland() at 0x4071b9c8
user trace: trap %o7=0x10ee78
pc 0x4071b9c8, sp 0x7fdffffcb21
done

Tracing command make pid 955 tid 100073 td 0xfffff80001858db0
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x1027e8 --
userland() at 0x152a88
user trace: trap %o7=0x1027e8
pc 0x152a88, sp 0x7fdffffbd21
done

Tracing command sh pid 832 tid 100079 td 0xfffff80001807240
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x10ee78 --
userland() at 0x4071b9c8
user trace: trap %o7=0x10ee78
pc 0x4071b9c8, sp 0x7fdffffbcd1
done

Tracing command make pid 828 tid 100074 td 0xfffff80001858920
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x1027e8 --
userland() at 0x152a88
user trace: trap %o7=0x1027e8
pc 0x152a88, sp 0x7fdffffc4e1
pc 0, sp 0x5
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x1dc2 %o7=0xc0641b40 --
copyin() at copyin+0xac
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command make pid 819 tid 100075 td 0xfffff80001858490
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x1027e8 --
userland() at 0x152a88
user trace: trap %o7=0x1027e8
pc 0x152a88, sp 0x7fdffffc9c1
done

Tracing command csh pid 797 tid 100072 td 0xfffff80001804000
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_sigsuspend() at kern_sigsuspend+0xc4
sys_sigsuspend() at sys_sigsuspend+0x28
syscall() at syscall+0x304
-- syscall (341, FreeBSD ELF64, sys_sigsuspend) %o7=0x123b50 --
userland() at 0x40744548
user trace: trap %o7=0x123b50
pc 0x40744548, sp 0x7fdffffd5a1
pc 0x4031edb8, sp 0x40224000
done

Tracing command getty pid 795 tid 100070 td 0xfffff80001804920
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_cv_wait_sig() at _cv_wait_sig+0x390
tty_wait() at tty_wait+0x84
ttydisc_read() at ttydisc_read+0x2a4
ttydev_read() at ttydev_read+0x84
devfs_read_f() at devfs_read_f+0xa8
dofileread() at dofileread+0x74
kern_readv() at kern_readv+0x58
sys_read() at sys_read+0x74
syscall() at syscall+0x304
-- syscall (3, FreeBSD ELF64, sys_read) %o7=0x102390 --
userland() at 0x40593ac8
user trace: trap %o7=0x102390
pc 0x40593ac8, sp 0x7fdffffe241
pc 0x59533d4672656542, sp 0x72633634004f5053
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x72633634004f5dc2 %o7=0xc0641b40 --
copyin() at copyin+0xac
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command getty pid 794 tid 100069 td 0xfffff80001804db0
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_cv_wait_sig() at _cv_wait_sig+0x390
tty_wait() at tty_wait+0x84
ttydisc_read() at ttydisc_read+0x2a4
ttydev_read() at ttydev_read+0x84
devfs_read_f() at devfs_read_f+0xa8
dofileread() at dofileread+0x74
kern_readv() at kern_readv+0x58
sys_read() at sys_read+0x74
syscall() at syscall+0x304
-- syscall (3, FreeBSD ELF64, sys_read) %o7=0x102390 --
userland() at 0x40593ac8
user trace: trap %o7=0x102390
pc 0x40593ac8, sp 0x7fdffffe241
pc 0x59533d4672656542, sp 0x72633634004f5053
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x72633634004f5dc2 %o7=0xc0641b40 --
copyin() at copyin+0xac
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command getty pid 793 tid 100068 td 0xfffff80001805240
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_cv_wait_sig() at _cv_wait_sig+0x390
tty_wait() at tty_wait+0x84
ttydisc_read() at ttydisc_read+0x2a4
ttydev_read() at ttydev_read+0x84
devfs_read_f() at devfs_read_f+0xa8
dofileread() at dofileread+0x74
kern_readv() at kern_readv+0x58
sys_read() at sys_read+0x74
syscall() at syscall+0x304
-- syscall (3, FreeBSD ELF64, sys_read) %o7=0x102390 --
userland() at 0x40593ac8
user trace: trap %o7=0x102390
pc 0x40593ac8, sp 0x7fdffffe251
pc 0x4e543d5945530043, sp 0x5344004e4f5f4c49
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x5344004e4f5f5dc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command getty pid 792 tid 100067 td 0xfffff800018056d0
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_cv_wait_sig() at _cv_wait_sig+0x390
tty_wait() at tty_wait+0x84
ttydisc_read() at ttydisc_read+0x2a4
ttydev_read() at ttydev_read+0x84
devfs_read_f() at devfs_read_f+0xa8
dofileread() at dofileread+0x74
kern_readv() at kern_readv+0x58
sys_read() at sys_read+0x74
syscall() at syscall+0x304
-- syscall (3, FreeBSD ELF64, sys_read) %o7=0x102390 --
userland() at 0x40593ac8
user trace: trap %o7=0x102390
pc 0x40593ac8, sp 0x7fdffffe251
pc 0x4e543d5945530043, sp 0x5344004e4f5f4c49
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x5344004e4f5f5dc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command getty pid 791 tid 100066 td 0xfffff80001805b60
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_cv_wait_sig() at _cv_wait_sig+0x390
tty_wait() at tty_wait+0x84
ttydisc_read() at ttydisc_read+0x2a4
ttydev_read() at ttydev_read+0x84
devfs_read_f() at devfs_read_f+0xa8
dofileread() at dofileread+0x74
kern_readv() at kern_readv+0x58
sys_read() at sys_read+0x74
syscall() at syscall+0x304
-- syscall (3, FreeBSD ELF64, sys_read) %o7=0x102390 --
userland() at 0x40593ac8
user trace: trap %o7=0x102390
pc 0x40593ac8, sp 0x7fdffffe251
pc 0x4e543d5945530043, sp 0x5344004e4f5f4c49
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x5344004e4f5f5dc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command getty pid 790 tid 100065 td 0xfffff80001806000
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_cv_wait_sig() at _cv_wait_sig+0x390
tty_wait() at tty_wait+0x84
ttydisc_read() at ttydisc_read+0x2a4
ttydev_read() at ttydev_read+0x84
devfs_read_f() at devfs_read_f+0xa8
dofileread() at dofileread+0x74
kern_readv() at kern_readv+0x58
sys_read() at sys_read+0x74
syscall() at syscall+0x304
-- syscall (3, FreeBSD ELF64, sys_read) %o7=0x102390 --
userland() at 0x40593ac8
user trace: trap %o7=0x102390
pc 0x40593ac8, sp 0x7fdffffe251
pc 0x4e543d5945530043, sp 0x5344004e4f5f4c49
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x5344004e4f5f5dc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command getty pid 789 tid 100064 td 0xfffff80001806490
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_cv_wait_sig() at _cv_wait_sig+0x390
tty_wait() at tty_wait+0x84
ttydisc_read() at ttydisc_read+0x2a4
ttydev_read() at ttydev_read+0x84
devfs_read_f() at devfs_read_f+0xa8
dofileread() at dofileread+0x74
kern_readv() at kern_readv+0x58
sys_read() at sys_read+0x74
syscall() at syscall+0x304
-- syscall (3, FreeBSD ELF64, sys_read) %o7=0x102390 --
userland() at 0x40593ac8
user trace: trap %o7=0x102390
pc 0x40593ac8, sp 0x7fdffffe251
pc 0x4e543d5945530043, sp 0x5344004e4f5f4c49
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x5344004e4f5f5dc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command login pid 788 tid 100060 td 0xfffff800017a4490
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x103ef4 --
userland() at 0x407c19c8
user trace: trap %o7=0x103ef4
pc 0x407c19c8, sp 0x7fdffffe251
pc 0x4e543d5945530043, sp 0x5344004e4f5f4c49
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x5344004e4f5f5dc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command login pid 787 tid 100056 td 0xfffff800017a56d0
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x103ef4 --
userland() at 0x407c19c8
user trace: trap %o7=0x103ef4
pc 0x407c19c8, sp 0x7fdffffe251
pc 0x4e543d5945530043, sp 0x5344004e4f5f4c49
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x5344004e4f5f5dc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command login pid 786 tid 100062 td 0xfffff80001551b60
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x103ef4 --
userland() at 0x407c19c8
user trace: trap %o7=0x103ef4
pc 0x407c19c8, sp 0x7fdffffe251
pc 0x4e543d5945530043, sp 0x5344004e4f5f4c49
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x5344004e4f5f5dc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command login pid 785 tid 100046 td 0xfffff80001551240
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x103ef4 --
userland() at 0x407c19c8
user trace: trap %o7=0x103ef4
pc 0x407c19c8, sp 0x7fdffffe251
pc 0x4e543d5945530043, sp 0x5344004e4f5f4c49
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x5344004e4f5f5dc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_trap() at kdb_trap+0xfc
trap() at trap+0x358
-- breakpoint %o7=0xc036ac40 --
kdb_enter() at kdb_enter+0x80
vpanic() at vpanic+0x140
kassert_panic() at kassert_panic+0xb0
vm_page_alloc() at vm_page_alloc+0x450
vm_fault_hold() at vm_fault_hold+0x6e0
vm_fault() at vm_fault+0x88
trap_pfault() at trap_pfault+0x1ac
trap() at trap+0xdc
-- data access protection tar=0x40597dc2 sfar=0x40596000 sfsr=0x800005 %o7=0x40205ec8 --
userland() at 0x4020612c
user trace: trap %o7=0x40205ec8
pc 0x4020612c, sp 0x7fdffffd3a1
pc 0x40209f8c, sp 0x7fdffffd481
pc 0x4020a150, sp 0x7fdffffd541
pc 0x4020dfe8, sp 0x7fdffffd601
pc 0x402058f4, sp 0x7fdffffde81
done

Tracing command login pid 784 tid 100044 td 0xfffff800011a8920
mi_switch() at mi_switch+0x2d8
sleepq_switch() at sleepq_switch+0x22c
sleepq_catch_signals() at sleepq_catch_signals+0x134
sleepq_wait_sig() at sleepq_wait_sig+0x8
_sleep() at _sleep+0x4a8
kern_wait6() at kern_wait6+0x7b4
kern_wait() at kern_wait+0x80
sys_wait4() at sys_wait4+0x20
syscall() at syscall+0x304
-- syscall (7, FreeBSD ELF64, sys_wait4) %o7=0x103ef4 --
userland() at 0x407c19c8
user trace: trap %o7=0x103ef4
pc 0x407c19c8, sp 0x7fdffffe251
pc 0x4e543d5945530043, sp 0x5344004e4f5f4c49
kdb_reenter() at kdb_reenter+0x5c
trap() at trap+0x2fc
-- fast data access mmu miss tar=0x5344004e4f5f5dc2 %o7=0xc0641b40 --
copyin() at copyin+0x8c
db_trace_thread() at db_trace_thread+0x18
db_stack_trace_all() at db_stack_trace_all+0xa4
db_command() at db_command+0x48c
db_command_script() at db_command_script+0x84
db_script_exec() at db_script_exec+0x1e0
db_script_kdbenter() at db_script_kdbenter+0x70
db_trap() at db_trap+0x1a4
kdb_

This panic looks similar to kern/187080.
Comment 1 Michael Moll freebsd_committer freebsd_triage 2015-11-29 19:10:48 UTC
Anton, what's the status here with an up-to-date -CURRENT?
Comment 2 Michael Moll freebsd_committer freebsd_triage 2015-12-06 15:55:43 UTC
No reaction, closing. I did have vm related panics on sparc64 around 05/2014 also, but can't reproduce on a recent CURRENT.