Bug 187816 - [panic] ia64 11.0-CURRENT #13 r262690: fatal kernel trap: Page Not Present
Summary: [panic] ia64 11.0-CURRENT #13 r262690: fatal kernel trap: Page Not Present
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: Normal Affects Only Me
Assignee: Marcel Moolenaar
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-03-21 13:10 UTC by Anton Shterenlikht
Modified: 2015-11-10 13:38 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Anton Shterenlikht 2014-03-21 13:10:00 UTC
Happened during poudriere build.
I collected some data via textdumps:

# cat info.last 
Dump header from device /dev/da0p3
  Architecture: ia64
  Architecture Version: 1
  Dump Length: 68096B (0 MB)
  Blocksize: 512
  Dumptime: Fri Mar 21 11:43:11 2014
  Hostname: mech-as221.men.bris.ac.uk
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 11.0-CURRENT #13 r262690M: Thu Mar 20 15:39:46 GMT 2014
    root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI
  Panic String: 
  Dump Parity: 2808021516
  Bounds: 1
  Dump Status: good
# 

# cat version.txt 
FreeBSD 11.0-CURRENT #13 r262690M: Thu Mar 20 15:39:46 GMT 2014
    root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI
#

****************************************************
config.txt
****************************************************
options	CONFIG_AUTOGENERATED
ident	UZI
machine	ia64
cpu	ITANIUM2
makeoptions	MODULES_OVERRIDE=geom/geom_part opensolaris zfs acl_nfs4 acl_posix1e
makeoptions	DEBUG=-g
options	USB_DEBUG
options	TEXTDUMP_VERBOSE
options	TEXTDUMP_PREFERRED
options	DEBUG_VFS_LOCKS
options	DEBUG_LOCKS
options	MALLOC_DEBUG_MAXZONES=8
options	WITNESS_SKIPSPIN
options	WITNESS
options	INVARIANT_SUPPORT
options	INVARIANTS
options	DEADLKRES
options	GDB
options	DDB
options	KDB_TRACE
options	KDB
options	_KPOSIX_PRIORITY_SCHEDULING
options	QUOTA
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SYSVSHM
options	SYSVSEM
options	SYSVMSG
options	STACK
options	SOFTUPDATES
options	SMP
options	SCTP
options	SCSI_DELAY=5000
options	SCHED_ULE
options	PSEUDOFS
options	PROCFS
options	PRINTF_BUFR_SIZE=128
options	PREEMPTION
options	P1003_1B_SEMAPHORES
options	NFS_ROOT
options	NFSD
options	NFSLOCKD
options	NFSCL
options	MSDOSFS
options	MD_ROOT
options	MAC
options	KTRACE
options	INET6
options	INET
options	INCLUDE_CONFIG_FILE
options	GEOM_LABEL
options	FFS
options	TMPFS
options	NULLFS
options	FDESCFS
options	COMPAT_FREEBSD7
options	CD9660
options	CAPABILITIES
options	CAPABILITY_MODE
options	AUDIT
options	NEW_PCIB
options	GEOM_PART_MBR
options	GEOM_PART_GPT
options	GEOM_PART_BSD
device	acpi
device	io
device	mem
device	uart_ns8250
device	miibus
device	pci
device	scbus
device	usb
device	mpt
device	cd
device	da
device	pass
device	ehci
device	ohci
device	uhci
device	umass
device	bge
device	ether
device	faith
device	firmware
device	gif
device	loop
device	md
device	puc
device	random
device	tun
device	uart
device	vlan


******************************************
ddb.txt
******************************************
db:0:on>  run lockinfo
db:1:lockinfo> show locks
db:1:locks>  show alllocks
db:1:alllocks>  show lockedvnods
Locked vnodes
db:0:lockedvnods>  show pcpu
cpuid        = 1
dynamic pcpu = 0x400005fc29800
curthread    = 0xe000000109938000: pid 93024 "automoc4"
curpcb       = 0
fpcurthread  = none
idlethread   = 0xe000000010b95680: tid 100004 "idle: cpu1"
MD: vhpt       = 0xe00000003f100000
MD: lid        = 0x1000000
MD: clock      = 0x603a6a274f37
MD: clock_mode = 2
MD: clock_load = 0
MD: stats      = 0xe000000010b6ea30
MD: pmap       = 0xe000000010ba0758
spin locks held:
db:0:pcpu>  bt
Tracing pid 93024 tid 100480 td 0xe000000109938000
kern_select(0xe000000109938000, 0x0, 0x7fffffffffbfdf28, 0x0, 0x0, 0xffffffffffffffff, 0xa) at kern_select+0x7a0
sys_select(0xe000000109938000, 0xa0000000907514e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0
syscall(0xe000000012f66940, 0x7fffffffffbfdf28, 0x0, 0xe000000109938000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
db:0:bt>  ps
  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
93043 68771 14849     0  R                                   sh
93042 68771 14849     0  S       piperd   0xe0000040f31cd0d8 awk
93041 68771 14849     0  R       CPU 0                       awk
93024 93023 14849     0  RJ      (threaded)                  automoc4
100480                   Run     CPU 1                       automoc4
100382                   S       select   0xe0000000123de0c0 automoc4
93023 90353 14849     0  SJ      wait     0xe000000109914940 make
93007 89041 14849     0  RJ                                  qmake-qt4
92828 92824 14849     0  SJ      piperd   0xe0000000130edce0 as
92825 92824 14849     0  RJ                                  cc1plus
92824 92776 14849     0  SJ      wait     0xe000000012dfade0 c++
92776 60325 14849     0  SJ      wait     0xe000000012dfa4a0 sh
90353 90339 14849     0  SJ      wait     0xe000000012f68000 make
90339 90337 14849     0  SJ      wait     0xe000000012f664a0 make
90337 89415 14849     0  SJ      wait     0xe000000012dfa940 sh
89415 89413 14849     0  SJ      wait     0xe0000040ba8df280 make
89413 79742 14849     0  S       wait     0xe0000040ba8e1280 sh
89041 89038 14849     0  SJ      wait     0xe000000013290940 sh
89038 88992 14849     0  SJ      wait     0xe0000040ba8e0de0 sh
88992 88989 14849     0  SJ      wait     0xe0000040ba77c000 make
88989 79716 14849     0  S       wait     0xe0000040ba8e0000 sh
80044 79742 14849     0  S       piperd   0xe0000040f31cd340 tee
79991 79716 14849     0  S       piperd   0xe0000040ba7abce0 tee
79742 17626 14849     0  S       select   0xe0000040ba9786c0 sh
79716 17626 14849     0  S       select   0xe0000040ba979840 sh
60325 31159 14849     0  SJ      wait     0xe0000040ba760de0 gmake
31159 31156 14849     0  SWJ     wait     0xe0000000122c8de0 gmake
31156 31094 14849     0  SWJ     wait     0xe0000040ba8de940 sh
31094 31092 14849     0  SWJ     wait     0xe0000040f2fd0000 make
31092 11467 14849     0  SW      wait     0xe0000040f2fd0de0 sh
11599 11467 14849     0  S       piperd   0xe0000040ba7ab340 tee
11467 17626 14849     0  S       select   0xe0000000123dd240 sh
 8190  8189  8189    80  S       kqread   0xe0000040bac54600 nginx
 8189     1  8189     0  SWs     pause    0xe0000040f31c29e8 nginx
68771 17626 14849     0  S       wait     0xe0000040ba77cde0 sh
17626 14849 14849     0  S       select   0xe00000001305c340 sh
14861 14847 14861    25  Ss      piperd   0xe0000000130ecc08 sendmail
14849 14847 14849     0  SWs     wait     0xe000000012b48de0 sh
14847   785   785     0  S       piperd   0xe000000012f655a8 cron
  844     1   844     0  Ss+     ttyin    0xe000000011c30ca8 getty
  843     1     1     0  S       ttydcd   0xe000000011c310e8 getty
  820     1   820     0  Ss      select   0xe000000011c3f6c0 inetd
  785     1   785     0  Ss      nanslp   0x9ffc000000c2fa38 cron
  781     1   781    25  Ss      pause    0xe000000012e520a8 sendmail
  778     1   778     0  Ss      select   0xe000000011c3f3c0 sendmail
  775     1   775     0  Ss      select   0xe000000011c3f4c0 sshd
  703     1   703     0  Ss      select   0xe0000000123df540 ntpd
  641     1   641     0  Ss      select   0xe0000000123df940 casperd
  640     1    22     0  S+      select   0xe0000000123df6c0 casperd
  616     1   616     0  Ss      select   0xe0000000123df8c0 syslogd
  490     1   490     0  Ss      select   0xe0000000123df740 devd
  139     1   139     0  SWs     pause    0xe0000000122cf328 adjkerntz
   40     0     0     0  DL      (threaded)                  [zfskern]
100341                   D       tx->tx_s 0xe0000000122d3208 [txg_thread_enter]
100340                   D       tx->tx_q 0xe0000000122d3228 [txg_thread_enter]
100334                   D       spa->spa 0xe0000000123c8ac0 [trim tank]
100067                   D       l2arc_fe 0xa0000000013dd540 [l2arc_feed_thread]
100064                   D       arc_recl 0xa0000000013cd720 [arc_reclaim_thread]
   21     0     0     0  DL      sdflush  0x9ffc000000bfb890 [softdepflush]
   20     0     0     0  DL      vlruwt   0xe0000000122ce4a0 [vnlru]
   19     0     0     0  DL      syncer   0x9ffc000000f65b10 [syncer]
   18     0     0     0  DL      psleep   0x9ffc000000bfac70 [bufdaemon]
   17     0     0     0  DL      pgzero   0x9ffc000000bfbbfc [pagezero]
    9     0     0     0  DL      psleep   0x9ffc000000bfbb64 [vmdaemon]
    8     0     0     0  DL      psleep   0x9ffc000000bfbb94 [pagedaemon]
    5     0     0     0  DL      waiting_ 0x9ffc000000f681f0 [sctp_iterator]
    4     0     0     0  DL      idle     0xa000000000a22000 [mpt_recovery1]
    3     0     0     0  DL      idle     0xa000000000a0a000 [mpt_recovery0]
   16     0     0     0  DL      (threaded)                  [usb]
100039                   D       -        0xa000000000a06e18 [usbus2]
100038                   D       -        0xa000000000a06dc0 [usbus2]
100037                   D       -        0xa000000000a06d68 [usbus2]
100036                   D       -        0xa000000000a06d10 [usbus2]
100034                   D       -        0xa0000000009fd460 [usbus1]
100033                   D       -        0xa0000000009fd408 [usbus1]
100032                   D       -        0xa0000000009fd3b0 [usbus1]
100031                   D       -        0xa0000000009fd358 [usbus1]
100029                   D       -        0xa0000000009f9460 [usbus0]
100028                   D       -        0xa0000000009f9408 [usbus0]
100027                   D       -        0xa0000000009f93b0 [usbus0]
100026                   D       -        0xa0000000009f9358 [usbus0]
   15     0     0     0  DL      tzpoll   0x9ffc000000bf9670 [acpi_thermal]
    2     0     0     0  DL      (threaded)                  [cam]
100051                   D       ccb_scan 0x9ffc000000bfbf38 [scanner]
100015                   D       -        0x9ffc000000bfc080 [doneq0]
   14     0     0     0  DL      -        0x9ffc000000bf98e8 [rand_harvestq]
   13     0     0     0  DL      (threaded)                  [geom]
100011                   D       -        0x9ffc000000bf9e28 [g_down]
100010                   D       -        0x9ffc000000bf9e20 [g_up]
100009                   D       -        0x9ffc000000bf9e10 [g_event]
   12     0     0     0  WL      (threaded)                  [intr]
100045                   I                                   [swi0: uart uart++++]
100044                   I                                   [irq29: bge0]
100042                   I                                   [irq28: mpt1]
100040                   I                                   [irq27: mpt0]
100035                   I                                   [irq18: ehci0]
100030                   I                                   [irq17: ohci1]
100025                   I                                   [irq16: ohci0]
100023                   I                                   [swi6: Giant taskq]
100021                   I                                   [swi5: fast taskq]
100014                   I                                   [swi6: task queue]
100008                   I                                   [swi1: netisr 0]
100007                   I                                   [swi3: vm]
100006                   I                                   [swi4: clock (1)]
100005                   I                                   [swi4: clock (0)]
   11     0     0     0  RL      (threaded)                  [idle]
100004                   CanRun                              [idle: cpu1]
100003                   CanRun                              [idle: cpu0]
    1     0     1     0  SLs     wait     0xe000000010b924a0 [init]
   10     0     0     0  DL      audit_wo 0x9ffc000000f6c888 [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100478                   D       -        0xe000000010bb8d00 [zil_clean]
100477                   D       -        0xe0000040bac55200 [zil_clean]
100476                   D       -        0xe00000010998cf00 [zil_clean]
100471                   D       -        0xe000000108edbd00 [zil_clean]
100394                   D       -        0xe000000010bb8e00 [zil_clean]
100350                   D       -        0xe000000010bb9800 [zil_clean]
100349                   D       -        0xe000000010bb9900 [zil_clean]
100348                   D       -        0xe000000012264e00 [zil_clean]
100347                   D       -        0xe000000012264f00 [zil_clean]
100346                   D       -        0xe000000012265000 [zil_clean]
100345                   D       -        0xe000000012265100 [zil_clean]
100344                   D       -        0xe000000012265200 [zil_clean]
100339                   D       -        0xe000000010bb9a00 [metaslab_group_task]
100338                   D       -        0xe000000010bb9b00 [metaslab_group_task]
100337                   D       -        0xe000000012265300 [zfs_vn_rele_taskq]
100336                   D       -        0xe000000010bb9c00 [metaslab_group_task]
100335                   D       -        0xe000000012265400 [metaslab_group_task]
100333                   D       -        0xe000000012265c00 [zio_ioctl_intr]
100332                   D       -        0xe000000010bbb300 [zio_ioctl_issue]
100331                   D       -        0xe000000010bbb200 [zio_claim_intr]
100330                   D       -        0xe000000010bbb100 [zio_claim_issue]
100329                   D       -        0xe000000010bbb000 [zio_free_intr]
100328                   D       -        0xe000000010bbaf00 [zio_free_issue_7_11]
100327                   D       -        0xe000000010bbaf00 [zio_free_issue_7_10]
100326                   D       -        0xe000000010bbaf00 [zio_free_issue_7_9]
100325                   D       -        0xe000000010bbaf00 [zio_free_issue_7_8]
100324                   D       -        0xe000000010bbaf00 [zio_free_issue_7_7]
100323                   D       -        0xe000000010bbaf00 [zio_free_issue_7_6]
100322                   D       -        0xe000000010bbaf00 [zio_free_issue_7_5]
100321                   D       -        0xe000000010bbaf00 [zio_free_issue_7_4]
100320                   D       -        0xe000000010bbaf00 [zio_free_issue_7_3]
100319                   D       -        0xe000000010bbaf00 [zio_free_issue_7_2]
100318                   D       -        0xe000000010bbaf00 [zio_free_issue_7_1]
100317                   D       -        0xe000000010bbaf00 [zio_free_issue_7_0]
100316                   D       -        0xe000000010bbae00 [zio_free_issue_6_11]
100315                   D       -        0xe000000010bbae00 [zio_free_issue_6_10]
100314                   D       -        0xe000000010bbae00 [zio_free_issue_6_9]
100313                   D       -        0xe000000010bbae00 [zio_free_issue_6_8]
100312                   D       -        0xe000000010bbae00 [zio_free_issue_6_7]
100311                   D       -        0xe000000010bbae00 [zio_free_issue_6_6]
100310                   D       -        0xe000000010bbae00 [zio_free_issue_6_5]
100309                   D       -        0xe000000010bbae00 [zio_free_issue_6_4]
100308                   D       -        0xe000000010bbae00 [zio_free_issue_6_3]
100307                   D       -        0xe000000010bbae00 [zio_free_issue_6_2]
100306                   D       -        0xe000000010bbae00 [zio_free_issue_6_1]
100305                   D       -        0xe000000010bbae00 [zio_free_issue_6_0]
100304                   D       -        0xe000000010bbad00 [zio_free_issue_5_11]
100303                   D       -        0xe000000010bbad00 [zio_free_issue_5_10]
100302                   D       -        0xe000000010bbad00 [zio_free_issue_5_9]
100301                   D       -        0xe000000010bbad00 [zio_free_issue_5_8]
100300                   D       -        0xe000000010bbad00 [zio_free_issue_5_7]
100299                   D       -        0xe000000010bbad00 [zio_free_issue_5_6]
100298                   D       -        0xe000000010bbad00 [zio_free_issue_5_5]
100297                   D       -        0xe000000010bbad00 [zio_free_issue_5_4]
100296                   D       -        0xe000000010bbad00 [zio_free_issue_5_3]
100295                   D       -        0xe000000010bbad00 [zio_free_issue_5_2]
100294                   D       -        0xe000000010bbad00 [zio_free_issue_5_1]
100293                   D       -        0xe000000010bbad00 [zio_free_issue_5_0]
100292                   D       -        0xe000000010bbac00 [zio_free_issue_4_11]
100291                   D       -        0xe000000010bbac00 [zio_free_issue_4_10]
100290                   D       -        0xe000000010bbac00 [zio_free_issue_4_9]
100289                   D       -        0xe000000010bbac00 [zio_free_issue_4_8]
100288                   D       -        0xe000000010bbac00 [zio_free_issue_4_7]
100287                   D       -        0xe000000010bbac00 [zio_free_issue_4_6]
100286                   D       -        0xe000000010bbac00 [zio_free_issue_4_5]
100285                   D       -        0xe000000010bbac00 [zio_free_issue_4_4]
100284                   D       -        0xe000000010bbac00 [zio_free_issue_4_3]
100283                   D       -        0xe000000010bbac00 [zio_free_issue_4_2]
100282                   D       -        0xe000000010bbac00 [zio_free_issue_4_1]
100281                   D       -        0xe000000010bbac00 [zio_free_issue_4_0]
100280                   D       -        0xe000000010bbab00 [zio_free_issue_3_11]
100279                   D       -        0xe000000010bbab00 [zio_free_issue_3_10]
100278                   D       -        0xe000000010bbab00 [zio_free_issue_3_9]
100277                   D       -        0xe000000010bbab00 [zio_free_issue_3_8]
100276                   D       -        0xe000000010bbab00 [zio_free_issue_3_7]
100275                   D       -        0xe000000010bbab00 [zio_free_issue_3_6]
100274                   D       -        0xe000000010bbab00 [zio_free_issue_3_5]
100273                   D       -        0xe000000010bbab00 [zio_free_issue_3_4]
100272                   D       -        0xe000000010bbab00 [zio_free_issue_3_3]
100271                   D       -        0xe000000010bbab00 [zio_free_issue_3_2]
100270                   D       -        0xe000000010bbab00 [zio_free_issue_3_1]
100269                   D       -        0xe000000010bbab00 [zio_free_issue_3_0]
100268                   D       -        0xe000000010bbaa00 [zio_free_issue_2_11]
100267                   D       -        0xe000000010bbaa00 [zio_free_issue_2_10]
100266                   D       -        0xe000000010bbaa00 [zio_free_issue_2_9]
100265                   D       -        0xe000000010bbaa00 [zio_free_issue_2_8]
100264                   D       -        0xe000000010bbaa00 [zio_free_issue_2_7]
100263                   D       -        0xe000000010bbaa00 [zio_free_issue_2_6]
100262                   D       -        0xe000000010bbaa00 [zio_free_issue_2_5]
100261                   D       -        0xe000000010bbaa00 [zio_free_issue_2_4]
100260                   D       -        0xe000000010bbaa00 [zio_free_issue_2_3]
100259                   D       -        0xe000000010bbaa00 [zio_free_issue_2_2]
100258                   D       -        0xe000000010bbaa00 [zio_free_issue_2_1]
100257                   D       -        0xe000000010bbaa00 [zio_free_issue_2_0]
100256                   D       -        0xe000000010bba900 [zio_free_issue_1_11]
100255                   D       -        0xe000000010bba900 [zio_free_issue_1_10]
100254                   D       -        0xe000000010bba900 [zio_free_issue_1_9]
100253                   D       -        0xe000000010bba900 [zio_free_issue_1_8]
100252                   D       -        0xe000000010bba900 [zio_free_issue_1_7]
100251                   D       -        0xe000000010bba900 [zio_free_issue_1_6]
100250                   D       -        0xe000000010bba900 [zio_free_issue_1_5]
100249                   D       -        0xe000000010bba900 [zio_free_issue_1_4]
100248                   D       -        0xe000000010bba900 [zio_free_issue_1_3]
100247                   D       -        0xe000000010bba900 [zio_free_issue_1_2]
100246                   D       -        0xe000000010bba900 [zio_free_issue_1_1]
100245                   D       -        0xe000000010bba900 [zio_free_issue_1_0]
100244                   D       -        0xe000000010bba800 [zio_free_issue_0_11]
100243                   D       -        0xe000000010bba800 [zio_free_issue_0_10]
100242                   D       -        0xe000000010bba800 [zio_free_issue_0_9]
100241                   D       -        0xe000000010bba800 [zio_free_issue_0_8]
100240                   D       -        0xe000000010bba800 [zio_free_issue_0_7]
100239                   D       -        0xe000000010bba800 [zio_free_issue_0_6]
100238                   D       -        0xe000000010bba800 [zio_free_issue_0_5]
100237                   D       -        0xe000000010bba800 [zio_free_issue_0_4]
100236                   D       -        0xe000000010bba800 [zio_free_issue_0_3]
100235                   D       -        0xe000000010bba800 [zio_free_issue_0_2]
100234                   D       -        0xe000000010bba800 [zio_free_issue_0_1]
100233                   D       -        0xe000000010bba800 [zio_free_issue_0_0]
100232                   D       -        0xe000000010bba700 [zio_write_intr_high]
100231                   D       -        0xe000000010bba700 [zio_write_intr_high]
100230                   D       -        0xe000000010bba700 [zio_write_intr_high]
100229                   D       -        0xe000000010bba700 [zio_write_intr_high]
100228                   D       -        0xe000000010bba700 [zio_write_intr_high]
100227                   D       -        0xe000000010bba600 [zio_write_intr_7]
100226                   D       -        0xe000000010bba600 [zio_write_intr_6]
100225                   D       -        0xe000000010bba600 [zio_write_intr_5]
100224                   D       -        0xe000000010bba600 [zio_write_intr_4]
100223                   D       -        0xe000000010bba600 [zio_write_intr_3]
100222                   D       -        0xe000000010bba600 [zio_write_intr_2]
100221                   D       -        0xe000000010bba600 [zio_write_intr_1]
100220                   D       -        0xe000000010bba600 [zio_write_intr_0]
100219                   D       -        0xe000000010bba500 [zio_write_issue_hig]
100218                   D       -        0xe000000010bba500 [zio_write_issue_hig]
100217                   D       -        0xe000000010bba500 [zio_write_issue_hig]
100216                   D       -        0xe000000010bba500 [zio_write_issue_hig]
100215                   D       -        0xe000000010bba500 [zio_write_issue_hig]
100214                   D       -        0xe000000010bba400 [zio_write_issue]
100213                   D       -        0xe000000010bba300 [zio_read_intr]
100212                   D       -        0xe000000010bba200 [zio_read_issue_7]
100211                   D       -        0xe000000010bba200 [zio_read_issue_6]
100210                   D       -        0xe000000010bba200 [zio_read_issue_5]
100209                   D       -        0xe000000010bba200 [zio_read_issue_4]
100208                   D       -        0xe000000010bba200 [zio_read_issue_3]
100207                   D       -        0xe000000010bba200 [zio_read_issue_2]
100206                   D       -        0xe000000010bba200 [zio_read_issue_1]
100205                   D       -        0xe000000010bba200 [zio_read_issue_0]
100204                   D       -        0xe000000010bba100 [zio_null_intr]
100203                   D       -        0xe000000010bba000 [zio_null_issue]
100201                   D       -        0xe000000010bb9d00 [metaslab_group_task]
100200                   D       -        0xe000000010bb9e00 [metaslab_group_task]
100066                   D       -        0xe000000012265b00 [system_taskq_1]
100065                   D       -        0xe000000012265b00 [system_taskq_0]
100050                   D       -        0xe000000010bbbb00 [CAM taskq]
100046                   D       -        0x9ffc000000c2e7d1 [deadlkres]
100022                   D       -        0xe000000010bbb500 [thread taskq]
100020                   D       -        0xe000000010bbb700 [kqueue taskq]
100019                   D       -        0xe000000010bbb800 [ffs_trim taskq]
100018                   D       -        0xe000000010bbb900 [acpi_task_2]
100017                   D       -        0xe000000010bbb900 [acpi_task_1]
100016                   D       -        0xe000000010bbb900 [acpi_task_0]
100012                   D       -        0xe000000010bbbd00 [firmware taskq]
100000                   D       swapin   0x9ffc000000c22388 [swapper]
93040 93024 14849     0  ZJ                                  cmake
db:0:ps>  alltrace

Tracing command sh pid 93043 tid 100404 td 0xe0000040ba77f680
fork_trampoline() at fork_trampoline

Tracing command awk pid 93042 tid 100445 td 0xe0000040f5950480
cpu_switch(0xe0000040f5950480, 0xe0000040ba762480, 0x9ffc000000c7fae0, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000040f5950480, 0xe0000040ba762480, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000040f5950480, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040f31cd0d8, 0x5c, 0xe0000040f59507f8, 0xe0000040f5950480) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040f31cd0d8, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000040f31cd0d8, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe0000040f31cd0d8, 0xe0000040f31cd318, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe000000012330c04, 0xa0000000906f9360, 0xa0000000906f9378, 0x0, 0xe0000040f31cd1e0) at pipe_read+0x740
dofileread(0xe0000040f5950480, 0x0, 0xe000000012330be0, 0xa0000000906f9360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe0000040f5950480, 0x0, 0xa0000000906f9360) at kern_readv+0xa0
sys_read(0xe0000040f5950480, 0xa0000000906f94e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
syscall(0xe0000040f58a64a0, 0x120c4c000, 0x2000, 0xe0000040f5950480, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command awk pid 93041 tid 100397 td 0xe0000040ba762480
ia64_ih_stop(0x9ffc00000096cb40, 0x40b) at ia64_ih_stop+0x80
ia64_handle_intr(0xa000000090591400) at ia64_handle_intr+0x110
ivt_External_Interrupt() at ivt_External_Interrupt+0x30

Tracing command automoc4 pid 93024 tid 100480 td 0xe000000109938000
kern_select(0xe000000109938000, 0x0, 0x7fffffffffbfdf28, 0x0, 0x0, 0xffffffffffffffff, 0xa) at kern_select+0x7a0
sys_select(0xe000000109938000, 0xa0000000907514e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0
syscall(0xe000000012f66940, 0x7fffffffffbfdf28, 0x0, 0xe000000109938000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command automoc4 pid 93024 tid 100382 td 0xe000000012f6e480
cpu_switch(0xe000000012f6e480, 0xe0000040ba8e9b00, 0x9ffc000000c7fea0, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012f6e480, 0xe0000040ba8e9b00, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012f6e480, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000000123de0c0, 0x0, 0xe000000012f6e7f8, 0xe000000012f6e480) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000123de0c0, 0x0, 0x9ffc000000b008c0, 0x9ffc00000042c640) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000000123de0c0, 0x0, 0x9ffc0000003b87b0, 0x611, 0x9ffc0000003b87f0) at sleepq_wait_sig+0x30
_cv_wait_sig(0xe0000000123de0c0, 0xe0000000123de0a0, 0xe000000012f6e480, 0x0, 0x9ffc000000c24d70) at _cv_wait_sig+0x4d0
seltdwait(0xe0000000123de080, 0xffffffffffffffff, 0x0, 0xe0000000123de0b8, 0x9ffc000000b04010, 0x9ffc0000005089a0) at seltdwait+0x110
kern_select(0xe000000012f6e480, 0x0, 0x7fffffffffffdc48, 0x7fffffffffffdcc8, 0x0, 0xffffffffffffffff, 0x18) at kern_select+0xb40
sys_select(0xe000000012f6e480, 0xa0000000905194e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0
syscall(0xe000000012f66940, 0x7fffffffffffdc48, 0x7fffffffffffdcc8, 0xe000000012f6e480, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command make pid 93023 tid 100451 td 0xe0000040f5876900
cpu_switch(0xe0000040f5876900, 0xe0000040ba762900, 0x9ffc000000c7fb08, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000040f5876900, 0xe0000040ba762900, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000040f5876900, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe000000109914940, 0x6c, 0xe0000040f5876c78, 0xe0000040f5876900) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000109914940, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000109914940, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe000000109914940, 0xe000000109914a40, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe0000040f5876900, 0x7, 0x0, 0xa000000090721390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe0000040f5876900, 0xffffffff, 0xa000000090721390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe0000040f5876900, 0xa0000000907214e8, 0xa000000090721390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe000000109914940, 0x7fffffffffffd254, 0x0, 0xe0000040f5876900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command qmake-qt4 pid 93007 tid 100399 td 0xe000000013261b00
cpu_switch(0xe000000013261b00, 0xe0000040ba77ed80, 0x9ffc000000c4a980, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000013261b00, 0xe0000040ba77ed80, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x207, 0x0, 0xe000000013261b00, 0x9ffc0000004ea880, 0x50f) at mi_switch+0x3f0
ast(0xa0000000905a1400, 0x9ffc000000b01640, 0x9ffc000000b01640) at ast+0x480
ia64_handle_intr(0xa0000000905a1400) at ia64_handle_intr+0x230
ivt_External_Interrupt() at ivt_External_Interrupt+0x30

Tracing command as pid 92828 tid 100411 td 0xe0000040ba763680
cpu_switch(0xe0000040ba763680, 0xe0000040ba762480, 0x9ffc000000c80300, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000040ba763680, 0xe0000040ba762480, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000040ba763680, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000000130edce0, 0x5c, 0xe0000040ba7639f8, 0xe0000040ba763680) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000130edce0, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000000130edce0, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe0000000130edce0, 0xe0000000130edf20, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe0000000123308e4, 0xa000000090601360, 0xa000000090601378, 0x0, 0xe0000000130edde8) at pipe_read+0x740
dofileread(0xe0000040ba763680, 0x0, 0xe0000000123308c0, 0xa000000090601360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe0000040ba763680, 0x0, 0xa000000090601360) at kern_readv+0xa0
sys_read(0xe0000040ba763680, 0xa0000000906014e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
syscall(0xe0000040ba761280, 0x1208e0000, 0x2000, 0xe0000040ba763680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command cc1plus pid 92825 tid 100396 td 0xe0000040ba762900
cpu_switch(0xe0000040ba762900, 0xe0000040ba762480, 0x9ffc000000c49c80, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000040ba762900, 0xe0000040ba762480, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x207, 0x0, 0xe0000040ba762900, 0x9ffc0000004ea880, 0x50f) at mi_switch+0x3f0
ast(0xa000000090589400, 0x9ffc000000b01640, 0x9ffc000000b01640) at ast+0x480
ia64_handle_intr(0xa000000090589400) at ia64_handle_intr+0x230
ivt_External_Interrupt() at ivt_External_Interrupt+0x30

Tracing command c++ pid 92824 tid 100375 td 0xe000000012df7200
cpu_switch(0xe000000012df7200, 0xe0000040ba762480, 0x9ffc000000c805a8, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012df7200, 0xe0000040ba762480, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012df7200, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe000000012dfade0, 0x6c, 0xe000000012df7578, 0xe000000012df7200) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012dfade0, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012dfade0, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe000000012dfade0, 0xe000000012dfaee0, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe000000012df7200, 0x0, 0x16a99, 0xa0000000904e1390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe000000012df7200, 0x16a99, 0xa0000000904e1390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe000000012df7200, 0xa0000000904e14e8, 0xa0000000904e1390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe000000012dfade0, 0x1208060f0, 0x0, 0xe000000012df7200, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 92776 tid 100359 td 0xe000000012df6900
cpu_switch(0xe000000012df6900, 0xe000000010b95680, 0x9ffc000000c7fa40, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012df6900, 0xe000000010b95680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012df6900, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe000000012dfa4a0, 0x6c, 0xe000000012df6c78, 0xe000000012df6900) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012dfa4a0, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012dfa4a0, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe000000012dfa4a0, 0xe000000012dfa5a0, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe000000012df6900, 0x7, 0x0, 0xa000000090461390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe000000012df6900, 0xffffffff, 0xa000000090461390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe000000012df6900, 0xa0000000904614e8, 0xa000000090461390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe000000012dfa4a0, 0x7fffffffffffaf20, 0x0, 0xe000000012df6900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command make pid 90353 tid 100378 td 0xe000000012f6f680
cpu_switch(0xe000000012f6f680, 0xe0000040ba762900, 0x9ffc000000c80da0, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012f6f680, 0xe0000040ba762900, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012f6f680, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe000000012f68000, 0x6c, 0xe000000012f6f9f8, 0xe000000012f6f680) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012f68000, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012f68000, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe000000012f68000, 0xe000000012f68100, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe000000012f6f680, 0x7, 0x0, 0xa0000000904f9390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe000000012f6f680, 0xffffffff, 0xa0000000904f9390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe000000012f6f680, 0xa0000000904f94e8, 0xa0000000904f9390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe000000012f68000, 0x7fffffffffffd2d4, 0x0, 0xe000000012f6f680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command make pid 90339 tid 100383 td 0xe000000012f6e000
cpu_switch(0xe000000012f6e000, 0xe0000040ba77f200, 0x9ffc000000c81840, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012f6e000, 0xe0000040ba77f200, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012f6e000, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe000000012f664a0, 0x6c, 0xe000000012f6e378, 0xe000000012f6e000) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012f664a0, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012f664a0, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe000000012f664a0, 0xe000000012f665a0, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe000000012f6e000, 0x7, 0x0, 0xa000000090521390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe000000012f6e000, 0xffffffff, 0xa000000090521390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe000000012f6e000, 0xa0000000905214e8, 0xa000000090521390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe000000012f664a0, 0x7fffffffffffd2f4, 0x0, 0xe000000012f6e000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 90337 tid 100358 td 0xe0000000125d5b00
cpu_switch(0xe0000000125d5b00, 0xe0000000121da900, 0x9ffc000000c81e08, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000000125d5b00, 0xe0000000121da900, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000125d5b00, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe000000012dfa940, 0x6c, 0xe0000000125d5e78, 0xe0000000125d5b00) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012dfa940, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012dfa940, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe000000012dfa940, 0xe000000012dfaa40, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe0000000125d5b00, 0x7, 0x0, 0xa000000090459390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe0000000125d5b00, 0xffffffff, 0xa000000090459390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe0000000125d5b00, 0xa0000000904594e8, 0xa000000090459390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe000000012dfa940, 0x7fffffffffffe0d0, 0x0, 0xe0000000125d5b00, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command make pid 89415 tid 100417 td 0xe0000040ba8e6d80
cpu_switch(0xe0000040ba8e6d80, 0xe0000040ba77f200, 0x9ffc000000c80b70, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000040ba8e6d80, 0xe0000040ba77f200, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000040ba8e6d80, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040ba8df280, 0x6c, 0xe0000040ba8e70f8, 0xe0000040ba8e6d80) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040ba8df280, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000040ba8df280, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe0000040ba8df280, 0xe0000040ba8df380, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe0000040ba8e6d80, 0x7, 0x0, 0xa000000090631390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe0000040ba8e6d80, 0xffffffff, 0xa000000090631390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe0000040ba8e6d80, 0xa0000000906314e8, 0xa000000090631390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe0000040ba8df280, 0x7fffffffffffd6d4, 0x0, 0xe0000040ba8e6d80, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 89413 tid 100436 td 0xe0000040f5877b00
cpu_switch(0xe0000040f5877b00, 0xe000000012ba0000, 0x9ffc000000c81070, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000040f5877b00, 0xe000000012ba0000, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000040f5877b00, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040ba8e1280, 0x6c, 0xe0000040f5877e78, 0xe0000040f5877b00) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040ba8e1280, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000040ba8e1280, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe0000040ba8e1280, 0xe0000040ba8e1380, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe0000040f5877b00, 0x7, 0x0, 0xa0000000906b1390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe0000040f5877b00, 0xffffffff, 0xa0000000906b1390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe0000040f5877b00, 0xa0000000906b14e8, 0xa0000000906b1390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe0000040ba8e1280, 0x7fffffffffffca80, 0x0, 0xe0000040f5877b00, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 89041 tid 100388 td 0xe0000000132e3b00
cpu_switch(0xe0000000132e3b00, 0xe000000013261b00, 0x9ffc000000c80508, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000000132e3b00, 0xe000000013261b00, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000132e3b00, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe000000013290940, 0x6c, 0xe0000000132e3e78, 0xe0000000132e3b00) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000013290940, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000013290940, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe000000013290940, 0xe000000013290a40, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe0000000132e3b00, 0x7, 0x0, 0xa000000090549390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe0000000132e3b00, 0xffffffff, 0xa000000090549390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe0000000132e3b00, 0xa0000000905494e8, 0xa000000090549390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe000000013290940, 0x7fffffffffffdd80, 0x0, 0xe0000000132e3b00, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 89038 tid 100437 td 0xe0000040f5877680
cpu_switch(0xe0000040f5877680, 0xe0000000132e3b00, 0x9ffc000000c81ea8, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000040f5877680, 0xe0000000132e3b00, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000040f5877680, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040ba8e0de0, 0x6c, 0xe0000040f58779f8, 0xe0000040f5877680) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040ba8e0de0, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000040ba8e0de0, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe0000040ba8e0de0, 0xe0000040ba8e0ee0, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe0000040f5877680, 0x7, 0x0, 0xa0000000906b9390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe0000040f5877680, 0xffffffff, 0xa0000000906b9390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe0000040f5877680, 0xa0000000906b94e8, 0xa0000000906b9390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe0000040ba8e0de0, 0x7fffffffffffdc10, 0x0, 0xe0000040f5877680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command make pid 88992 tid 100409 td 0xe0000040ba77e000
cpu_switch(0xe0000040ba77e000, 0xe0000040f5877680, 0x9ffc000000c817a0, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000040ba77e000, 0xe0000040f5877680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000040ba77e000, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040ba77c000, 0x6c, 0xe0000040ba77e378, 0xe0000040ba77e000) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040ba77c000, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000040ba77c000, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe0000040ba77c000, 0xe0000040ba77c100, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe0000040ba77e000, 0x7, 0x0, 0xa0000000905f1390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe0000040ba77e000, 0xffffffff, 0xa0000000905f1390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe0000040ba77e000, 0xa0000000905f14e8, 0xa0000000905f1390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe0000040ba77c000, 0x7fffffffffffd6d4, 0x0, 0xe0000040ba77e000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 88989 tid 100415 td 0xe0000040ba8e7680
cpu_switch(0xe0000040ba8e7680, 0xe0000040ba77e000, 0x9ffc000000c7f9a0, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000040ba8e7680, 0xe0000040ba77e000, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000040ba8e7680, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040ba8e0000, 0x6c, 0xe0000040ba8e79f8, 0xe0000040ba8e7680) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040ba8e0000, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000040ba8e0000, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe0000040ba8e0000, 0xe0000040ba8e0100, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe0000040ba8e7680, 0x7, 0x0, 0xa000000090621390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe0000040ba8e7680, 0xffffffff, 0xa000000090621390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe0000040ba8e7680, 0xa0000000906214e8, 0xa000000090621390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe0000040ba8e0000, 0x7fffffffffffca80, 0x0, 0xe0000040ba8e7680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command tee pid 80044 tid 100361 td 0xe000000012df6000
cpu_switch(0xe000000012df6000, 0xe0000040ba762480, 0x9ffc000000c81098, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000012df6000, 0xe0000040ba762480, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012df6000, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040f31cd340, 0x5c, 0xe000000012df6378, 0xe000000012df6000) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040f31cd340, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000040f31cd340, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe0000040f31cd340, 0xe0000040f31cd580, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe00000010983ef74, 0xa000000090471360, 0xa000000090471378, 0x0, 0xe0000040f31cd448) at pipe_read+0x740
dofileread(0xe000000012df6000, 0x0, 0xe00000010983ef50, 0xa000000090471360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe000000012df6000, 0x0, 0xa000000090471360) at kern_readv+0xa0
sys_read(0xe000000012df6000, 0xa0000000904714e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
syscall(0xe000000012bcf720, 0x120804000, 0x2000, 0xe000000012df6000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command tee pid 79991 tid 100402 td 0xe000000013260d80
cpu_switch(0xe000000013260d80, 0xe000000013261b00, 0x9ffc000000c80800, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe000000013260d80, 0xe000000013261b00, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000013260d80, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040ba7abce0, 0x5c, 0xe0000000132610f8, 0xe000000013260d80) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040ba7abce0, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000040ba7abce0, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
_sleep(0xe0000040ba7abce0, 0xe0000040ba7abf20, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe0000000138fd3d4, 0xa0000000905b9360, 0xa0000000905b9378, 0x0, 0xe0000040ba7abde8) at pipe_read+0x740
dofileread(0xe000000013260d80, 0x0, 0xe0000000138fd3b0, 0xa0000000905b9360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe000000013260d80, 0x0, 0xa0000000905b9360) at kern_readv+0xa0
sys_read(0xe000000013260d80, 0xa0000000905b94e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
syscall(0xe0000040ba75a4a0, 0x120804000, 0x2000, 0xe000000013260d80, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 79742 tid 100398 td 0xe0000040ba762000
cpu_switch(0xe0000040ba762000, 0xe000000012ba0000, 0x9ffc000000c80490, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000040ba762000, 0xe000000012ba0000, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000040ba762000, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040ba9786c0, 0x0, 0xe0000040ba762378, 0xe0000040ba762000) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040ba9786c0, 0x0, 0x9ffc000000b008c0, 0xe0000040ba9786a0) at sleepq_catch_signals+0x190
sleepq_timedwait_sig(0xe0000040ba9786c0, 0x0, 0x9ffc0000003b79b0, 0x9ffc0000003b79f0, 0x713) at sleepq_timedwait_sig+0x30
_cv_timedwait_sig_sbt(0xe0000040ba9786c0, 0xe0000040ba9786a0, 0x115a34cb79459, 0x2d00000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0
seltdwait(0xe0000040ba978680, 0x115a34cb79459, 0x2d00000000, 0xe0000040ba9786b8, 0x9ffc000000b04010, 0x9ffc0000005089a0) at seltdwait+0xf0
kern_select(0xe0000040ba762000, 0x0, 0x7fffffffffffcdd0, 0x0, 0x0, 0x115a34cb79459, 0x1) at kern_select+0xb40
sys_select(0xe0000040ba762000, 0xa0000000905994e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0
syscall(0xe0000040ba75b720, 0x7fffffffffffcdd0, 0x0, 0xe0000040ba762000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 79716 tid 100408 td 0xe0000040ba77e480
cpu_switch(0xe0000040ba77e480, 0xe000000012ba0000, 0x9ffc000000c81b60, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000040ba77e480, 0xe000000012ba0000, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000040ba77e480, 0x9ffc0000004e3290) at mi_switch+0x3f0
sleepq_switch(0xe0000040ba979840, 0x0, 0xe0000040ba77e7f8, 0xe0000040ba77e480) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000040ba979840, 0x0, 0x9ffc000000b008c0, 0xe0000040ba979820) at sleepq_catch_signals+0x190
sleepq_timedwait_sig(0xe0000040ba979840, 0x0, 0x9ffc0000003b79b0, 0x9ffc0000003b79f0, 0x713) at sleepq_timedwait_sig+0x30
_cv_timedwait_sig_sbt(0xe0000040ba979840, 0xe0000040ba979820, 0x115996bf2f4f1, 0x2d00000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0
seltdwait(0xe0000040ba979800, 0x115996bf2f4f1, 0x2d00000000, 0xe0000040ba979838, 0x9ffc000000b04010, 0x9ffc0000005089a0) at seltdwait+0xf0
kern_select(0xe0000040ba77e480, 0x0, 0x7fffffffffffcdd0, 0x0, 0x0, 0x115996bf2f4f1, 0x1) at kern_select+0xb40
sys_select(0xe0000040ba77e480, 0xa0000000905e94e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0
syscall(0xe0000040ba77c4a0, 0x7fffffffffffcdd0, 0x0, 0xe0000040ba77e480, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command gmake pid 60325 tid 100412 td 0xe0000040ba763200
cpu_switch(0xe0000040ba763200, 0xe0000040ba8e9200, 0x9ffc000000c81ea8, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
sched_switch(0xe0000040ba763200, 0xe0000040ba8e9200, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000040ba763200, 0x9ffc0000004e329
Comment 1 Anton Shterenlikht 2014-03-21 14:06:36 UTC
Just noticed that there is no msgbuf.txt file
created. I got the same trap again, so pasting
the actual trap message here:

db> show msgbuf    
msgbufp = 0xe0000040fffbdfb8
magic = 63062, size = 98232, r= 14587, w = 14922, ptr = 0xe0000040fffa6000, cksum= 1131957

fatal kernel trap (cpu 0):

    trap vector = 0x14 (Page Not Present)
    cr.iip      = 0x9ffc000000508600
    cr.ipsr     = 0x10080a6018 (ac,mfl,ic,i,dt,dfh,rt,cpl=0,it,ri=0)
    cr.isr      = 0x400000000 (code=0,vector=0,r,ei=0)
    cr.ifa      = 0x2b00669
    curthread   = 0xe0000040708a7b00
        pid = 63114, comm = automoc4
Comment 2 Enji Cooper freebsd_committer freebsd_triage 2015-11-10 13:38:18 UTC
marcel slayed ia64 back in July 2014:

------------------------------------------------------------------------
r268191 | marcel | 2014-07-02 15:34:06 -0700 (Wed, 02 Jul 2014) | 11 lines

Remove ia64 from the list of known architectures and add an entry to
UPDATING. This is the first step towards the removal of ia64 from
head. A buildworld for ia64 will now yield:

% make buildworld
make[1]: "/usr/src/Makefile.inc1" line 151: Unknown target ia64:ia64.

While here, trim the ia64-specific additions from ObsoleteFiles.inc

Discussed at: BSDcan