I should clarify: "We originally hit this on an old 6.1 machine, and
have upgraded through 6.1-p24, 6.2-RELEASE, and finally 6.4-p1"

We:
* did a 6.1-RELEASE install and it panicked as described.
* upgraded it to 6.1-RELEASE-p24 and the panic persisted.
* did a fresh 6.2-RELEASE install, panic again.
* upgraded this to 6.2-RELEASE-p9 (customer version), panic.
* did a fresh 6.4-RELEASE install, panic.
* upgraded this fresh install to 6.4-RELEASE-p1, panic.

The important point being that we didn't upgrade between versions, 6.4
was a fresh install.

Thanks,
Yvan

In trying to replicate this more simply (still using our complex test
scripts unfortunately) I'm seeing some slightly different panics.

I've seen the following one just a couple of times before, but figure it
must be related as it is also under knlist_cleardel.  To my untrained
eye things look to be in an even worse state in this case, should
knl->kl_list.slh_first->kn_kq.kq_lock.mtx_lock ever have a value of
0x06?  On all occurrences of this form of the panic this has value 0x06,
seemingly not random clobbering.

The 'kq' is in state 0x10 - KQ_CLOSING
The 'kn' has status 0x11 - KN_ACTIVE | KN_INFLUX

Notably: 0x78 = 0x04+0x74 - i.e. "mov 0x74(%ecx),%eax"

And: 0x04 = 0x06 & MTX_FLAGMASK (see #define mtx_owner)

Perhaps: mtx_lock = MTX_UNOWNED | MTX_CONTESTED = MTX_DESTROYED


More details:
-----------------------------------------------------------------------
Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x78
fault code             = supervisor read, page not present
instruction pointer    = 0x20:0xc06dc281
stack pointer          = 0x28:0xd184cb64
frame pointer          = 0x28:0xd184cb68
code segment           = base rx0, limit 0xfffff, type 0x1b
                       = DPL 0, pres 1, def32 1, gran 1
processor eflags       = resume, IOPL = 0
current process        = 24539 (perl)
panic: from debugger
KDB: stack backtrace:
Uptime: 2h42m13s
<SNIP/>
#10 0xc092388a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#11 0xc06dc281 in turnstile_setowner (ts=0xc25124c0, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:456
#12 0xc06dc5de in turnstile_wait (lock=0xc2ec6600, owner=0x4, queue=0) at /usr/src/sys/kern/subr_turnstile.c:661
#13 0xc06b1a5e in _mtx_lock_sleep (m=0xc2ec6600, tid=3272086656, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:579
#14 0xc069c961 in knlist_cleardel (knl=0xc27bdb98, td=0x0, islocked=1, killkn=0) at /usr/src/sys/kern/kern_event.c:1730
#15 0xc06e2597 in pipeclose (cpipe=0xc27bdb28) at /usr/src/sys/kern/sys_pipe.c:1526
#16 0xc06e2216 in pipe_close (fp=0xc30814a0, td=0xc3081480) at /usr/src/sys/kern/sys_pipe.c:1443
#17 0xc06980d8 in fdrop_locked (fp=0xc2fa83a8, td=0xc3081480) at file.h:296
#18 0xc0698001 in fdrop (fp=0xc2fa83a8, td=0xc3081480) at /usr/src/sys/kern/kern_descrip.c:2173
#19 0xc069662f in closef (fp=0xc2fa83a8, td=0xc3081480) at /usr/src/sys/kern/kern_descrip.c:1993
#20 0xc06939c3 in kern_close (td=0xc3081480, fd=5) at /usr/src/sys/kern/kern_descrip.c:1083
#21 0xc06937b4 in close (td=0xc3081480, uap=0xc30814a0) at /usr/src/sys/kern/kern_descrip.c:1035
#22 0xc0937903 in syscall (frame=
      {tf_fs = 59, tf_es = 140116027, tf_ds = -1078001605, tf_edi = 0, tf_esi = 673782480, tf_ebp = -1077942568, tf_isp = -779825820, tf_ebx = 673694016, tf_edx = 0, tf_ecx = 0, tf_eax = 6, tf_trapno = 12, tf_err = 2, tf_eip = 673632627, tf_cs = 51, tf_eflags = 530, tf_esp = -1077942596, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:984
#23 0xc09238df in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#24 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
<SNIP/>
(kgdb) p/x *knl
st = {
    slh_first = 0xc2859770
  }, 
  kl_lock = 0xc069c7ec, 
  kl_unlock = 0xc069c820, 
  kl_locked = 0xc069c85c, 
  kl_lockarg = 0xc27bdc98
}
(kgdb) p/x *knl->kl_list.slh_first
$3 = {
  kn_link = {
    sle_next = 0x0
  }, 
  kn_selnext = {
    sle_next = 0x0
  }, 
  kn_knlist = 0xc27bdb98, 
  kn_tqe = {
    tqe_next = 0x0, 
    tqe_prev = 0xc285a848
  }, 
  kn_kq = 0xc2ec6600, 
  kn_kevent = {
    ident = 0x1, 
    filter = 0xfffe, 
    flags = 0x0, 
    fflags = 0x0, 
    data = 0x4000, 
    udata = 0x0
  }, 
  kn_status = 0x11, 
  kn_sfflags = 0x0, 
  kn_sdata = 0x0, 
  kn_ptr = {
    p_fp = 0x0, 
    p_proc = 0x0
  }, 
  kn_fop = 0x0, 
  kn_hook = 0x0
}
(kgdb) p/x *knl->kl_list.slh_first->kn_kq
$4 = {
  kq_lock = {
    mtx_object = {
      lo_class = 0xc0a32c84, 
      lo_name = 0xc09b8585, 
      lo_type = 0xc09b8585, 
      lo_flags = 0x420000, 
      lo_list = {
        tqe_next = 0x0, 
        tqe_prev = 0x0
      }, 
      lo_witness = 0x0
    }, 
    mtx_lock = 0x6,   <<<<======================= ???????
    mtx_recurse = 0x0
  }, 
  kq_refcnt = 0x1, 
  kq_list = {
    sle_next = 0x0
  }, 
  kq_head = {
    tqh_first = 0x0, 
    tqh_last = 0xc2ec662c
  }, 
  kq_count = 0x0, 
  kq_sel = {
    si_thrlist = {
      tqe_next = 0x0, 
      tqe_prev = 0x0
    }, 
    si_thread = 0x0, 
    si_note = {
      kl_list = {
        slh_first = 0x0
      }, 
      kl_lock = 0x0, 
      kl_unlock = 0x0, 
      kl_locked = 0xc069c85c, 
      kl_lockarg = 0x0
    }, 
    si_flags = 0x0
  }, 
  kq_sigio = 0x0, 
  kq_fdp = 0x0, 
  kq_state = 0x10, 
  kq_knlistsize = 0x100, 
  kq_knlist = 0xc268e800, 
  kq_knhashmask = 0x0, 
  kq_knhash = 0x0, 
  kq_task = {
    ta_link = {
      stqe_next = 0x0
    }, 
    ta_pending = 0x0, 
    ta_priority = 0x0, 
    ta_func = 0xc069b788, 
    ta_context = 0xc2ec6600
  }
}
-----------------------------------------------------------------------

Regards,
-Yvan

Comment 3 Eitan Adler 2017-12-31 08:01:33 UTC

For bugs matching the following criteria:

Status: In Progress Changed: (is less than) 2014-06-01

Reset to default assignee and clear in-progress tags.

Mail being skipped

Comment 4 Graham Perrin 2022-10-17 12:17:19 UTC

Keyword: 

    crash

– in lieu of summary line prefix: 

    [panic]

* bulk change for the keyword
* summary lines may be edited manually (not in bulk). 

Keyword descriptions and search interface: 

    <https://bugs.freebsd.org/bugzilla/describekeywords.cgi>