Bug 14141

More news on this one.  We moved to new hardware, and now I'm getting
multiple crashes with the following signature:

The common things here seem to be the vfork() --> fork1() --> fdcopy()

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0xd
fault code              = supervisor write, page not present
instruction pointer     = 0x8:0xc01435d7
stack pointer           = 0x10:0xcc348f08
frame pointer           = 0x10:0xcc348f2c
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 38090 (httpd_1.3.9)
interrupt mask          = 
trap number             = 12
panic: page fault

(kgdb) bt
#0  boot (howto=256) at ../../kern/kern_shutdown.c:285
#1  0xc014aad1 in panic (fmt=0xc02385ba "page fault")
    at ../../kern/kern_shutdown.c:446
#2  0xc0209712 in trap_fatal (frame=0xcc348ecc, eva=13)
    at ../../i386/i386/trap.c:942
#3  0xc02093cb in trap_pfault (frame=0xcc348ecc, usermode=0, eva=13)
    at ../../i386/i386/trap.c:835
#4  0xc0208ffe in trap (frame={tf_es = 16, tf_ds = 16, tf_edi = -1044124544, 
      tf_esi = -1037650432, tf_ebp = -868970708, tf_isp = -868970764, 
      tf_ebx = 1769, tf_edx = -1052106376, tf_ecx = 0, tf_eax = 1, 
      tf_trapno = 12, tf_err = 2, tf_eip = -1072417321, tf_cs = 8, 
      tf_eflags = 66050, tf_esp = -1052187648, tf_ss = -1052089344})
    at ../../i386/i386/trap.c:437
#5  0xc01435d7 in fdcopy (p=0xcc3079c0) at ../../kern/kern_descrip.c:951
#6  0xc014587b in fork1 (p1=0xcc3079c0, flags=-2147483596)
    at ../../kern/kern_fork.c:379
#7  0xc014533b in vfork (p=0xcc3079c0, uap=0xcc348f94)
    at ../../kern/kern_fork.c:109
#8  0xc020995b in syscall (frame={tf_es = 39, tf_ds = 39, tf_edi = 226087504, 
      tf_esi = 226086800, tf_ebp = -1077949752, tf_isp = -868970524, 
      tf_ebx = 673171048, tf_edx = 226066448, tf_ecx = 672877149, tf_eax = 66, 
      tf_trapno = 7, tf_err = 2, tf_eip = 672936705, tf_cs = 31, 
      tf_eflags = 514, tf_esp = -1077949796, tf_ss = 39})
    at ../../i386/i386/trap.c:1100
#9  0xc01feedc in Xint0x80_syscall ()

This is very worrysome.  What we seem to have here is memory
corruption of some sort, I think (but years of sysadm has dulled by
gdb usage)...

From a simlar, but hour later crash dump:

(kgdb) bt
#0  boot (howto=256) at ../../kern/kern_shutdown.c:285
#1  0xc014aad1 in panic (fmt=0xc02385ba "page fault")
    at ../../kern/kern_shutdown.c:446
#2  0xc0209712 in trap_fatal (frame=0xcc332ecc, eva=43)
    at ../../i386/i386/trap.c:942
#3  0xc02093cb in trap_pfault (frame=0xcc332ecc, usermode=0, eva=43)
    at ../../i386/i386/trap.c:835
#4  0xc0208ffe in trap (frame={tf_es = 16, tf_ds = 16, tf_edi = -1051776000, 
      tf_esi = -1051100288, tf_ebp = -869060820, tf_isp = -869060876, 
      tf_ebx = 1863, tf_edx = -1050021888, tf_ecx = 0, tf_eax = 31, 
      tf_trapno = 12, tf_err = 2, tf_eip = -1072417321, tf_cs = 8, 
      tf_eflags = 66050, tf_esp = -1051614208, tf_ss = -1050004480})
    at ../../i386/i386/trap.c:437
#5  0xc01435d7 in fdcopy (p=0xcc304f40) at ../../kern/kern_descrip.c:951
#6  0xc014587b in fork1 (p1=0xcc304f40, flags=-2147483596)
    at ../../kern/kern_fork.c:379
#7  0xc014533b in vfork (p=0xcc304f40, uap=0xcc332f94)
    at ../../kern/kern_fork.c:109
#8  0xc020995b in syscall (frame={tf_es = 39, tf_ds = 39, tf_edi = 226316880, 
      tf_esi = 226066288, tf_ebp = -1077949592, tf_isp = -869060636, 
      tf_ebx = 673171048, tf_edx = 226250768, tf_ecx = 672877149, tf_eax = 66, 
      tf_trapno = 12, tf_err = 2, tf_eip = 672936705, tf_cs = 31, 
      tf_eflags = 518, tf_esp = -1077949636, tf_ss = 39})
    at ../../i386/i386/trap.c:1100
#9  0xc01feedc in Xint0x80_syscall ()

(kgdb) frame 5
#5  0xc01435d7 in fdcopy (p=0xcc304f40) at ../../kern/kern_descrip.c:951
951                             (*fpp)->f_count++;
(kgdb) p fpp
$1 = (struct file **) 0x0
(kgdb) l
946             bcopy(fdp->fd_ofiles, newfdp->fd_ofiles, i * sizeof(struct file **));
947             bcopy(fdp->fd_ofileflags, newfdp->fd_ofileflags, i * sizeof(char));
948             fpp = newfdp->fd_ofiles;
949             for (i = newfdp->fd_lastfile; i-- >= 0; fpp++)
950                     if (*fpp != NULL)
951                             (*fpp)->f_count++;
952             return (newfdp);
953     }
954
955     /*
(kgdb) p newfdp
$2 = (struct filedesc *) 0xc1597b80
(kgdb) p *newfdp
$3 = {fd_ofiles = 0xc169d000, fd_ofileflags = 0xc16a3400 "", 
  fd_cdir = 0xcc3b5bc0, fd_rdir = 0xcc1cfe00, fd_nfiles = 6400, 
  fd_lastfile = 3912, fd_freefile = 6, fd_cmask = 18, fd_refcnt = 1}

now... this implies that fpp somehow gets set to NULL, but:

(kgdb) p **(newfdp->fd_ofiles)
$7 = {f_list = {le_next = 0xc1696fc0, le_prev = 0xc1567b00}, f_flag = 1, 
  f_type = 1, f_count = 41, f_msgcount = 0, f_cred = 0xc0e27200, 
  f_ops = 0xc024a294, f_seqcount = 1, f_nextread = 0, f_offset = 0, 
  f_data = 0xcc1cde80 ""}
(kgdb) p **(newfdp->fd_ofiles+1)
$8 = {f_list = {le_next = 0xc13958c0, le_prev = 0xc1419f40}, f_flag = 2, 
  f_type = 1, f_count = 41, f_msgcount = 0, f_cred = 0xc0e27200, 
  f_ops = 0xc024a294, f_seqcount = 1, f_nextread = 0, f_offset = 0, 
  f_data = 0xcc1cde80 ""}
(kgdb) p **(newfdp->fd_ofiles+2)
$9 = {f_list = {le_next = 0xc13c60c0, le_prev = 0xc139b900}, f_flag = 10, 
  f_type = 1, f_count = 82, f_msgcount = 0, f_cred = 0xc0e27200, 
  f_ops = 0xc024a294, f_seqcount = 1, f_nextread = 0, f_offset = 26016907, 
  f_data = 0xcc27b2c0 ""}

Then I thought about it... and fpp itself should not become NULL in
this case... it's being incremented from some valid pointer value.

Ideas?

Dave.

-- 
============================================================================
|David Gilbert, Velocet Communications.       | Two things can only be     |
|Mail:       dgilbert@velocet.net             |  equal if and only if they |
|http://www.velocet.net/~dgilbert             |   are precisely opposite.  |
=========================================================GLO================

As shown below, the problem has been resolved.

Close the bug.

--
[ Joseph Mallett            <jmallett@xMach.org> ] [ http://srcsys.org ]
[ xMach Core Team         xMach: Proactively Unbloated Microkernel BSD ]
[ FreeBSD, NetBSD, & xMach User; (Obj)C(++) Coder ] [ http://xMach.org ]

On Thu, 24 May 2001, David Gilbert wrote:
> Oi... I'm sure that the issue got solved somehow ... 'cause that was a
> core server (if I'm not mistaken, it's the server that xmach.org is
> on).  That machine is running some concocktion of 3.4-STABLE (IIRC)
> now... not 3.3 as the PR intones.  It may have been the whole tl0
> driver debacle... in which our solution was to swap out the card for
> dc0 drivers.
>
> I don't think I can be more helpful because I don't have the resources
> to attempt repeating the problem.
>
> Dave.
>

Comment 4 Peter Pentchev 2001-05-25 12:48:44 UTC

State Changed
From-To: open->closed

Closed at originator's request.