Bug 42260

Summary: 4.6-STABLE kernel panic, fxp related
Product: Base System Reporter: Pawe&#322; Ma&#322;achowski <pawmal>
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: Closed FIXED    
Severity: Affects Only Me    
Priority: Normal    
Version: 4.6-STABLE   
Hardware: Any   
OS: Any   

Description Pawe&#322; Ma&#322;achowski 2002-08-31 20:20:01 UTC 
      After few minutes of heavy networking, kernel or double kernel panic happens.

Hardware:
Soltek DRV5 VIA KT333 motherboard, AMD Athlon XP 1600+ CPU, 256MB DDR RAM, RealTek 8139C and Compaq NC3121 (Intel SB82558B) NICs. Only one NIC (fxp1) is used.

Machine works *without* problems under 4.6.1- (I've noted this problem some time ago and downgraded machine to RELEASE) and 4.6.2-RELEASE. Kernel panic occurs only when I switch to 4.6-STABLE.

I've compiled a GENERIC kernel with debugging option and with make defaults. Here are results:


1st crash:
----------
gdbmods using kernel: /usr/obj/usr/src/sys/GENERICDBG/kernel.debug
gdbmods using core:   vmcore.0
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...
IdlePTD at phsyical address 0x00528000
initial pcb at physical address 0x00461a60
panicstr: page fault
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0xc0ffe782
fault code              = supervisor write, page not present
instruction pointer     = 0x8:0xc0224a77
stack pointer           = 0x10:0xc03fa72c
frame pointer           = 0x10:0xc03fa740
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = Idle
interrupt mask          = net tty
trap number             = 12
panic: page fault

syncing disks...

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x30
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc030c048
stack pointer           = 0x10:0xc03fa54c
frame pointer           = 0x10:0xc03fa554
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = Idle
interrupt mask          = net tty bio cam
trap number             = 12
panic: page fault
Uptime: 3m8s

dumping to dev #ad/0x30001, offset 1572992
dump ata0: resetting devices .. done
255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
---
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487             if (dumping++) {
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1  0xc020901f in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
#2  0xc0209444 in poweroff_wait (junk=0xc03f12ec, howto=-1069609457)
    at /usr/src/sys/kern/kern_shutdown.c:595
#3  0xc0378d0e in trap_fatal (frame=0xc03fa50c, eva=48)
    at /usr/src/sys/i386/i386/trap.c:974
#4  0xc03789e1 in trap_pfault (frame=0xc03fa50c, usermode=0, eva=48)
    at /usr/src/sys/i386/i386/trap.c:867
#5  0xc037859f in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi = 0,
      tf_esi = -1049870592, tf_ebp = -1069570732, tf_isp = -1069570760,
      tf_ebx = -1069354116, tf_edx = 6864960, tf_ecx = -841606208, tf_eax = 0,
      tf_trapno = 12, tf_err = 0, tf_eip = -1070546872, tf_cs = 8,
      tf_eflags = 66054, tf_esp = -1049870592, tf_ss = -1049870592})
    at /usr/src/sys/i386/i386/trap.c:466
#6  0xc030c048 in acquire_lock (lk=0xc042f37c)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:266
#7  0xc0310148 in softdep_update_inodeblock (ip=0xc16c3f00, bp=0xc683bf88,
    waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3813
#8  0xc030b17d in ffs_update (vp=0xcdd61bc0, waitfor=0)
    at /usr/src/sys/ufs/ffs/ffs_inode.c:106
#9  0xc03134ae in ffs_sync (mp=0xc164fc00, waitfor=2, cred=0xc0e3c400,
    p=0xc047ace0) at /usr/src/sys/ufs/ffs/ffs_vfsops.c:1025
#10 0xc0238d9f in sync (p=0xc047ace0, uap=0x0)
    at /usr/src/sys/kern/vfs_syscalls.c:576
#11 0xc0208dba in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
#12 0xc0209444 in poweroff_wait (junk=0xc03f12ec, howto=-1069609457)
    at /usr/src/sys/kern/kern_shutdown.c:595
#13 0xc0378d0e in trap_fatal (frame=0xc03fa6ec, eva=3237996418)
    at /usr/src/sys/i386/i386/trap.c:974
#14 0xc03789e1 in trap_pfault (frame=0xc03fa6ec, usermode=0, eva=3237996418)
    at /usr/src/sys/i386/i386/trap.c:867
#15 0xc037859f in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16,
      tf_edi = 6756538, tf_esi = 1, tf_ebp = -1069570240, tf_isp = -1069570280,
---Type <return> to continue, or q <return> to quit---
      tf_ebx = -1058483712, tf_edx = -1058799616, tf_ecx = -1608507392,
      tf_eax = 1828738, tf_trapno = 12, tf_err = 2, tf_eip = -1071494537,
      tf_cs = 8, tf_eflags = 66070, tf_esp = -1058571520, tf_ss = -1058512894})
    at /usr/src/sys/i386/i386/trap.c:466
#16 0xc0224a77 in m_getcl (how=1, type=1, flags=2)
    at /usr/src/sys/kern/uipc_mbuf.c:589
#17 0xc01ae05f in fxp_add_rfabuf (sc=0xc15bca00, oldm=0xc0e77b00)
    at /usr/src/sys/dev/fxp/if_fxp.c:1863
#18 0xc01acca0 in fxp_intr_body (sc=0xc15bca00, statack=64 '@', count=-1)
    at /usr/src/sys/dev/fxp/if_fxp.c:1323
#19 0xc01acbb5 in fxp_intr (xsc=0xc15bca00) at /usr/src/sys/dev/fxp/if_fxp.c:1224
#20 0xc036fa06 in cpu_idle () at /usr/src/sys/i386/i386/machdep.c:1024
(kgdb) up 6
#6  0xc030c048 in acquire_lock (lk=0xc042f37c)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:266
266             lk->lkt_held = CURPROC->p_pid;
(kgdb) up 10
#16 0xc0224a77 in m_getcl (how=1, type=1, flags=2)
    at /usr/src/sys/kern/uipc_mbuf.c:589
589                     MCLGET(mp, how);

Fix: 

Unknown.

Work-around: downgrade to 4.6.2-RELEASE or use other non-fxp NIC.
How-To-Repeat:       Install FreeBSD 4.6-STABLE on machine as described above and try to use it harder (transferring big files) over Compaq/Intel NIC.
Comment 1 David G. Lawrence 2002-08-31 20:40:51 UTC 
>#15 0xc037859f in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16,
>      tf_edi = 6756538, tf_esi = 1, tf_ebp = -1069570240, tf_isp = -1069570280,
>      tf_ebx = -1058483712, tf_edx = -1058799616, tf_ecx = -1608507392,
>      tf_eax = 1828738, tf_trapno = 12, tf_err = 2, tf_eip = -1071494537,
>      tf_cs = 8, tf_eflags = 66070, tf_esp = -1058571520, tf_ss = -1058512894})
>    at /usr/src/sys/i386/i386/trap.c:466
>#16 0xc0224a77 in m_getcl (how=1, type=1, flags=2)
>    at /usr/src/sys/kern/uipc_mbuf.c:589
>#17 0xc01ae05f in fxp_add_rfabuf (sc=0xc15bca00, oldm=0xc0e77b00)
>    at /usr/src/sys/dev/fxp/if_fxp.c:1863

   Looks like a problem in the new m_getcl() function, which has had some
problems recently, but were thought to be fixed.

-DG

D.G.Lawrence
Download Technologies, Inc. - http://www.downloadtech.com - (866) 399 8500
TeraSolutions, Inc. - http://www.terasolutions.com - (503) 288 9544
The FreeBSD Project - http://www.freebsd.org
Pave the road of life with opportunities.
Comment 2 dwmalone 2002-08-31 20:53:19 UTC 
On Sat, Aug 31, 2002 at 12:17:47PM -0700, Pawel Malachowski wrote:
> #16 0xc0224a77 in m_getcl (how=1, type=1, flags=2)
>     at /usr/src/sys/kern/uipc_mbuf.c:589

This looks like it might be the mcl_pool problem. Can you check what
version of /usr/src/sys/kern/uipc_mbuf.c you have? You might also
like to check and see what "sysctl -a | fgrep mcl_pool_max" outputs.

	David.
Comment 3 dwmalone freebsd_committer freebsd_triage 2002-08-31 20:53:30 UTC 
State Changed
From-To: open->feedback

Waiting for feedback from submitter.
Comment 4 Pawe&#322; Ma&#322;achowski 2002-09-03 14:39:20 UTC 
OK, as Ian Dowse suggested, I've tried to use the `date'
in my cvsup file to find out the latest RELENG_4 stable
for me.

The procedure was:
0) edit the supfile with the proper date tag
   (for example, date=2002.08.11.10.20.00)
1) fetch src-all
2) make clean and remove everyting from /usr/obj
3) uncomment the `makeoptions DEBUG=-g' in GENERIC config file 
4) make kernel
5) reboot


Some of my `checkpoints' were:
----------
2002.07.*		RELENG_4 looks stable

----------
2002.08.05		RELENG_4 looks stable

----------
2002.08.08		RELENG_4 looks stable

----------
2002.08.08.18.45	RELENG_4 looks stable

----------
2002.08.09.00.00	RELENG_4 looks stable

----------
2002.08.09.02.11	cannot compile,
			known and fixed (USB)

----------
2002.08.09.12.00	RELENG_4 is unstable,
			double kernel panic.

#6  0xc02eb420 in acquire_lock (lk=0xc040101c)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:266
266             lk->lkt_held = CURPROC->p_pid;
(kgdb) up 10
#16 0xc0204e20 in m_freem (m=0x0) at /usr/src/sys/kern/uipc_mbuf.c:706
706             if (mcl_pool_now < mcl_pool_max && m->m_next == NULL &&

----------
2002.08.10.12.35	RELENG_4 is unstable,
			kernel panic

I've tried to compile with:
	static int mcl_pool_max = 0;
but kernel happened:
(kgdb) up 6
#6  0xc0207737 in sosend (so=0xcccf1a80, addr=0x0, uio=0xcdf0ced4, top=0x0,
    control=0x0, flags=0, p=0xcc2ff1e0) at /usr/src/sys/kern/uipc_socket.c:567
567                                     MCLGET(m, M_WAIT);

----------
2002.08.11.10.20	RELENG_4 is unstable,
			kernel panic
(kgdb) up 6
#6  0xc0207737 in sosend (so=0xcccf1a80, addr=0x0, uio=0xcdf03ed4, top=0x0,
    control=0x0, flags=0, p=0xcc2ff1e0) at /usr/src/sys/kern/uipc_socket.c:567
567                                     MCLGET(m, M_WAIT);

----------
2002.08.12		RELENG_4 is unstable,
			double kernel panic
(kgdb) up 6
#6  0xc020485f in m_getcl (how=1, type=1, flags=2)
    at /usr/src/sys/kern/uipc_mbuf.c:586
586                     MCLGET(mp, how);
(kgdb) up 16
#22 0xc0207737 in sosend (so=0xcccf3dc0, addr=0x0, uio=0xcdf0ded4, top=0x0,
    control=0x0, flags=0, p=0xcc2ff1e0) at /usr/src/sys/kern/uipc_socket.c:567
567                                     MCLGET(m, M_WAIT);

----------
2002.08.13		RELENG_4 is unstable,
			double kernel panic
(kgdb) up 6
#6  0xc020489b in m_getcl (how=1, type=1, flags=2)
    at /usr/src/sys/kern/uipc_mbuf.c:589
589                     MCLGET(mp, how);
(kgdb) up 16
#22 0xc0207773 in sosend (so=0xcccf3dc0, addr=0x0, uio=0xcdf0ded4, top=0x0,
    control=0x0, flags=0, p=0xcc2ff1e0) at /usr/src/sys/kern/uipc_socket.c:567
567                                     MCLGET(m, M_WAIT);

----------
2002.08.14		RELENG_4 is unstable,
			kernel panic

(kgdb) up 6
#6  0xc02077d3 in sosend (so=0xcccf1900, addr=0x0, uio=0xcdf0ced4, top=0x0,
    control=0x0, flags=0, p=0xcc2ff1e0) at /usr/src/sys/kern/uipc_socket.c:567
567                                     MCLGET(m, M_WAIT);

----------
2002.08.15		RELENG_4 is unstable,
			double kernel panic

(kgdb) up 6
#6  0xc0204917 in m_getcl (how=1, type=1, flags=2)
    at /usr/src/sys/kern/uipc_mbuf.c:589
589                     MCLGET(mp, how);

(kgdb) up 16
#22 0xc02077ef in sosend (so=0xcccf1c00, addr=0x0, uio=0xcdf0ced4, top=0x0,
    control=0x0, flags=0, p=0xcc2ff1e0) at /usr/src/sys/kern/uipc_socket.c:567
567                                     MCLGET(m, M_WAIT);


----------
2002.08.16		RELENG_4 is unstable,
			kernel panic

(kgdb) up 6
#6  0xc02077ef in sosend (so=0xcccf1a80, addr=0x0, uio=0xcdf0ced4, top=0x0,
    control=0x0, flags=0, p=0xcc2ff1e0) at /usr/src/sys/kern/uipc_socket.c:567
567                                     MCLGET(m, M_WAIT);

----------
2002.08.20		RELENG_4 is unstable,
			double kernel panic

(kgdb) up 6
#6  0xc0204963 in m_getcl (how=1, type=1, flags=2)
    at /usr/src/sys/kern/uipc_mbuf.c:589
589                     MCLGET(mp, how);
(kgdb) up 16
#22 0xc020783b in sosend (so=0xcccf1c00, addr=0x0, uio=0xcdf0fed4, top=0x0,
    control=0x0, flags=0, p=0xcc2ff1e0) at /usr/src/sys/kern/uipc_socket.c:567
567                                     MCLGET(m, M_WAIT);

----------
2002.08.30		RELENG_4 is unstable,
			kernel panic

----------
2002.09.03		RELENG_4 is unstable,
			kernel panic
(kgdb) up 6
#6  0xc0227f87 in sosend (so=0xcccdab40, addr=0x0, uio=0xcda28ed4, top=0x0,
    control=0x0, flags=0, p=0xcc2e81e0) at /usr/src/sys/kern/uipc_socket.c:567
567                                     MCLGET(m, M_WAIT);

>sysctl -a | grep pool
kern.ipc.mcl_pool_max: 0
kern.ipc.mcl_pool_now: 0
>netstat -m -M vmcore.26 -N /usr/obj/usr/src/sys/GENERIC/kernel.debug
161/640/10048 mbufs in use (current/peak/max):
        161 mbufs allocated to data
159/326/2512 mbuf clusters in use (current/peak/max)
812 Kbytes allocated to network (10% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
vmstat -M vmcore.26 -N /usr/obj/usr/src/sys/GENERIC/kernel.debug
 procs      memory      page                    disks     faults      cpu
 r b w     avm    fre  flt  re  pi  po  fr  sr ad0 md0   in   sy  cs us sy id
 1 0 0   26188  36156   16   0   0   0  28   0   0   0  257  509 176  0  2 97

----------

So, it looks as if the problem was introduced beetween
2002.08.09 00:00 and 2002.08.10 12:35.

This could by interesting:
 Edit src/sys/dev/fxp/if_fxp.c
  Add delta 1.110.2.24 2002.08.09.02.04.20 luigi
 Edit src/sys/kern/uipc_mbuf.c
  Add delta 1.51.2.17 2002.08.09.02.11.08 luigi
  Add delta 1.51.2.18 2002.08.10.12.34.31 iedowse

The instruction pointer in panic message points to sosend()
and sometimes to m_getcl().
Crashdumps were removed from my HDD cause I don't have enough
space to keep all 26 cores. I have screenshots with `bt' info
from gdb for almost every panic message. It's also easy for me
to reproduce these panics on machine described in this PR.


Was this helpfull?


-- 
Pawe³ Ma³achowski
Comment 5 iedowse freebsd_committer freebsd_triage 2002-10-30 19:14:46 UTC 
State Changed
From-To: feedback->patched


Fixed in -CURRENT for the default non-DEVICE_POLLING case, awaiting 
MFC to -STABLE.
Comment 6 iedowse freebsd_committer freebsd_triage 2002-11-13 21:01:24 UTC 
State Changed
From-To: patched->closed


Fixed in both -STABLE and -CURRENT.