Bug 202494

I've encountered these panics 3 times so far (that I recall) -- and each time, it's been at work (vs. any of the other places I use wireless).

The first (from 24 April 2015) is mentioned in <http://docs.FreeBSD.org/cgi/mid.cgi?20150424230151.GQ37361>; the second (from yesterday, 18 August) in <http://docs.FreeBSD.org/cgi/mid.cgi?20150818232007.GN1189>, and the third (this morning) in <http://docs.FreeBSD.org/cgi/mid.cgi?20150819160716.GK63584> (same thread as the 2nd one).

The "uname -a" output for today (as mentioned in the above-cited messages) is:
FreeBSD localhost 10.2-STABLE FreeBSD 10.2-STABLE #123  r286912M/286918:1002500: Wed Aug 19 04:05:06 PDT 2015     root@g1-252.catwhisker.org:/common/S1/obj/usr/src/sys/CANARY  amd64


In following up on a suggestion, I have found the following from today's crash dump:

localhost(10.2-S)[6] kgdb /boot/kernel/kernel.symbols vmcore.1 
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: page fault
cpuid = 1
KDB: stack backtrace:
#0 0xffffffff80946e00 at kdb_backtrace+0x60
#1 0xffffffff8090a9e6 at vpanic+0x126
#2 0xffffffff8090a8b3 at panic+0x43
#3 0xffffffff80c8467b at trap_fatal+0x36b
#4 0xffffffff80c8497d at trap_pfault+0x2ed
#5 0xffffffff80c8401a at trap+0x47a
#6 0xffffffff80c6a1b2 at calltrap+0x8
#7 0xffffffff809eff5e at ieee80211_crypto_delkey+0x1e
#8 0xffffffff80a04d45 at ieee80211_ioctl_delkey+0x65
#11 0xffffffff809cd57b at ifioctl+0x15eb
#12 0xffffffff8095ecf5 at kern_ioctl+0x255
#13 0xffffffff8095e9f0 at sys_ioctl+0x140
#14 0xffffffff80c84f97 at amd64_syscall+0x357
#15 0xffffffff80c6a49b at Xfast_syscall+0xfb
Uptime: 3h16m49s
Dumping 584 out of 8095 MB:..3%..11%..22%..31%..42%..53%..61%..72%..83%..91%

Reading symbols from /boot/kernel/geom_eli.ko.symbols...done.
Loaded symbols for /boot/kernel/geom_eli.ko.symbols
Reading symbols from /boot/kernel/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel/crypto.ko.symbols
Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel/coretemp.ko.symbols
Reading symbols from /boot/kernel/iwn5000fw.ko.symbols...done.
Loaded symbols for /boot/kernel/iwn5000fw.ko.symbols
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/modules/cuse4bsd.ko...done.
Loaded symbols for /boot/modules/cuse4bsd.ko
Reading symbols from /boot/kernel/tmpfs.ko.symbols...done.
Loaded symbols for /boot/kernel/tmpfs.ko.symbols
Reading symbols from /boot/kernel/fdescfs.ko.symbols...done.
Loaded symbols for /boot/kernel/fdescfs.ko.symbols
Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
Loaded symbols for /boot/kernel/linprocfs.ko.symbols
Reading symbols from /boot/modules/vboxnetflt.ko...done.
Loaded symbols for /boot/modules/vboxnetflt.ko
Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
Loaded symbols for /boot/kernel/netgraph.ko.symbols
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ether.ko.symbols
Reading symbols from /boot/modules/vboxnetadp.ko...done.
Loaded symbols for /boot/modules/vboxnetadp.ko
Reading symbols from /usr/local/modules/rtc.ko...done.
Loaded symbols for /usr/local/modules/rtc.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
219     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff8090a642 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451
#2  0xffffffff8090aa25 in vpanic (fmt=<value optimized out>, ap=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:758
#3  0xffffffff8090a8b3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:687
#4  0xffffffff80c8467b in trap_fatal (frame=<value optimized out>, eva=<value optimized out>)
    at /usr/src/sys/amd64/amd64/trap.c:851
#5  0xffffffff80c8497d in trap_pfault (frame=0xfffffe060d5ea510, usermode=<value optimized out>)
    at /usr/src/sys/amd64/amd64/trap.c:674
#6  0xffffffff80c8401a in trap (frame=0xfffffe060d5ea510) at /usr/src/sys/amd64/amd64/trap.c:440
#7  0xffffffff80c6a1b2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236
#8  0xffffffff809f003a in _ieee80211_crypto_delkey () at /usr/src/sys/net80211/ieee80211_crypto.c:105
#9  0xffffffff809eff5e in ieee80211_crypto_delkey (vap=0xfffffe03dd31a000, key=0xfffffe03dd31a800)
    at /usr/src/sys/net80211/ieee80211_crypto.c:461
#10 0xffffffff80a04d45 in ieee80211_ioctl_delkey (vap=0xfffffe03dd31a000, ireq=<value optimized out>)
    at /usr/src/sys/net80211/ieee80211_ioctl.c:1252
#11 0xffffffff80a03bd2 in ieee80211_ioctl_set80211 () at /usr/src/sys/net80211/ieee80211_ioctl.c:2814
#12 0xffffffff80a2c323 in in_control (so=<value optimized out>, cmd=9214790412651315593, data=0xfffffe060d5eab80 "", 
    ifp=0x3, td=<value optimized out>) at /usr/src/sys/netinet/in.c:308
#13 0xffffffff809cd57b in ifioctl (so=0xfffffe03dd31a800, cmd=2149607914, data=0xfffffe060d5ea8e0 "wlan0", 
    td=0xfffff800098b5940) at /usr/src/sys/net/if.c:2770
#14 0xffffffff8095ecf5 in kern_ioctl (td=0xfffff800098b5940, fd=<value optimized out>, com=18446741891282216960)
    at file.h:320
#15 0xffffffff8095e9f0 in sys_ioctl (td=0xfffff800098b5940, uap=0xfffffe060d5eaa40)
    at /usr/src/sys/kern/sys_generic.c:718
#16 0xffffffff80c84f97 in amd64_syscall (td=0xfffff800098b5940, traced=0) at subr_syscall.c:134
#17 0xffffffff80c6a49b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
#18 0x00000008012a2f9a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb) frame 8
#8  0xffffffff809f003a in _ieee80211_crypto_delkey () at /usr/src/sys/net80211/ieee80211_crypto.c:105
105             key->wk_cipher->ic_detach(key);
(kgdb) print *key
Cannot access memory at address 0x0
(kgdb) print key
Cannot access memory at address 0x0
(kgdb) frame 9
#9  0xffffffff809eff5e in ieee80211_crypto_delkey (vap=0xfffffe03dd31a000, key=0xfffffe03dd31a800)
    at /usr/src/sys/net80211/ieee80211_crypto.c:461
461             status = _ieee80211_crypto_delkey(vap, key);
(kgdb) print *key                                                                                                         
$1 = {wk_keylen = 0 '\0', wk_pad = 0 '\0', wk_flags = 3, wk_keyix = 65535, wk_rxkeyix = 65535, 
  wk_key = '\0' <repeats 31 times>, wk_keyrsc = {0 <repeats 17 times>}, wk_keytsc = 0, wk_cipher = 0xffffffff80ef5018, 
  wk_private = 0xfffffe03dd31a000, wk_macaddr = "\000\000\000\000\000"}
(kgdb) 


So:  It seems that at the point in ieee80211_crypto_delkey() that _ieee80211_crypto_delkey() is invoked, "key" actually points at something, but by the time we get to /usr/src/sys/net80211/ieee80211_crypto.c:461, "key" has a value of 0 (so attempting to dereference it is a Bad Idea).

I will plan on copying a gzipped tarball (later today) of the kernel directory for today to the same Web site as everything else.

I'm happy to poke at dumps & test things; I track for stable/10 & head daily (on different slices of the laptop's drive) -- but I don't normally run head for long (or at work).  But I can do that if it would help figure out what the problem is.