Created attachment 166529 [details] full core.txt Got the following panic: borg.lerctr.org dumped core - see /var/crash/vmcore.20 Tue Feb 2 20:59:14 CST 2016 FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #4 r294926: Wed Jan 27 12:37:06 CST 2016 root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 panic: Bad tailq NEXT(0xffffffff81e8b5f8->tqh_last) != NULL GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: panic: Bad tailq NEXT(0xffffffff81e8b5f8->tqh_last) != NULL cpuid = 4 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe2e025122c0 vpanic() at vpanic+0x182/frame 0xfffffe2e02512340 panic() at panic+0x43/frame 0xfffffe2e025123a0 nd6_ra_input() at nd6_ra_input+0x13da/frame 0xfffffe2e02512680 icmp6_input() at icmp6_input+0x97e/frame 0xfffffe2e02512820 ip6_input() at ip6_input+0xc3c/frame 0xfffffe2e02512900 netisr_dispatch_src() at netisr_dispatch_src+0x81/frame 0xfffffe2e02512960 ether_demux() at ether_demux+0x15e/frame 0xfffffe2e02512990 ether_nh_input() at ether_nh_input+0x344/frame 0xfffffe2e025129d0 netisr_dispatch_src() at netisr_dispatch_src+0x81/frame 0xfffffe2e02512a30 ether_input() at ether_input+0x4f/frame 0xfffffe2e02512a60 if_input() at if_input+0xa/frame 0xfffffe2e02512a70 em_rxeof() at em_rxeof+0x2f5/frame 0xfffffe2e02512ae0 em_handle_que() at em_handle_que+0x40/frame 0xfffffe2e02512b20 taskqueue_run_locked() at taskqueue_run_locked+0xf0/frame 0xfffffe2e02512b80 taskqueue_thread_loop() at taskqueue_thread_loop+0x88/frame 0xfffffe2e02512bb0 fork_exit() at fork_exit+0x84/frame 0xfffffe2e02512bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe2e02512bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Uptime: 8h40m34s Dumping 3340 out of 64467 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% core *IS* available.
I can reliably reproduce this by rebooting my pfSense router that is doing the rtadv's.
Created attachment 166582 [details] another one
Created attachment 166583 [details] and a 3rd
vmcore's are ALL available, and I can give a @FreeBSD.org dev access.
Created attachment 166686 [details] Another one
this seems to be at the root of my tcp6 issues. I've put a bunch more core.txt's at: http://www.lerctr.org/~ler/FreeBSD/ I've also put dmesg, loader.conf, rc.conf, sysctl.conf there. I'd really like to get to the bottom of this.
I'm working on this.
Larry's confirmed that the patch here fixes the crash: https://people.freebsd.org/~markj/patches/defrouter_locking.diff I'm going to commit some trivial cleanup portions of that patch and put the rest up for review. It's a bit incomplete. In particular, defrouter_reset() is not locked.
A commit references this bug: Author: markj Date: Thu Feb 25 20:12:05 UTC 2016 New revision: 296063 URL: https://svnweb.freebsd.org/changeset/base/296063 Log: Lock the NDP default router list and count defrouter references. This addresses a number of race conditions that can cause crashes as a result of unsynchronized access to the list. PR: 206904 Tested by: Larry Rosenman <ler@lerctr.org>, Kevin Bowling <kevin.bowling@kev009.com> MFC after: 2 months Differential Revision: https://reviews.freebsd.org/D5315 Changes: head/sys/netinet6/nd6.c head/sys/netinet6/nd6.h head/sys/netinet6/nd6_nbr.c head/sys/netinet6/nd6_rtr.c
This was MFCed to stable/10 in r303458.