Bug 237329 - Panic in mld_fasttimo() during reboot or shutdown
Summary: Panic in mld_fasttimo() during reboot or shutdown
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-STABLE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Andrey V. Elsukov
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2019-04-17 10:12 UTC by Trond.Endrestol
Modified: 2019-05-29 04:06 UTC (History)
4 users (show)

See Also:
koobs: mfc-stable12+


Attachments
Configuration file for custom kernel (2.83 KB, text/plain)
2019-04-17 10:13 UTC, Trond.Endrestol
no flags Details
Proposed patch (518 bytes, patch)
2019-04-25 12:54 UTC, Andrey V. Elsukov
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Trond.Endrestol 2019-04-17 10:12:52 UTC
During reboot or shutdown of
FreeBSD FQDN 12.0-STABLE FreeBSD 12.0-STABLE #1 r345045: Tue Mar 12 03:43:08 CET 2019     root@FQDN:/usr/obj/usr/src/amd64.amd64/sys/PE1950  amd64 1200503 1200503
this happens:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x410
fault code              = supervisor read data  , page not present
instruction pointer     = 0x20:0xffffffff807ea33d
stack pointer           = 0x28:0xfffffe005ad3c8d0
frame pointer           = 0x28:0xfffffe005ad3c960
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (swi4: clock (0))
trap number             = 12
panic: page fault
cpuid = 0
time = 1555402802
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff8054125b = db_trace_self_wrapper+0x2b/frame 0xfffffe005ad3c570
vpanic() at 0xffffffff8080aae4 = vpanic+0x1b4/frame 0xfffffe005ad3c5d0
panic() at 0xffffffff8080a923 = panic+0x43/frame 0xfffffe005ad3c630
trap_fatal() at 0xffffffff80b76244 = trap_fatal+0x394/frame 0xfffffe005ad3c690
trap_pfault() at 0xffffffff80b762a9 = trap_pfault+0x49/frame 0xfffffe005ad3c6f0
trap() at 0xffffffff80b7588f = trap+0x29f/frame 0xfffffe005ad3c800
calltrap() at 0xffffffff80b514c5 = calltrap+0x8/frame 0xfffffe005ad3c800
--- trap 0xc, rip = 0xffffffff807ea33d, rsp = 0xfffffe005ad3c8d0, rbp = 0xfffffe005ad3c960 ---
__mtx_lock_sleep() at 0xffffffff807ea33d = __mtx_lock_sleep+0xbd/frame 0xfffffe005ad3c960
mld_fasttimo() at 0xffffffff80a3ae32 = mld_fasttimo+0x492/frame 0xfffffe005ad3ca50
pffasttimo() at 0xffffffff80899fa4 = pffasttimo+0x54/frame 0xfffffe005ad3ca80
softclock_call_cc() at 0xffffffff80824e0e = softclock_call_cc+0x12e/frame 0xfffffe005ad3cb30
softclock() at 0xffffffff808252f9 = softclock+0x79/frame 0xfffffe005ad3cb50
ithread_loop() at 0xffffffff807cd824 = ithread_loop+0x1d4/frame 0xfffffe005ad3cbb0
fork_exit() at 0xffffffff807ca2d3 = fork_exit+0x83/frame 0xfffffe005ad3cbf0
fork_trampoline() at 0xffffffff80b524be = fork_trampoline+0xe/frame 0xfffffe005ad3cbf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 34d16h8m2s
Dumping 4593 out of 12258 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

The system runs a custom kernel. The configuration file will be added shortly. /etc/rc.conf specifies ddb_enable="YES".
Comment 1 Trond.Endrestol 2019-04-17 10:13:27 UTC
Created attachment 203736 [details]
Configuration file for custom kernel
Comment 2 Trond.Endrestol 2019-04-17 10:43:07 UTC
Here are additional details from kgdb:

#9  __mtx_lock_sleep (c=0xfffff800103eb9a0, v=<optimized out>) at /usr/src/sys/kern/kern_mutex.c:565
565                     if (TD_IS_RUNNING(owner)) {
(kgdb) list
560                     /*
561                      * If the owner is running on another CPU, spin until the
562                      * owner stops running or the state of the lock changes.
563                      */
564                     owner = lv_mtx_owner(v);
565                     if (TD_IS_RUNNING(owner)) {
566                             if (LOCK_LOG_TEST(&m->lock_object, 0))
567                                     CTR3(KTR_LOCK,
568                                         "%s: spinning on %p held by %p",
569                                         __func__, m, owner);
(kgdb) print owner
$1 = (struct thread *) 0x0
(kgdb) print v
$2 = <optimized out>
(kgdb) up
#10 0xffffffff80a3ae32 in mld_fasttimo_vnet (inmh=<optimized out>) at /usr/src/sys/netinet6/mld6.c:1413
1413                    IF_ADDR_WLOCK(ifp);
(kgdb) list
1408                                PR_FASTHZ);
1409                            mbufq_init(&qrq, MLD_MAX_G_GS_PACKETS);
1410                            mbufq_init(&scq, MLD_MAX_STATE_CHANGE_PACKETS);
1411                    }
1412
1413                    IF_ADDR_WLOCK(ifp);
1414                    NET_EPOCH_ENTER_ET(et);
1415                    CK_STAILQ_FOREACH(ifma, &ifp->if_multiaddrs, ifma_link) {
1416                            inm = in6m_ifmultiaddr_get_inm(ifma);
1417                            if (inm == NULL)
(kgdb) print ifp
$3 = (struct ifnet *) 0xfffff800103eb800
(kgdb) print *ifp
$4 = {if_link = {cstqe_next = 0x0}, if_clones = {le_next = 0x0, le_prev = 0xfffff8000b8be728}, if_groups = {cstqh_first = 0xfffff8002ef82b00, cstqh_last = 0xfffff80013e04308}, if_alloctype = 6 '\006', if_softc = 0xfffff8002e3a1200, if_llsoftc = 0x0, if_l2com = 0x0,
  if_dname = 0xffffffff80d94b78 "lagg", if_dunit = 0, if_index = 6, if_index_reserved = 0, if_xname = "lagg0\000\000\000\000\000\000\000\000\000\000", if_description = 0x0, if_flags = 34819, if_drv_flags = 64, if_capabilities = 262555, if_capenable = 411,
  if_linkmib = 0x0, if_linkmiblen = 0, if_refcount = 0, if_type = 6 '\006', if_addrlen = 6 '\006', if_hdrlen = 14 '\016', if_link_state = 2 '\002', if_mtu = 1500, if_metric = 0, if_baudrate = 4000000000, if_hwassist = 22, if_epoch = 11, if_lastchange = {
    tv_sec = 1552406977, tv_usec = 971493}, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0, ifq_maxlen = 50, ifq_mtx = {lock_object = {lo_name = 0xfffff800103eb858 "lagg0", lo_flags = 16908288, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, ifq_drv_head = 0x0,
    ifq_drv_tail = 0x0, ifq_drv_len = 0, ifq_drv_maxlen = 0, altq_type = 0, altq_flags = 0, altq_disc = 0x0, altq_ifp = 0xfffff800103eb800, altq_enqueue = 0x0, altq_dequeue = 0x0, altq_request = 0x0, altq_clfier = 0x0, altq_classify = 0x0, altq_tbr = 0x0,
    altq_cdnr = 0x0}, if_linktask = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xffffffff80912410 <do_link_state_change>, ta_context = 0xfffff800103eb800}, if_addr_lock = {lock_object = {lo_name = 0xffffffff80c2b74c "if_addr_lock",
      lo_flags = 16908288, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, if_addrhead = {cstqh_first = 0xfffff8002ebce700, cstqh_last = 0xfffff800130d3e28}, if_multiaddrs = {cstqh_first = 0xfffff8000ab24100, cstqh_last = 0xfffff800430d7100}, if_amcount = 0,
  if_addr = 0xfffff8002ebce700, if_hw_addr = 0xfffff8002eefa500, if_broadcastaddr = 0xffffffff80d945c0 <etherbroadcastaddr> "\377\377\377\377\377\377", if_afdata_lock = {lock_object = {lo_name = 0xffffffff80c73b50 "if_afdata", lo_flags = 16908288, lo_data = 0,
      lo_witness = 0x0}, mtx_lock = 4}, if_afdata = {0x0, 0x0, 0xfffff8002ec26d00, 0x0 <repeats 25 times>, 0xfffff8002eefab00, 0x0 <repeats 13 times>}, if_afdata_initialized = 2, if_fib = 0, if_vnet = 0xfffff80002472900, if_home_vnet = 0xfffff80002472900,
  if_vlantrunk = 0x0, if_bpf = 0xfffff8002ec62700, if_pcount = 0, if_bridge = 0x0, if_lagg = 0x0, if_pf_kif = 0x0, if_carp = 0x0, if_label = 0x0, if_netmap = 0x0, if_output = 0xffffffff8091d7b0 <ether_output>, if_input = 0xffffffff8091e4f0 <ether_input>,
  if_bridge_input = 0x0, if_bridge_output = 0x0, if_bridge_linkstate = 0x0, if_start = 0x0, if_ioctl = 0xffffffff80921a30 <lagg_ioctl>, if_init = 0xffffffff809218b0 <lagg_init>, if_resolvemulti = 0xffffffff8091e560 <ether_resolvemulti>,
  if_qflush = 0xffffffff809218a0 <lagg_qflush>, if_transmit = 0xffffffff809216e0 <lagg_transmit>, if_reassign = 0xffffffff8091e750 <ether_reassign>, if_get_counter = 0xffffffff80923980 <lagg_get_counter>, if_requestencap = 0xffffffff8091e680 <ether_requestencap>,
  if_counters = {0xfffffe0000481b50, 0xfffffe0000481b40, 0xfffffe0000481b30, 0xfffffe0000481b20, 0xfffffe0000481b10, 0xfffffe0000481b00, 0xfffffe0000481af0, 0xfffffe0000481ae0, 0xfffffe0000481ad0, 0xfffffe0000481ac0, 0xfffffe0000481ab0, 0xfffffe0000481aa0},
  if_hw_tsomax = 65518, if_hw_tsomaxsegcount = 35, if_hw_tsomaxsegsize = 2048, if_snd_tag_alloc = 0x0, if_snd_tag_modify = 0x0, if_snd_tag_query = 0x0, if_snd_tag_free = 0x0, if_pcp = 255 '\377', if_netdump_methods = 0x0, if_epoch_ctx = {data = {
      0xffffffff80912910 <if_destroy>, 0x0}}, if_addr_et = {datap = {0x0, 0x0, 0x0}, datai = {0}}, if_maddr_et = {datap = {0x0, 0x0, 0x0}, datai = {0}}, if_ispare = {0, 0, 0, 0}}
Comment 3 Trond.Endrestol 2019-04-17 10:48:15 UTC
This backtrace is more to the point:

(kgdb) bt
#0  __curthread () at ./machine/pcpu.h:230
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:371
#2  0xffffffff8080a6a0 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451
#3  0xffffffff8080ab40 in vpanic (fmt=<optimized out>, ap=0xfffffe005ad3c610) at /usr/src/sys/kern/kern_shutdown.c:877
#4  0xffffffff8080a923 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:804
#5  0xffffffff80b76244 in trap_fatal (frame=0xfffffe005ad3c810, eva=1040) at /usr/src/sys/amd64/amd64/trap.c:946
#6  0xffffffff80b762a9 in trap_pfault (frame=0xfffffe005ad3c810, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:765
#7  0xffffffff80b7588f in trap (frame=0xfffffe005ad3c810) at /usr/src/sys/amd64/amd64/trap.c:441
#8  <signal handler called>
#9  __mtx_lock_sleep (c=0xfffff800103eb9a0, v=<optimized out>) at /usr/src/sys/kern/kern_mutex.c:565
#10 0xffffffff80a3ae32 in mld_fasttimo_vnet (inmh=<optimized out>) at /usr/src/sys/netinet6/mld6.c:1413
#11 mld_fasttimo () at /usr/src/sys/netinet6/mld6.c:1333
#12 0xffffffff80899fa4 in pffasttimo (arg=0xfffff800103eb9a0) at /usr/src/sys/kern/uipc_domain.c:521
#13 0xffffffff80824e0e in softclock_call_cc (c=0xffffffff8121af90 <pffast_callout>, cc=0xffffffff812ed900 <cc_cpu>, direct=0) at /usr/src/sys/kern/kern_timeout.c:731
#14 0xffffffff808252f9 in softclock (arg=0xffffffff812ed900 <cc_cpu>) at /usr/src/sys/kern/kern_timeout.c:869
#15 0xffffffff807cd824 in intr_event_execute_handlers (ie=<optimized out>, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1119
#16 ithread_execute_handlers (ie=<optimized out>, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1132
#17 ithread_loop (arg=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1212
#18 0xffffffff807ca2d3 in fork_exit (callout=0xffffffff807cd650 <ithread_loop>, arg=0xfffff80002591100, frame=0xfffffe005ad3cc00) at /usr/src/sys/kern/kern_fork.c:1059
#19 <signal handler called>
Comment 4 Andrey V. Elsukov freebsd_committer 2019-04-18 10:13:19 UTC
Is it possible for you to add options INVARIANTS and INVARIANT_SUPPORT to your kernel config and then try to reproduce the panic? Note, this will impact performance.
Comment 5 Trond.Endrestol 2019-04-18 10:20:53 UTC
(In reply to Andrey V. Elsukov from comment #4)
Sure, but it will have to wait until Tuesday, 2019-04-23.
Comment 6 Trond.Endrestol 2019-04-24 15:16:49 UTC
Good news everyone! I've recompiled r346627 with options INVARIANTS and options INVARIANT_SUPPORT, and I simply wanted to return to singleuser mode when this happened.

Unread portion of the kernel message buffer:
<118>[797] Stopping rpcbind.
<118>[797] Waiting for PIDS: 884.
<118>[797] Stopping devd.
<118>[797] Waiting for PIDS: 786.
[797] panic: Assertion inm->in6m_ifp == NULL failed at /usr/src/sys/netinet6/in6_var.h:794
[797] cpuid = 1
[797] time = 1556117159
[797] KDB: stack backtrace:
[797] db_trace_self_wrapper() at 0xffffffff8059cf6b = db_trace_self_wrapper+0x2b/frame 0xfffffe00004e9340
[797] vpanic() at 0xffffffff808bb56d = vpanic+0x19d/frame 0xfffffe00004e9390
[797] panic() at 0xffffffff808bb333 = panic+0x43/frame 0xfffffe00004e93f0
[797] mld_set_version() at 0xffffffff80ad00a5 = mld_set_version+0x2a5/frame 0xfffffe00004e9450
[797] mld_input() at 0xffffffff80acdd0d = mld_input+0x2fd/frame 0xfffffe00004e9500
[797] icmp6_input() at 0xffffffff80aac86c = icmp6_input+0x41c/frame 0xfffffe00004e96a0
[797] ip6_input() at 0xffffffff80ac69ce = ip6_input+0xdde/frame 0xfffffe00004e9790
[797] netisr_dispatch_src() at 0xffffffff809db842 = netisr_dispatch_src+0xa2/frame 0xfffffe00004e9800
[797] ether_demux() at 0xffffffff809bfbf7 = ether_demux+0x157/frame 0xfffffe00004e9830
[797] ether_nh_input() at 0xffffffff809c1003 = ether_nh_input+0x403/frame 0xfffffe00004e9890
[797] netisr_dispatch_src() at 0xffffffff809db842 = netisr_dispatch_src+0xa2/frame 0xfffffe00004e9900
[797] ether_input() at 0xffffffff809c0063 = ether_input+0x73/frame 0xfffffe00004e9930
[797] t4_eth_rx() at 0xffffffff805fb8c8 = t4_eth_rx+0xa8/frame 0xfffffe00004e9950
[797] service_iq_fl() at 0xffffffff805fff4a = service_iq_fl+0x45a/frame 0xfffffe00004e99f0
[797] t4_intr() at 0xffffffff805ffadd = t4_intr+0x2d/frame 0xfffffe00004e9a10
[797] ithread_loop() at 0xffffffff8087ee97 = ithread_loop+0x187/frame 0xfffffe00004e9a70
[797] fork_exit() at 0xffffffff8087bb04 = fork_exit+0x84/frame 0xfffffe00004e9ab0
[797] fork_trampoline() at 0xffffffff80be002e = fork_trampoline+0xe/frame 0xfffffe00004e9ab0
[797] --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
[797] Uptime: 13m17s
[797] Dumping 4546 out of 32705 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Here's the backtrace:

(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu.h:230
#1  doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:371
#2  0xffffffff808bb180 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451
#3  0xffffffff808bb5c9 in vpanic (fmt=<optimized out>, ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:877
#4  0xffffffff808bb333 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:804
#5  0xffffffff80ad00a5 in in6m_rele_locked (inmh=<optimized out>, inm=<optimized out>) at /usr/src/sys/netinet6/in6_var.h:794
#6  mld_v2_cancel_link_timers (mli=<optimized out>) at /usr/src/sys/netinet6/mld6.c:1707
#7  mld_set_version (mli=<optimized out>, version=<optimized out>) at /usr/src/sys/netinet6/mld6.c:1650
#8  0xffffffff80acdd0d in mld_v1_input_query (ifp=<optimized out>, ip6=<optimized out>, mld=<optimized out>) at /usr/src/sys/netinet6/mld6.c:699
#9  mld_input (m=<optimized out>, off=<optimized out>, icmp6len=<optimized out>) at /usr/src/sys/netinet6/mld6.c:1292
#10 0xffffffff80aac86c in icmp6_input (mp=<optimized out>, offp=0xfffffe00004e96ec, proto=<optimized out>) at /usr/src/sys/netinet6/icmp6.c:622
#11 0xffffffff80ac69ce in ip6_input (m=0xfffff80011dde800) at /usr/src/sys/netinet6/ip6_input.c:964
#12 0xffffffff809db842 in netisr_dispatch_src (proto=6, source=<optimized out>, m=<unavailable>) at /usr/src/sys/net/netisr.c:1122
#13 0xffffffff809bfbf7 in ether_demux (ifp=0xfffff8000c8dd800, m=<unavailable>) at /usr/src/sys/net/if_ethersubr.c:874
#14 0xffffffff809c1003 in ether_input_internal (ifp=0xfffff8000c8dd800, m=<unavailable>) at /usr/src/sys/net/if_ethersubr.c:662
#15 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:692
#16 0xffffffff809db842 in netisr_dispatch_src (proto=5, source=<optimized out>, m=<unavailable>) at /usr/src/sys/net/netisr.c:1122
#17 0xffffffff809c0063 in ether_input (ifp=0xfffff8000c8dd800, m=0x0) at /usr/src/sys/net/if_ethersubr.c:782
#18 0xffffffff805fb8c8 in t4_eth_rx (iq=<optimized out>, rss=<optimized out>, m0=0xfffff80011dde800) at /usr/src/sys/dev/cxgbe/t4_sge.c:2055
#19 0xffffffff805fff4a in service_iq_fl (iq=<optimized out>, budget=0) at /usr/src/sys/dev/cxgbe/t4_sge.c:1692
#20 0xffffffff805ffadd in t4_intr (arg=0xfffffe0096b581c0) at /usr/src/sys/dev/cxgbe/t4_sge.c:1432
#21 0xffffffff8087ee97 in intr_event_execute_handlers (p=<optimized out>, ie=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1129
#22 ithread_execute_handlers (p=<optimized out>, ie=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1142
#23 ithread_loop (arg=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1222
#24 0xffffffff8087bb04 in fork_exit (callout=0xffffffff8087ed10 <ithread_loop>, arg=0xfffff8000c8c5300, frame=0xfffffe00004e9ac0) at /usr/src/sys/kern/kern_fork.c:1060
#25 <signal handler called>
(kgdb) up
(kgdb) up
(kgdb) up
(kgdb) up
(kgdb) up
#5  0xffffffff80ad00a5 in in6m_rele_locked (inmh=<optimized out>, inm=<optimized out>) at /usr/src/sys/netinet6/in6_var.h:794
794                     MPASS(inm->in6m_ifp == NULL);
(kgdb) list
789     {
790             KASSERT(inm->in6m_refcount > 0, ("refcount == %d inm: %p", inm->in6m_refcount, inm));
791             IN6_MULTI_LIST_LOCK_ASSERT();
792
793             if (--inm->in6m_refcount == 0) {
794                     MPASS(inm->in6m_ifp == NULL);
795                     inm->in6m_ifma->ifma_protospec = NULL;
796                     MPASS(inm->in6m_ifma->ifma_llifma == NULL);
797                     SLIST_INSERT_HEAD(inmh, inm, in6m_nrele);
798             }
(kgdb) up
#6  mld_v2_cancel_link_timers (mli=<optimized out>) at /usr/src/sys/netinet6/mld6.c:1707
1707                            in6m_rele_locked(&inmh, inm);
(kgdb) list
1702                            /*
1703                             * If we are leaving the group and switching
1704                             * version, we need to release the final
1705                             * reference held for issuing the INCLUDE {}.
1706                             */
1707                            in6m_rele_locked(&inmh, inm);
1708                            /* FALLTHROUGH */
1709                    case MLD_G_QUERY_PENDING_MEMBER:
1710                    case MLD_SG_QUERY_PENDING_MEMBER:
1711                            in6m_clear_recorded(inm);
(kgdb) print inmh
$2 = {slh_first = 0x0}
(kgdb) print &inmh
$3 = (struct in6_multi_head *) 0xfffffe00004e9428
(kgdb) print inm
$4 = (struct in6_multi *) 0xfffff800382a3100
(kgdb) print *inm
$5 = {in6m_addr = {__u6_addr = {__u6_addr8 = "\377\002\000\002\000\000\000\000\000\000\000\000\000\000\002\002", __u6_addr16 = {767, 512, 0, 0, 0, 0, 0, 514}, __u6_addr32 = {33555199, 0, 0, 33685504}}}, in6m_ifp = 0xfffff8000c8dd800, in6m_ifma = 0xfffff8003372d100,
  in6m_refcount = 0, in6m_state = 9, in6m_timer = 0, in6m_mli = 0xfffff80011df1700, in6m_nrele = {sle_next = 0x0}, in6m_defer = {sle_next = 0x0}, in6m_srcs = {rbh_root = 0x0}, in6m_nsrc = 0, in6m_scq = {mq_head = {stqh_first = 0xfffff801939f2d00,
      stqh_last = 0xfffff801939f2d08}, mq_len = 1, mq_maxlen = 24}, in6m_lastgsrtv = {tv_sec = 0, tv_usec = 0}, in6m_sctimer = 7, in6m_scrv = 1, in6m_st = {{iss_fmode = 0, iss_asm = 0, iss_ex = 0, iss_in = 0, iss_rec = 0}, {iss_fmode = 0, iss_asm = 0, iss_ex = 0,
      iss_in = 0, iss_rec = 0}}}
(kgdb) print inm->in6m_ifp
$6 = (struct ifnet *) 0xfffff8000c8dd800
(kgdb) print *inm->in6m_ifp
$7 = {if_link = {cstqe_next = 0xfffff8000c93a800}, if_clones = {le_next = 0x0, le_prev = 0x0}, if_groups = {cstqh_first = 0xfffff8000c8db300, cstqh_last = 0xfffff8000c8db308}, if_alloctype = 6 '\006', if_softc = 0xfffff8000c8b1a00, if_llsoftc = 0x0, if_l2com = 0x0,
  if_dname = 0xfffff8000c397558 "cc", if_dunit = 0, if_index = 2, if_index_reserved = 0, if_xname = "cc0", '\000' <repeats 12 times>, if_description = 0x0, if_flags = 34819, if_drv_flags = 64, if_capabilities = 49072059, if_capenable = 49022907, if_linkmib = 0x0,
  if_linkmiblen = 0, if_refcount = 1, if_type = 6 '\006', if_addrlen = 6 '\006', if_hdrlen = 14 '\016', if_link_state = 2 '\002', if_mtu = 1500, if_metric = 0, if_baudrate = 10000000000, if_hwassist = 5655, if_epoch = 1, if_lastchange = {tv_sec = 1556116386,
    tv_usec = 478762}, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0, ifq_maxlen = 50, ifq_mtx = {lock_object = {lo_name = 0xfffff8000c8dd858 "cc0", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, ifq_drv_head = 0x0, ifq_drv_tail = 0x0,
    ifq_drv_len = 0, ifq_drv_maxlen = 0, altq_type = 0, altq_flags = 0, altq_disc = 0x0, altq_ifp = 0xfffff8000c8dd800, altq_enqueue = 0x0, altq_dequeue = 0x0, altq_request = 0x0, altq_clfier = 0x0, altq_classify = 0x0, altq_tbr = 0x0, altq_cdnr = 0x0}, if_linktask = {
    ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xffffffff809b4810 <do_link_state_change>, ta_context = 0xfffff8000c8dd800}, if_addr_lock = {lock_object = {lo_name = 0xffffffff80cd0ca1 "if_addr_lock", lo_flags = 16973824, lo_data = 0,
      lo_witness = 0x0}, mtx_lock = 18446735277827061120}, if_addrhead = {cstqh_first = 0xfffff8000c8dad00, cstqh_last = 0xfffff8003376f228}, if_multiaddrs = {cstqh_first = 0xfffff8003372d100, cstqh_last = 0xfffff80028b7b700}, if_amcount = 0, if_addr = 0xfffff8000c8dad00,
  if_hw_addr = 0xfffff8000c8dab00, if_broadcastaddr = 0xffffffff80e6b8f0 <etherbroadcastaddr> "\377\377\377\377\377\377", if_afdata_lock = {lock_object = {lo_name = 0xffffffff80d406b2 "if_afdata", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0},
  if_afdata = {0x0, 0x0, 0xfffff80011e03900, 0x0 <repeats 25 times>, 0xfffff80011e03300, 0x0 <repeats 13 times>}, if_afdata_initialized = 2, if_fib = 0, if_vnet = 0xfffff800035ccd00, if_home_vnet = 0xfffff800035ccd00, if_vlantrunk = 0x0, if_bpf = 0xfffff8000c8da500,
  if_pcount = 0, if_bridge = 0x0, if_lagg = 0x0, if_pf_kif = 0x0, if_carp = 0x0, if_label = 0x0, if_netmap = 0x0, if_output = 0xffffffff809bf2b0 <ether_output>, if_input = 0xffffffff809bfff0 <ether_input>, if_bridge_input = 0x0, if_bridge_output = 0x0,
  if_bridge_linkstate = 0x0, if_start = 0x0, if_ioctl = 0xffffffff805ed8c0 <cxgbe_ioctl>, if_init = 0xffffffff805ed760 <cxgbe_init>, if_resolvemulti = 0xffffffff809c0110 <ether_resolvemulti>, if_qflush = 0xffffffff805ee760 <cxgbe_qflush>,
  if_transmit = 0xffffffff805ee610 <cxgbe_transmit>, if_reassign = 0xffffffff809c0300 <ether_reassign>, if_get_counter = 0xffffffff805e1770 <cxgbe_get_counter>, if_requestencap = 0xffffffff809c0230 <ether_requestencap>, if_counters = {0xfffffe00004cdd10,
    0xfffffe00004cdd00, 0xfffffe00004cdcf0, 0xfffffe00004cdce0, 0xfffffe00004cdcd0, 0xfffffe00004cdcc0, 0xfffffe00004cdcb0, 0xfffffe00004cdca0, 0xfffffe00004cdc90, 0xfffffe00004cdc80, 0xfffffe00004cdc70, 0xfffffe00004cdc60}, if_hw_tsomax = 65535,
  if_hw_tsomaxsegcount = 38, if_hw_tsomaxsegsize = 65536, if_snd_tag_alloc = 0x0, if_snd_tag_modify = 0x0, if_snd_tag_query = 0x0, if_snd_tag_free = 0x0, if_pcp = 255 '\377', if_netdump_methods = 0x0, if_epoch_ctx = {data = {0x0, 0x0}}, if_unused = {0x0, 0x0, 0x0, 0x0},
  if_ispare = {0, 0, 0, 0}}

Please let me know if you need anything else from the dump.
Comment 7 Andrey V. Elsukov freebsd_committer 2019-04-25 12:54:58 UTC
Created attachment 204004 [details]
Proposed patch

Can you try this patch?
Comment 8 Trond.Endrestol 2019-04-26 07:34:00 UTC
I've compiled, installed, and booted the modified kernel. I ran "shutdown now" as soon as multiuser boot was completed. No crash this time. I'm letting the system run until this afternoon and then I'll reboot again to see if the kernel is done crashing. Thank you for the patch.
Comment 9 Hans Petter Selasky freebsd_committer 2019-04-26 08:31:27 UTC
FYI:

@ae : make sure the changes doesn't collide with these ones:

https://reviews.freebsd.org/D19886
Comment 10 Trond.Endrestol 2019-04-26 08:43:22 UTC
(In reply to Hans Petter Selasky from comment #9)
Thank you for the heads up, Hans Petter. I'll follow the developments in the review until it's committed.
Comment 11 Trond.Endrestol 2019-04-26 21:27:04 UTC
(In reply to Trond.Endrestol from comment #8)
Another reboot confirms the bug has been eliminated by the patch. I'm looking forward to the forthcoming multicast related changes.
Comment 12 commit-hook freebsd_committer 2019-05-09 07:58:23 UTC
A commit references this bug:

Author: ae
Date: Thu May  9 07:57:34 UTC 2019
New revision: 347383
URL: https://svnweb.freebsd.org/changeset/base/347383

Log:
  In mld_v2_cancel_link_timers() check number of references and disconnect
  inm before releasing the last reference. This fixes possible panics and
  assertion.

  PR:		237329
  Reviewed by:	mmacy
  MFC after:	2 weeks

Changes:
  head/sys/netinet6/mld6.c
Comment 13 commit-hook freebsd_committer 2019-05-24 08:41:01 UTC
A commit references this bug:

Author: ae
Date: Fri May 24 08:40:38 UTC 2019
New revision: 348232
URL: https://svnweb.freebsd.org/changeset/base/348232

Log:
  MFC r347383:
    In mld_v2_cancel_link_timers() check number of references and disconnect
    inm before releasing the last reference. This fixes possible panics and
    assertion.

    PR:		237329
    Reviewed by:	mmacy

Changes:
_U  stable/12/
  stable/12/sys/netinet6/mld6.c
Comment 14 Andrey V. Elsukov freebsd_committer 2019-05-24 08:53:55 UTC
Fixed in head/ and stable/12. Thanks!