I have encountered this crash twice on FreeBSD 15.0, the later one precisely on 15.0-p9. (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xffffffff80b55969 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:519 #3 0xffffffff80b55e77 in vpanic (fmt=0xffffffff811a72af "%s", ap=ap@entry=0xfffffe011f345730) at /usr/src/sys/kern/kern_shutdown.c:974 #4 0xffffffff80b55ca3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:887 #5 0xffffffff81053c08 in trap_fatal (frame=<optimized out>, eva=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:969 #6 0xffffffff81053c08 in trap_pfault (frame=0xfffffe011f3457b0, usermode=false, signo=<optimized out>, ucode=<optimized out>) #7 <signal handler called> #8 m_copydata (m=0x0, m@entry=0xfffff80b6ea88100, off=0, len=len@entry=1, cp=<optimized out>) at /usr/src/sys/kern/uipc_mbuf.c:694 #9 0xffffffff80d35cb8 in tcp_default_output (tp=0xfffff801d7df1000) at /usr/src/sys/netinet/tcp_output.c:1080 #10 0xffffffff80d2dce3 in tcp_output (tp=0xfffff80b6ea88100, tp@entry=0xfffff801d7df1000) at /usr/src/sys/netinet/tcp_var.h:662 #11 0xffffffff80d2c372 in tcp_do_segment (tp=0xfffff801d7df1000, m=<optimized out>, th=0xfffff8014447c296, drop_hdrlen=100, tlen=0, iptos=<optimized out>) at /usr/src/sys/netinet/tcp_input.c:2842 #12 0xffffffff80d29b8c in tcp_input_with_port (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>, port=port@entry=0) at /usr/src/sys/netinet/tcp_input.c:1160 #13 0xffffffff80d29fff in tcp6_input_with_port (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>, port=0) at /usr/src/sys/netinet/tcp_input.c:582 #14 tcp6_input (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>) at /usr/src/sys/netinet/tcp_input.c:589 #15 0xffffffff80d6d14e in ip6_input (m=0x0) at /usr/src/sys/netinet6/ip6_input.c:963 #16 0xffffffff80c9b668 in netisr_process_workstream_proto (nwsp=0xfffffe00469d5e00, proto=6) at /usr/src/sys/net/netisr.c:926 #17 swi_net (arg=0xfffffe00469d5e00) at /usr/src/sys/net/netisr.c:973 #18 0xffffffff80b11709 in intr_event_execute_handlers (ie=0xfffff80001482a00, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1191 #19 ithread_execute_handlers (ie=0xfffff80001482a00, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1204 #20 ithread_loop (arg=arg@entry=0xfffff80001401c20) at /usr/src/sys/kern/kern_intr.c:1297 #21 0xffffffff80b0dcab in fork_exit (callout=0xffffffff80b114d0 <ithread_loop>, arg=0xfffff80001401c20, frame=0xfffffe011f345f40) at /usr/src/sys/kern/kern_fork.c:1153 #22 <signal handler called> (kgdb) I have a full memory dump, so I can provide additional information.
(In reply to Kajetan Staszkiewicz from comment #0) Hi Kajetan, The issue was most likely fixed in f3de667137e90679cd20fa5c1dcd93a4c51ad848, but the fix will not have MFC as discussed by devs. Perhaps you can give 16.0-CURRENT a try on this machine?
I have some similar crash (while restarting wlan0). I would like to test with 16.0-CURRENT, if I could somehow trigger the bug.
No, I don't think this is what f3de667137e90679cd20fa5c1dcd93a4c51ad848 fixed. That one fixed accessing freed memory in in6_selecthlim() as IPv6 software contexts were freed immediately (w/o epoch) on interface removal bug 279653. This panic looks more like an invalid metadata in an mbuf, which is not protected by epoch. satanist reports that his panic is associated with interface departure. Kajetan, is your crash associated with interface departure or not?
Is it possible to run with INVARIANTS for a while to get a crash?
The panic header (instruction pointer, fault address) will also be helpful. While here will also suggest to upgrade to 15.1-RELEASE or to recent CURRENT stabweek.
I added the bug to the TCP meeting agenda. Next meeting is going to happen on 25th June. Would be great to gather more data before this date. Thanks and sorry for flood of email. I could have typed this all in one comment. Sorry.
Is the bug somewhat reproducible? Would you be willing to enable BlackBox Logging? You can find instructions at: https://wiki.freebsd.org/MichaelTuexen
Gleb Smirnoff wrote: > satanist reports that his panic is associated with interface departure. Kajetan, is your crash associated with interface departure or not? "departure", as in interface being shut down or removed? No, this happened on a router, all its interfaces are permanent. Gleb Smirnoff wrote: > Is it possible to run with INVARIANTS for a while to get a crash? I have DEBUG kernels running on a few machines, but the issue has never happened there. And this is not something I know how to trigger, it only happened on 2 15.0 hosts so far. Let me check if only having INVARIANTS cause noticeable performance degradation first. Michael Tuexen wrote: > Is the bug somewhat reproducible? I don't think so. And the machines mostly route and firewall (pf) the traffic, there's very few services running on them. > Would you be willing to enable BlackBox Logging? You can find instructions at: https://wiki.freebsd.org/MichaelTuexen I'll give it a try.
(In reply to Gleb Smirnoff from comment #3) > satanist reports that his panic is associated with interface departure. To clarify this: I only seen this panic once. This was at the same time as I have restarted my wlan0 interface. But I do restart the wlan0 interface quite often without a panic. So the panic might be associated with the interface departure or it could just be an coincidence.