I've seen this panic for a while randomly. I have cores. this latest one is on r352025 Unread portion of the kernel message buffer: panic: rcv_start < rcv_end cpuid = 0 time = 1568101636 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01f7ccb380 vpanic() at vpanic+0x19d/frame 0xfffffe01f7ccb3d0 panic() at panic+0x43/frame 0xfffffe01f7ccb430 tcp_update_dsack_list() at tcp_update_dsack_list+0x489/frame 0xfffffe01f7ccb4b0 tcp_do_segment() at tcp_do_segment+0x1fbd/frame 0xfffffe01f7ccb590 tcp_input() at tcp_input+0xde8/frame 0xfffffe01f7ccb6e0 ip_input() at ip_input+0x11d/frame 0xfffffe01f7ccb780 netisr_dispatch_src() at netisr_dispatch_src+0x89/frame 0xfffffe01f7ccb7f0 ether_demux() at ether_demux+0x13b/frame 0xfffffe01f7ccb820 ng_ether_rcv_upper() at ng_ether_rcv_upper+0x95/frame 0xfffffe01f7ccb840 ng_apply_item() at ng_apply_item+0x100/frame 0xfffffe01f7ccb8c0 ng_snd_item() at ng_snd_item+0x2ad/frame 0xfffffe01f7ccb900 ng_apply_item() at ng_apply_item+0x100/frame 0xfffffe01f7ccb980 ng_snd_item() at ng_snd_item+0x2ad/frame 0xfffffe01f7ccb9c0 ng_ether_input() at ng_ether_input+0x4c/frame 0xfffffe01f7ccb9f0 ether_nh_input() at ether_nh_input+0x2cd/frame 0xfffffe01f7ccba40 netisr_dispatch_src() at netisr_dispatch_src+0x89/frame 0xfffffe01f7ccbab0 ether_input() at ether_input+0x48/frame 0xfffffe01f7ccbad0 bce_intr() at bce_intr+0x697/frame 0xfffffe01f7ccbb50 ithread_loop() at ithread_loop+0x187/frame 0xfffffe01f7ccbbb0 fork_exit() at fork_exit+0x84/frame 0xfffffe01f7ccbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01f7ccbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Uptime: 2d7h30m34s Dumping 26161 out of 131027 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:392 #2 0xffffffff804bd160 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:479 #3 0xffffffff804bd5d9 in vpanic (fmt=<optimized out>, ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:908 #4 0xffffffff804bd313 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:835 #5 0xffffffff80676049 in tcp_update_dsack_list (tp=0xfffff81092c91760, rcv_start=1781970437, rcv_end=1781970437) at /usr/src/sys/netinet/tcp_sack.c:166 #6 0xffffffff8066b01d in tcp_do_segment (m=0xfffff806144cfa00, th=<optimized out>, so=0xfffff8018caf2a98, tp=0xfffff81092c91760, drop_hdrlen=40, tlen=<optimized out>, iptos=0 '\000') at /usr/src/sys/netinet/tcp_input.c:3076 #7 0xffffffff80668378 in tcp_input (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>) at /usr/src/sys/netinet/tcp_input.c:1373 #8 0xffffffff805f2c6d in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:829 #9 0xffffffff805ce9c9 in netisr_dispatch_src (proto=1, source=<optimized out>, m=<unavailable>) at /usr/src/sys/net/netisr.c:1123 #10 0xffffffff805c3bfb in ether_demux (ifp=0xfffff80129859000, m=<unavailable>) at /usr/src/sys/net/if_ethersubr.c:913 #11 0xffffffff827f4045 in ng_ether_rcv_upper (hook=<optimized out>, item=<optimized out>) at /usr/src/sys/netgraph/ng_ether.c:741 #12 0xffffffff827e1750 in ng_apply_item (node=0xfffff8012ee3e500, item=0xfffff805ab929b00, rw=0) at /usr/src/sys/netgraph/ng_base.c:2403 #13 0xffffffff827e144d in ng_snd_item (item=0xfffff805ab929b00, flags=0) at /usr/src/sys/netgraph/ng_base.c:2320 #14 0xffffffff827e1750 in ng_apply_item (node=0xfffff8012ee3e000, item=0xfffff805ab929b00, rw=0) at /usr/src/sys/netgraph/ng_base.c:2403 #15 0xffffffff827e144d in ng_snd_item (item=0xfffff805ab929b00, flags=0) at /usr/src/sys/netgraph/ng_base.c:2320 #16 0xffffffff827f3cbc in ng_ether_input (ifp=<optimized out>, mp=0xfffffe01f7ccba18) at /usr/src/sys/netgraph/ng_ether.c:255 #17 0xffffffff805c4e4d in ether_input_internal (ifp=0xfffff80129859000, m=0xfffff806144cfa00) at /usr/src/sys/net/if_ethersubr.c:654 #18 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:735 #19 0xffffffff805ce9c9 in netisr_dispatch_src (proto=5, source=<optimized out>, m=<unavailable>) at /usr/src/sys/net/netisr.c:1123 #20 0xffffffff805c4038 in ether_input (ifp=0xfffff80129859000, m=0x0) at /usr/src/sys/net/if_ethersubr.c:823 #21 0xffffffff8262f877 in bce_rx_intr (sc=<optimized out>) at /usr/src/sys/dev/bce/if_bce.c:6848 #22 bce_intr (xsc=0xfffffe0200820000) at /usr/src/sys/dev/bce/if_bce.c:8017 #23 0xffffffff80485ba7 in intr_event_execute_handlers (p=<optimized out>, ie=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1148 #24 ithread_execute_handlers (p=<optimized out>, ie=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1161 #25 ithread_loop (arg=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1241 #26 0xffffffff80482704 in fork_exit ( callout=0xffffffff80485a20 <ithread_loop>, arg=0xfffff80105945820, frame=0xfffffe01f7ccbc00) at /usr/src/sys/kern/kern_fork.c:1046 #27 <signal handler called> (kgdb)
This panic should be fixed in https://svnweb.freebsd.org/changeset/base/352072. Can you update and see if the problem persists?
I've upgraded to r352200. This has been a VERY intermittent crash. We'll see.
I encountered this at r352060 on a ThunderX2 while trying to install git in order to build and install a new kernel (that would include the fix).
A commit references this bug: Author: tuexen Date: Fri Sep 13 08:14:46 UTC 2019 New revision: 352284 URL: https://svnweb.freebsd.org/changeset/base/352284 Log: MFC r352072: Only update SACK/DSACK lists when a non-empty segment was received. This fixes hitting a KASSERT with a valid packet exchange. PR: 240471 Reviewed by: rrs@, Richard Scheffenegger Sponsored by: Neflix, Inc. Differential Revision: https://reviews.freebsd.org/D21567 Changes: _U stable/12/ stable/12/sys/netinet/tcp_input.c stable/12/sys/netinet/tcp_stacks/rack.c
base r352072 will be MFCed to stable/11 when DSACK support will be MFCed. Right now this bug does not exist in stable/11. However, there is one more known bug in the DSACK support, which needs to be fixed before I'm MFCing DSACK support to stable/11.
Syzkaller https://syzkaller.appspot.com/bug?id=9a6683e229cb20cfeb47212466198661d69f533f
Does this PR need to be kept open?
Not sure. My plan was to keep it open until the DSACK changes (the feature and the corresponding bug fixes) are MFCed to stable/11. If this is not the right way to handle PR, feel free to close it. koobs@ added the mfc-stable11? flag. That is why I haven't closed it.
(In reply to Michael Tuexen from comment #8) Since the bug isn't present in stable/11 I believe it is not necessary to keep the PR open. Of course, there is no problem keeping it open, I have just been looking at a few PRs in the backlog and making sure they are up to date.
So I'm closing this bug, since the problem is not present in stable/11. I'll also MFC the corresponding patch when MFCing the DSACK feature (which was requested more than one time).