Bug 240471 - panic: rcv_start < rcv_end
Summary: panic: rcv_start < rcv_end
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Michael Tuexen
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2019-09-10 14:13 UTC by Larry Rosenman
Modified: 2019-09-13 20:40 UTC (History)
4 users (show)

See Also:
koobs: mfc-stable11?
tuexen: mfc-stable12+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Larry Rosenman freebsd_committer 2019-09-10 14:13:00 UTC
I've seen this panic for a while randomly.  I have cores.

this latest one is on r352025

Unread portion of the kernel message buffer:
panic: rcv_start < rcv_end
cpuid = 0
time = 1568101636
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01f7ccb380
vpanic() at vpanic+0x19d/frame 0xfffffe01f7ccb3d0
panic() at panic+0x43/frame 0xfffffe01f7ccb430
tcp_update_dsack_list() at tcp_update_dsack_list+0x489/frame 0xfffffe01f7ccb4b0
tcp_do_segment() at tcp_do_segment+0x1fbd/frame 0xfffffe01f7ccb590
tcp_input() at tcp_input+0xde8/frame 0xfffffe01f7ccb6e0
ip_input() at ip_input+0x11d/frame 0xfffffe01f7ccb780
netisr_dispatch_src() at netisr_dispatch_src+0x89/frame 0xfffffe01f7ccb7f0
ether_demux() at ether_demux+0x13b/frame 0xfffffe01f7ccb820
ng_ether_rcv_upper() at ng_ether_rcv_upper+0x95/frame 0xfffffe01f7ccb840
ng_apply_item() at ng_apply_item+0x100/frame 0xfffffe01f7ccb8c0
ng_snd_item() at ng_snd_item+0x2ad/frame 0xfffffe01f7ccb900
ng_apply_item() at ng_apply_item+0x100/frame 0xfffffe01f7ccb980
ng_snd_item() at ng_snd_item+0x2ad/frame 0xfffffe01f7ccb9c0
ng_ether_input() at ng_ether_input+0x4c/frame 0xfffffe01f7ccb9f0
ether_nh_input() at ether_nh_input+0x2cd/frame 0xfffffe01f7ccba40
netisr_dispatch_src() at netisr_dispatch_src+0x89/frame 0xfffffe01f7ccbab0
ether_input() at ether_input+0x48/frame 0xfffffe01f7ccbad0
bce_intr() at bce_intr+0x697/frame 0xfffffe01f7ccbb50
ithread_loop() at ithread_loop+0x187/frame 0xfffffe01f7ccbbb0
fork_exit() at fork_exit+0x84/frame 0xfffffe01f7ccbbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01f7ccbbf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 2d7h30m34s
Dumping 26161 out of 131027 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:392
#2  0xffffffff804bd160 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:479
#3  0xffffffff804bd5d9 in vpanic (fmt=<optimized out>, ap=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:908
#4  0xffffffff804bd313 in panic (fmt=<unavailable>)
    at /usr/src/sys/kern/kern_shutdown.c:835
#5  0xffffffff80676049 in tcp_update_dsack_list (tp=0xfffff81092c91760,
    rcv_start=1781970437, rcv_end=1781970437)
    at /usr/src/sys/netinet/tcp_sack.c:166
#6  0xffffffff8066b01d in tcp_do_segment (m=0xfffff806144cfa00,
    th=<optimized out>, so=0xfffff8018caf2a98, tp=0xfffff81092c91760,
    drop_hdrlen=40, tlen=<optimized out>, iptos=0 '\000')
    at /usr/src/sys/netinet/tcp_input.c:3076
#7  0xffffffff80668378 in tcp_input (mp=<optimized out>,
    offp=<optimized out>, proto=<optimized out>)
    at /usr/src/sys/netinet/tcp_input.c:1373
#8  0xffffffff805f2c6d in ip_input (m=0x0)
    at /usr/src/sys/netinet/ip_input.c:829
#9  0xffffffff805ce9c9 in netisr_dispatch_src (proto=1,
    source=<optimized out>, m=<unavailable>) at /usr/src/sys/net/netisr.c:1123
#10 0xffffffff805c3bfb in ether_demux (ifp=0xfffff80129859000,
    m=<unavailable>) at /usr/src/sys/net/if_ethersubr.c:913
#11 0xffffffff827f4045 in ng_ether_rcv_upper (hook=<optimized out>,
    item=<optimized out>) at /usr/src/sys/netgraph/ng_ether.c:741
#12 0xffffffff827e1750 in ng_apply_item (node=0xfffff8012ee3e500,
    item=0xfffff805ab929b00, rw=0) at /usr/src/sys/netgraph/ng_base.c:2403
#13 0xffffffff827e144d in ng_snd_item (item=0xfffff805ab929b00, flags=0)
    at /usr/src/sys/netgraph/ng_base.c:2320
#14 0xffffffff827e1750 in ng_apply_item (node=0xfffff8012ee3e000,
    item=0xfffff805ab929b00, rw=0) at /usr/src/sys/netgraph/ng_base.c:2403
#15 0xffffffff827e144d in ng_snd_item (item=0xfffff805ab929b00, flags=0)
    at /usr/src/sys/netgraph/ng_base.c:2320
#16 0xffffffff827f3cbc in ng_ether_input (ifp=<optimized out>,
    mp=0xfffffe01f7ccba18) at /usr/src/sys/netgraph/ng_ether.c:255
#17 0xffffffff805c4e4d in ether_input_internal (ifp=0xfffff80129859000,
    m=0xfffff806144cfa00) at /usr/src/sys/net/if_ethersubr.c:654
#18 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:735
#19 0xffffffff805ce9c9 in netisr_dispatch_src (proto=5,
    source=<optimized out>, m=<unavailable>) at /usr/src/sys/net/netisr.c:1123
#20 0xffffffff805c4038 in ether_input (ifp=0xfffff80129859000, m=0x0)
    at /usr/src/sys/net/if_ethersubr.c:823
#21 0xffffffff8262f877 in bce_rx_intr (sc=<optimized out>)
    at /usr/src/sys/dev/bce/if_bce.c:6848
#22 bce_intr (xsc=0xfffffe0200820000) at /usr/src/sys/dev/bce/if_bce.c:8017
#23 0xffffffff80485ba7 in intr_event_execute_handlers (p=<optimized out>,
    ie=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1148
#24 ithread_execute_handlers (p=<optimized out>, ie=<optimized out>)
    at /usr/src/sys/kern/kern_intr.c:1161
#25 ithread_loop (arg=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1241
#26 0xffffffff80482704 in fork_exit (
    callout=0xffffffff80485a20 <ithread_loop>, arg=0xfffff80105945820,
    frame=0xfffffe01f7ccbc00) at /usr/src/sys/kern/kern_fork.c:1046
#27 <signal handler called>
(kgdb)
Comment 1 Michael Tuexen freebsd_committer 2019-09-10 21:16:09 UTC
This panic should be fixed in https://svnweb.freebsd.org/changeset/base/352072. Can you update and see if the problem persists?
Comment 2 Larry Rosenman freebsd_committer 2019-09-11 03:29:22 UTC
I've upgraded to r352200.

This has been a VERY intermittent crash. 

We'll see.
Comment 3 Ed Maste freebsd_committer 2019-09-12 14:15:42 UTC
I encountered this at r352060 on a ThunderX2 while trying to install git in order to build and install a new kernel (that would include the fix).
Comment 4 commit-hook freebsd_committer 2019-09-13 08:15:23 UTC
A commit references this bug:

Author: tuexen
Date: Fri Sep 13 08:14:46 UTC 2019
New revision: 352284
URL: https://svnweb.freebsd.org/changeset/base/352284

Log:
  MFC r352072:

  Only update SACK/DSACK lists when a non-empty segment was received.
  This fixes hitting a KASSERT with a valid packet exchange.

  PR:			240471
  Reviewed by:		rrs@, Richard Scheffenegger
  Sponsored by:		Neflix, Inc.
  Differential Revision:	https://reviews.freebsd.org/D21567

Changes:
_U  stable/12/
  stable/12/sys/netinet/tcp_input.c
  stable/12/sys/netinet/tcp_stacks/rack.c
Comment 5 Michael Tuexen freebsd_committer 2019-09-13 08:24:48 UTC
base r352072 will be MFCed to stable/11 when DSACK support will be MFCed. Right now this bug does not exist in stable/11.
However, there is one more known bug in the DSACK support, which needs to be fixed before I'm MFCing DSACK support to stable/11.