Bug 217214 - [tcp] frequent panics in tcp_output/sbsndptr
Summary: [tcp] frequent panics in tcp_output/sbsndptr
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.0-STABLE
Hardware: Any Any
: --- Affects Many People
Assignee: freebsd-net (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-02-18 21:35 UTC by G. Paul Ziemba
Modified: 2021-06-24 18:49 UTC (History)
1 user (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description G. Paul Ziemba 2017-02-18 21:35:06 UTC 
11.0-Stable r313801M

Panics in sbsndptr when called from tcp_output, not always the same place. kgdb traces from two different core dumps follow.

I tried rebuilding kernel with optimization disabled (/etc/src.conf:  COPTFLAGS=-pipe) so I could examine variable values in kgdb, but that kernel crashed before fully coming up (i.e., reboot loop).

Not sure how to proceed from here.

Trace #1

(kgdb) where
#0  doadump (textdump=<value optimized out>) at pcpu.h:222
#1  0xffffffff80abc999 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80abcf50 in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80abcd83 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:690
#4  0xffffffff80b5317a in sbsndptr (sb=<value optimized out>, 
    off=<value optimized out>, len=<value optimized out>, 
    moff=<value optimized out>) at /usr/src/sys/kern/uipc_sockbuf.c:1196
#5  0xffffffff80cddeb8 in tcp_output (tp=<value optimized out>)
    at /usr/src/sys/netinet/tcp_output.c:1047
#6  0xffffffff80cdabd2 in tcp_do_segment (m=0xfffff80023bc4300, 
    th=<value optimized out>, so=0xfffff8026e26b000, tp=0xfffff8019470a820, 
    drop_hdrlen=52, tlen=<value optimized out>, iptos=<value optimized out>, 
    ti_locked=Cannot access memory at address 0x1
) at /usr/src/sys/netinet/tcp_input.c:3173
#7  0xffffffff80cd7d5a in tcp_input (mp=<value optimized out>, 
    offp=<value optimized out>, proto=<value optimized out>)
    at /usr/src/sys/netinet/tcp_input.c:1453
#8  0xffffffff80c4a6d9 in ip_input (m=<value optimized out>)
    at /usr/src/sys/netinet/ip_input.c:820
#9  0xffffffff80be6fd5 in netisr_dispatch_src (proto=1, 
    source=<value optimized out>, m=<value optimized out>)
    at /usr/src/sys/net/netisr.c:1120
#10 0xffffffff80bd0169 in ether_demux (ifp=<value optimized out>, 
    m=<value optimized out>) at /usr/src/sys/net/if_ethersubr.c:850
#11 0xffffffff830825fc in vboxNetFltFreeBSDinput ()
   from /boot/modules/vboxnetflt.ko
#12 0xffffffff80b18e7a in taskqueue_run_locked (queue=<value optimized out>)
    at /usr/src/sys/kern/subr_taskqueue.c:454
#13 0xffffffff80b18c6f in taskqueue_run (queue=0xfffff8000a3a9900)
    at /usr/src/sys/kern/subr_taskqueue.c:473
#14 0xffffffff80a780ef in intr_event_execute_handlers (
    p=<value optimized out>, ie=<value optimized out>)
    at /usr/src/sys/kern/kern_intr.c:1262
#15 0xffffffff80a78356 in ithread_loop (arg=<value optimized out>)
    at /usr/src/sys/kern/kern_intr.c:1275
#16 0xffffffff80a74db5 in fork_exit (
    callout=0xffffffff80a78290 <ithread_loop>, arg=0xfffff8000a37d4e0, 
    frame=0xfffffe07c72baac0) at /usr/src/sys/kern/kern_fork.c:1040
#17 0xffffffff80f9218e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:611
#18 0x0000000000000000 in ?? ()

Trace #2:

(kgdb) where
#0  doadump (textdump=<value optimized out>) at pcpu.h:222
#1  0xffffffff80abc999 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80abcf50 in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80abcd83 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:690
#4  0xffffffff80faecd2 in trap_fatal (frame=0xfffffe07c72ba2f0, eva=24)
    at /usr/src/sys/amd64/amd64/trap.c:801
#5  0xffffffff80faee9c in trap_pfault (frame=0xfffffe07c72ba2f0, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:658
#6  0xffffffff80fae550 in trap (frame=0xfffffe07c72ba2f0)
    at /usr/src/sys/amd64/amd64/trap.c:421
#7  0xffffffff80f91c51 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:236
#8  0xffffffff80b49ce3 in m_copym (m=0x0, off0=<value optimized out>, 
    len=<value optimized out>, wait=1) at /usr/src/sys/kern/uipc_mbuf.c:456
#9  0xffffffff80cddee7 in tcp_output (tp=<value optimized out>)
    at /usr/src/sys/netinet/tcp_output.c:1054
#10 0xffffffff80cdb118 in tcp_do_segment (m=0xfffff800b77c0700, 
    th=<value optimized out>, so=0xfffff800b70616c0, tp=0xfffff800b721a410, 
    drop_hdrlen=80, tlen=<value optimized out>, iptos=<value optimized out>, 
    ti_locked=Cannot access memory at address 0x1
) at /usr/src/sys/netinet/tcp_input.c:2609
#11 0xffffffff80cd7d5a in tcp_input (mp=<value optimized out>, 
    offp=<value optimized out>, proto=<value optimized out>)
    at /usr/src/sys/netinet/tcp_input.c:1453
#12 0xffffffff80c4a6d9 in ip_input (m=<value optimized out>)
    at /usr/src/sys/netinet/ip_input.c:820
#13 0xffffffff80be6fd5 in netisr_dispatch_src (proto=1, 
    source=<value optimized out>, m=<value optimized out>)
    at /usr/src/sys/net/netisr.c:1120
#14 0xffffffff80bd0169 in ether_demux (ifp=<value optimized out>, 
    m=<value optimized out>) at /usr/src/sys/net/if_ethersubr.c:850
#15 0xffffffff830825fc in vboxNetFltFreeBSDinput ()
   from /boot/modules/vboxnetflt.ko
#16 0xffffffff80b18e7a in taskqueue_run_locked (queue=<value optimized out>)
    at /usr/src/sys/kern/subr_taskqueue.c:454
#17 0xffffffff80b18c6f in taskqueue_run (queue=0xfffff8000a3a9900)
    at /usr/src/sys/kern/subr_taskqueue.c:473
#18 0xffffffff80a780ef in intr_event_execute_handlers (
    p=<value optimized out>, ie=<value optimized out>)
    at /usr/src/sys/kern/kern_intr.c:1262
#19 0xffffffff80a78356 in ithread_loop (arg=<value optimized out>)
    at /usr/src/sys/kern/kern_intr.c:1275
#20 0xffffffff80a74db5 in fork_exit (
    callout=0xffffffff80a78290 <ithread_loop>, arg=0xfffff8000a37d4c0, 
    frame=0xfffffe07c72baac0) at /usr/src/sys/kern/kern_fork.c:1040
#21 0xffffffff80f9218e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:611
#22 0x0000000000000000 in ?? ()
Comment 1 Hiren Panchasara freebsd_committer freebsd_triage 2017-02-19 01:00:51 UTC 
Looks like you are running FreeBSD as a guest inside VirtualBox. We've had quite a few occurrences of this problem of sbsndptr panics nothing this consistent (i.e. reboot loop like you are seeing). So this looks like some sort of regression to me either in VirtualBox or kernel. Can you help bisect it? Try different/older version of FreeBSD (or VBox) and see if that helps? 

Sorry, we don't have a solution to this problem but bisecting may help.
Comment 2 G. Paul Ziemba 2017-02-19 01:39:35 UTC 
(In reply to Hiren Panchasara from comment #1)

Thank you for your input. Yes, I am running VirtualBox, but just to be clear, the panics are occurring on the host system, not the guest.

I have rebuilt the kernel with INVARIANTS in hopes of obtaining additional info, but will also see about trying some older versions as you suggest.

It's not entirely clear to me that the reboot loop with the non -O2 kernel is panicinn at the same place. I'm also seeing some startup time panics at line 760 of /usr/src/sys/ufs/ufs/ufs_lookup.c (ufs_dirbad: /v2: bad dir ino 270866706 at offset 1024: mangled entry) in the -O2 kernel. I speculate these could be the result of a prior crash leaving the filesystem in an improper state.
Comment 3 Michael Tuexen freebsd_committer freebsd_triage 2021-06-24 18:49:26 UTC 
Please reopen, if the problem still exists.