221120 – [panic] network stack when using FLOWTABLE

Bug 221120 - [panic] network stack when using FLOWTABLE

Summary: [panic] network stack when using FLOWTABLE

Status:	Closed Overcome By Events

Alias:	None

Product:	Base System
Classification:	Unclassified
Component:	kern (show other bugs)
Version:	11.1-RELEASE
Hardware:	Any Any

Importance:	--- Affects Only Me
Assignee:	Eugene Grosbein

URL:
Keywords:

Depends on:
Blocks:

Reported:	2017-07-31 13:00 UTC by Stéphane D'Alu
Modified:	2017-08-02 08:29 UTC (History)
CC List:	5 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Stéphane D'Alu 2017-07-31 13:00:32 UTC

During this last week i've got numerous kernel panic (about 2 or 3 a day). Just after upgrading to 11.1-release (all packages have also been upgraded), during this period OVH seemed also to have ipv6 problems.  
Panic just stopped two days ago (ipv6 is also working again at ovh), So I guess it was perhaps related to ipv6?

Below are 2 crash dump (I can provide the full crash log if requested)


----------------------------------------------------

Fatal trap 12: page fault while in kernel mode
cpuid = 6; apic id = 06
fault virtual address	= 0x3b8
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff805213df
stack pointer	        = 0x28:0xfffffe085ddfd090
frame pointer	        = 0x28:0xfffffe085ddfd120
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 971 (ping)
trap number		= 12
panic: page fault
cpuid = 6
Uptime: 11h52m49s
Dumping 1629 out of 32659 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%


#0  doadump (textdump=<value optimized out>) at pcpu.h:222
222	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:222
#1  0xffffffff805251d1 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80525656 in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff805254c3 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:690
#4  0xffffffff80810942 in trap_fatal (frame=0xfffffe085ddfcfd0, eva=952)
    at /usr/src/sys/amd64/amd64/trap.c:801
#5  0xffffffff80810be0 in trap_pfault (frame=0xfffffe085ddfcfd0, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:717
#6  0xffffffff8081018f in trap (frame=0xfffffe085ddfcfd0)
    at /usr/src/sys/amd64/amd64/trap.c:421
#7  0xffffffff807f62e1 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:236
#8  0xffffffff805213df in __rw_wlock_hard (c=0xfffff8001d451310, 
    v=<value optimized out>, tid=18446735279442007392, 
    file=<value optimized out>, line=<value optimized out>)
    at /usr/src/sys/kern/kern_rwlock.c:903
#9  0xffffffff806fc5c1 in in6_lltable_lookup (llt=<value optimized out>, 
    flags=8192, l3addr=<value optimized out>)
    at /usr/src/sys/netinet6/in6.c:2275
#10 0xffffffff8062baa9 in llentry_alloc (ifp=0xfffff8000740c000, 
    lt=0xfffff8000790e700, dst=0xfffff8001fe84520) at if_llatbl.h:253
#11 0xffffffff8061eda0 in flowtable_lookup_common (ft=0xffffffff80d107b8, 
    key=0xfffffe085ddfd280, keylen=16, fibnum=0)
    at /usr/src/sys/net/flowtable.c:588
#12 0xffffffff8061e3c5 in flowtable_lookup (sa=<value optimized out>, 
    m=0xfffff8000bf9d500, ro=0xfffffe085ddfd3b8)
    at /usr/src/sys/net/flowtable.c:395
#13 0xffffffff8070c2ee in ip6_output (m0=<value optimized out>, 
    opt=<value optimized out>, ro=0xfffffe085ddfd3b8, flags=0, im6o=0x0, 
    ifpp=0xfffffe085ddfd690, inp=<value optimized out>)
    at /usr/src/sys/netinet6/ip6_output.c:506
#14 0xffffffff8071fd7f in rip6_output (m=0xfffff8000bf9d500, 
    so=0xfffff80033950360) at /usr/src/sys/netinet6/raw_ip6.c:538
#15 0xffffffff80720c24 in rip6_send (so=0xfffff80033950360, 
    flags=<value optimized out>, m=0xfffff8000bf9d500, 
    nam=<value optimized out>, control=0x0, td=<value optimized out>)
    at /usr/src/sys/netinet6/raw_ip6.c:890
#16 0xffffffff805b0a6c in sosend_generic (so=<value optimized out>, 
    addr=0xfffff801e9cb5e40, uio=<value optimized out>, 
    top=<value optimized out>, control=<value optimized out>, 
    flags=<value optimized out>, td=<value optimized out>)
    at /usr/src/sys/kern/uipc_socket.c:1360
#17 0xffffffff805b7341 in kern_sendit (td=<value optimized out>, 
    s=<value optimized out>, mp=0xfffffe085ddfd8e0, flags=0, control=0x0, 
    segflg=UIO_USERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:873
#18 0xffffffff805b76a3 in sendit (td=0xfffff8006cce9560, 
    s=<value optimized out>, mp=0xfffffe085ddfd8e0, 
    flags=<value optimized out>) at /usr/src/sys/kern/uipc_syscalls.c:793
#19 0xffffffff805b74ed in sys_sendto (td=<value optimized out>, 
    uap=<value optimized out>) at /usr/src/sys/kern/uipc_syscalls.c:924
#20 0xffffffff808114a4 in amd64_syscall (td=0xfffff8006cce9560, traced=0)
    at subr_syscall.c:135
#21 0xffffffff807f65cb in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:396
#22 0x000000080130bd3a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb) 

------------------------------------------------------------------------

Fatal trap 12: page fault while in kernel mode
cpuid = 6; apic id = 06
fault virtual address	= 0x3b8
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff805213df
stack pointer	        = 0x28:0xfffffe085d537920
frame pointer	        = 0x28:0xfffffe085d5379b0
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 28 (flowcleaner)
trap number		= 12
panic: page fault
cpuid = 6
Uptime: 4h10m43s
Dumping 1602 out of 32659 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

#0  doadump (textdump=<value optimized out>) at pcpu.h:222
222	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:222
#1  0xffffffff805251d1 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80525656 in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff805254c3 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:690
#4  0xffffffff80810942 in trap_fatal (frame=0xfffffe085d537860, eva=952)
    at /usr/src/sys/amd64/amd64/trap.c:801
#5  0xffffffff80810be0 in trap_pfault (frame=0xfffffe085d537860, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:717
#6  0xffffffff8081018f in trap (frame=0xfffffe085d537860)
    at /usr/src/sys/amd64/amd64/trap.c:421
#7  0xffffffff807f62e1 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:236
#8  0xffffffff805213df in __rw_wlock_hard (c=0xfffff80024079510, 
    v=<value optimized out>, tid=18446735277952892928, 
    file=<value optimized out>, line=<value optimized out>)
    at /usr/src/sys/kern/kern_rwlock.c:903
#9  0xffffffff8061e8df in flowtable_clean_vnet (ft=<value optimized out>, 
    rt=0x0, maxidle=<value optimized out>) at /usr/src/sys/net/flowtable.c:827
#10 0xffffffff8061f41e in flowtable_cleaner ()
    at /usr/src/sys/net/flowtable.c:904
#11 0xffffffff804e8f85 in fork_exit (
    callout=0xffffffff8061f380 <flowtable_cleaner>, arg=0x0, 
    frame=0xfffffe085d537ac0) at /usr/src/sys/kern/kern_fork.c:1042
#12 0xffffffff807f681e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:611
#13 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb) 

------------------------------------------------------------------------

Comment 1 Andrey V. Elsukov freebsd_committer

2017-07-31 14:25:10 UTC

Did you try to remove option FLOWTABLE from the kernel?

Comment 2 Stéphane D'Alu 2017-07-31 16:22:31 UTC

I didn't try removing FLOWTABLE, I will  if I got another crash.

Comment 3 Andrey V. Elsukov freebsd_committer

2017-07-31 17:45:50 UTC

(In reply to sdalu from comment #2)
> I didn't try removing FLOWTABLE, I will  if I got another crash.

Both your panics are related to FLOWTABLE. Also, note that this option was removed already from FreeBSD 12.0-CURRENT.

Comment 4 Stéphane D'Alu 2017-08-01 08:22:04 UTC

Thanks, I'm removing FLOWTABLE.
If it's already removed in FreeBSD 12, not sure there is a point to keep this bug open.

Comment 5 Eugene Grosbein freebsd_committer

2017-08-01 19:43:53 UTC

Close this as FLOWTABLE deorbit sequence has already been initiated and it is not present in the GENERIC for any supported release.