Description Eugene Grosbein 2021-11-17 06:56:44 UTC

One of our servers has been upgraded from 11.4-STABLE/amd64 (completely stable, no panics) to 12.3-PRERELEASE recently. Since then, it panices every several days. I managed to collect crashdump. Unfortunately, the system built with WITHOUT_KERNEL_SYMBOLS, so I have kernel.debug for the kernel but no debugging symbols for modules.

Here are kernel config file and kgdb session:

include GENERIC
ident           Base12
options         IPDIVERT
options         IPFIREWALL
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=100
options         IPFIREWALL_DEFAULT_TO_ACCEPT
device          enc
# EOF

GNU gdb (GDB) 11.1 [GDB v11.1 for FreeBSD]
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.3".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from kernel.debug...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 5; apic id = 15
fault virtual address   = 0x1c
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80c6fab6
stack pointer           = 0x28:0xfffffe009c6ce9d0
frame pointer           = 0x28:0xfffffe009c6cea10
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 1558 (ng_queue2)
trap number             = 12
panic: page fault
cpuid = 5
time = 1637072171
KDB: stack backtrace:
#0 0xffffffff80c30235 at kdb_backtrace+0x65
#1 0xffffffff80be43cb at vpanic+0x17b
#2 0xffffffff80be4243 at panic+0x43
#3 0xffffffff811239a1 at trap_fatal+0x391
#4 0xffffffff811239ff at trap_pfault+0x4f
#5 0xffffffff81123046 at trap+0x286
#6 0xffffffff810fadc8 at calltrap+0x8
#7 0xffffffff82b4c023 at ng_l2tp_seq_rack_timeout+0x203
#8 0xffffffff82b2d5c9 at ng_apply_item+0xd9
#9 0xffffffff82b302a9 at ngthread+0x1e9
#10 0xffffffff80ba579e at fork_exit+0x7e
#11 0xffffffff810fbdfe at fork_trampoline+0xe
Uptime: 10d7h48m54s
Dumping 6651 out of 24261 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:371
#2  0xffffffff80be3fe5 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452
#3  0xffffffff80be4423 in vpanic (fmt=<optimized out>, ap=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:881
#4  0xffffffff80be4243 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:808
#5  0xffffffff811239a1 in trap_fatal (frame=0xfffffe009c6ce910, eva=28)
    at /usr/src/sys/amd64/amd64/trap.c:921
#6  0xffffffff811239ff in trap_pfault (frame=0xfffffe009c6ce910, usermode=<optimized out>,
    signo=<optimized out>, ucode=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:739
#7  0xffffffff81123046 in trap (frame=0xfffffe009c6ce910) at /usr/src/sys/amd64/amd64/trap.c:405
#8  <signal handler called>
#9  m_copypacket (m=0x0, how=1) at /usr/src/sys/kern/uipc_mbuf.c:539
#10 0xffffffff82b4c023 in ng_l2tp_seq_rack_timeout () from /boot/kernel/ng_l2tp.ko
#11 0xffffffff82b2d5c9 in ng_apply_item () from /boot/kernel/netgraph.ko
#12 0xffffffff82b302a9 in ngthread () from /boot/kernel/netgraph.ko
#13 0xffffffff80ba579e in fork_exit (callout=0xfffff805c8026300, arg=0xfffff8057303f090,
    frame=0xfffff8002babe740) at /usr/src/sys/kern/kern_fork.c:1080
#14 <signal handler called>
(kgdb) frame 10
#10 0xffffffff82b4c023 in ng_l2tp_seq_rack_timeout () from /boot/kernel/ng_l2tp.ko
(kgdb) l
50      static __inline __pure2 struct thread *
51      __curthread(void)
52      {
53              struct thread *td;
54
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
56                  pc_curthread)));
57              return (td);
58      }
59      #define curthread               (__curthread())
(kgdb)