Bug 253717 - panic: pfslowtimo (swi4: clock)
Summary: panic: pfslowtimo (swi4: clock)
Status: Closed Not A Bug
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.2-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-02-20 11:04 UTC by Christos Chatzaras
Modified: 2021-02-24 18:00 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christos Chatzaras 2021-02-20 11:04:17 UTC
This server was running for 3 months without issues. Also I have the same FreeBSD version, software and similar hardware in other 100 servers without seen this panic again. Vmcore is available if needed.

------------------------------------------------------------------------

server90.example.com dumped core - see /var/crash/vmcore.1

Sat Feb 20 02:29:41 EET 2021

FreeBSD server90.example.com 12.2-RELEASE-p2 FreeBSD 12.2-RELEASE-p2 GENERIC  amd64

panic: general protection fault

GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD]
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.2".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:


Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer     = 0x20:0xffffffff80c493e2
stack pointer           = 0x28:0xffffffff824b1960
frame pointer           = 0x28:0xffffffff824b1980
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (swi4: clock (0))
trap number             = 9
panic: general protection fault
cpuid = 0
time = 1613780494
KDB: stack backtrace:
#0 0xffffffff80c0a8e5 at kdb_backtrace+0x65
#1 0xffffffff80bbeb9b at vpanic+0x17b
#2 0xffffffff80bbea13 at panic+0x43
#3 0xffffffff8108f911 at trap_fatal+0x391
#4 0xffffffff8108ed97 at trap+0x67
#5 0xffffffff81066f68 at calltrap+0x8
#6 0xffffffff80bd8f61 at softclock_call_cc+0x141
#7 0xffffffff80bd9419 at softclock+0x79
#8 0xffffffff80b8361c at ithread_loop+0x23c
#9 0xffffffff80b804ce at fork_exit+0x7e
#10 0xffffffff81067f9e at fork_trampoline+0xe
Uptime: 48d23h42m37s
Dumping 4992 out of 65278 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55      /usr/src/sys/amd64/include/pcpu_aux.h: No such file or directory.
(kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:371
#2  0xffffffff80bbe7b5 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:451
#3  0xffffffff80bbebf3 in vpanic (fmt=<optimized out>, ap=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:880
#4  0xffffffff80bbea13 in panic (fmt=<unavailable>)
    at /usr/src/sys/kern/kern_shutdown.c:807
#5  0xffffffff8108f911 in trap_fatal (frame=0xffffffff824b18a0, eva=0)
    at /usr/src/sys/amd64/amd64/trap.c:921
#6  0xffffffff8108ed97 in trap (frame=0xffffffff824b18a0)
    at /usr/src/sys/amd64/amd64/trap.c:214
#7  <signal handler called>
#8  pfslowtimo (arg=<optimized out>) at /usr/src/sys/kern/uipc_domain.c:508
#9  0xffffffff80bd8f61 in softclock_call_cc (c=<optimized out>,
    cc=0xffffffff81ea2000 <cc_cpu>, direct=0)
    at /usr/src/sys/kern/kern_timeout.c:734
#10 0xffffffff80bd9419 in softclock (arg=0xffffffff81ea2000 <cc_cpu>)
    at /usr/src/sys/kern/kern_timeout.c:870
#11 0xffffffff80b8361c in intr_event_execute_handlers (p=<optimized out>,
    ie=0xfffff80003772700) at /usr/src/sys/kern/kern_intr.c:1143
#12 ithread_execute_handlers (p=<optimized out>, ie=0xfffff80003772700)
    at /usr/src/sys/kern/kern_intr.c:1156
#13 ithread_loop (arg=0xfffff800037d10e0)
    at /usr/src/sys/kern/kern_intr.c:1236
#14 0xffffffff80b804ce in fork_exit (
    callout=0xffffffff80b833e0 <ithread_loop>, arg=0xfffff800037d10e0,
    frame=0xffffffff824b1b00) at /usr/src/sys/kern/kern_fork.c:1080
#15 <signal handler called>
(kgdb)
Comment 1 Mark Johnston freebsd_committer 2021-02-20 17:32:19 UTC
From the vmcore it might be useful to see:

(kgdb) info loc
(kgdb) p *pr
(kgdb) p *dp

Was some kernel module unloaded around the time of the panic?
Comment 2 Christos Chatzaras 2021-02-23 18:36:51 UTC
I did something wrong with the commands? The /usr/src is from 12.2-RELEASE-p3 and the systen is running 12.2-RELEASE-p2. If you want I can provide SSH access to have a look.

-------

[chris@server2 ~]$ kgdb /boot/kernel/kernel vmcore.1

GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD]
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.2".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:


Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer     = 0x20:0xffffffff80c493e2
stack pointer           = 0x28:0xffffffff824b1960
frame pointer           = 0x28:0xffffffff824b1980
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (swi4: clock (0))
trap number             = 9
panic: general protection fault
cpuid = 0
time = 1613780494
KDB: stack backtrace:
#0 0xffffffff80c0a8e5 at kdb_backtrace+0x65
#1 0xffffffff80bbeb9b at vpanic+0x17b
#2 0xffffffff80bbea13 at panic+0x43
#3 0xffffffff8108f911 at trap_fatal+0x391
#4 0xffffffff8108ed97 at trap+0x67
#5 0xffffffff81066f68 at calltrap+0x8
#6 0xffffffff80bd8f61 at softclock_call_cc+0x141
#7 0xffffffff80bd9419 at softclock+0x79
#8 0xffffffff80b8361c at ithread_loop+0x23c
#9 0xffffffff80b804ce at fork_exit+0x7e
#10 0xffffffff81067f9e at fork_trampoline+0xe
Uptime: 48d23h42m37s
Dumping 4992 out of 65278 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
warning: Source file is more recent than executable.
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) info loc
td = <optimized out>
(kgdb)  p *pr
No symbol "pr" in current context.
(kgdb) p *dp
No symbol "dp" in current context.
(kgdb)
Comment 3 Christos Chatzaras 2021-02-23 18:38:05 UTC
(In reply to Mark Johnston from comment #1)

> Was some kernel module unloaded around the time of the panic?

No module unloaded around that time.
Comment 4 Mark Johnston freebsd_committer 2021-02-24 15:48:37 UTC
(In reply to Christos Chatzaras from comment #2)
Sorry, I forgot to include a step.  Before executing the other kgdb commands, type "frame 8".
Comment 5 Christos Chatzaras 2021-02-24 17:25:24 UTC
kgdb /boot/kernel/kernel vmcore.1
GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD]
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.2".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:


Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer     = 0x20:0xffffffff80c493e2
stack pointer           = 0x28:0xffffffff824b1960
frame pointer           = 0x28:0xffffffff824b1980
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (swi4: clock (0))
trap number             = 9
panic: general protection fault
cpuid = 0
time = 1613780494
KDB: stack backtrace:
#0 0xffffffff80c0a8e5 at kdb_backtrace+0x65
#1 0xffffffff80bbeb9b at vpanic+0x17b
#2 0xffffffff80bbea13 at panic+0x43
#3 0xffffffff8108f911 at trap_fatal+0x391
#4 0xffffffff8108ed97 at trap+0x67
#5 0xffffffff81066f68 at calltrap+0x8
#6 0xffffffff80bd8f61 at softclock_call_cc+0x141
#7 0xffffffff80bd9419 at softclock+0x79
#8 0xffffffff80b8361c at ithread_loop+0x23c
#9 0xffffffff80b804ce at fork_exit+0x7e
#10 0xffffffff81067f9e at fork_trampoline+0xe
Uptime: 48d23h42m37s
Dumping 4992 out of 65278 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
warning: Source file is more recent than executable.
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) frame 8
#8  pfslowtimo (arg=<optimized out>) at /usr/src/sys/kern/uipc_domain.c:508
warning: Source file is more recent than executable.
508                                     (*pr->pr_slowtimo)();
(kgdb) info loc
dp = 0xffffffff81b38c60 <inetdomain>
pr = 0xffffffff81b395d0 <inetsw+2304>
(kgdb) p *pr
$1 = {pr_type = 3, pr_domain = 0xffffffff81b38c60 <inetdomain>, pr_protocol = 0, pr_flags = 3, pr_input = 0xffffffff80d4f8d0 <rip_input>, pr_output = 0x0, pr_ctlinput = 0x0,
  pr_ctloutput = 0xffffffff80d50270 <rip_ctloutput>, pr_init = 0xffffffff80d4f810 <rip_init>, pr_fasttimo = 0x0, pr_slowtimo = 0x800000000000000, pr_drain = 0x0,
  pr_usrreqs = 0xffffffff81b3acd0 <rip_usrreqs>}
(kgdb) p *dp
$2 = {dom_family = 2, dom_name = 0xffffffff81251bb8 "internet", dom_init = 0x0, dom_destroy = 0x0, dom_externalize = 0x0, dom_dispose = 0x0, dom_protosw = 0xffffffff81b38cd0 <inetsw>,
  dom_protoswNPROTOSW = 0xffffffff81b39630 <domain_add_inet_sys_init>, dom_next = 0xffffffff81b25430 <localdomain>, dom_rtattach = 0xffffffff80d45750 <in_inithead>,
  dom_rtdetach = 0xffffffff80d45850 <in_detachhead>, dom_ifattach = 0xffffffff80d39370 <in_domifattach>, dom_ifdetach = 0xffffffff80d39430 <in_domifdetach>, dom_ifmtu = 0x0}
(kgdb)
Comment 6 Mark Johnston freebsd_committer 2021-02-24 17:36:04 UTC
So we have "pr_slowtimo = 0x800000000000000" instead of "pf_slowtimo = 0", so this looks like a single bit-flip.  Are you using ECC RAM?
Comment 7 Christos Chatzaras 2021-02-24 17:47:00 UTC
No I don't, so this is possible. I will also do a RAM test.
Comment 8 Mark Johnston freebsd_committer 2021-02-24 18:00:50 UTC
(In reply to Christos Chatzaras from comment #7)
Ok.  I will close this for now since it is most likely caused by hardware, feel free to re-open if you have some evidence otherwise.