I just recently upgraded our fileserver to FreeBSD 15.1-RELEASE and this evening I got 3 kernel panics in a fairly short amount of time. This is during a time when most people are away on vacation so nothing much is happening...

The panics:

kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 32; apic id = 20
fault virtual address   = 0x18
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff80b6306f
stack pointer           = 0x28:0xfffffe04271f0bd0
frame pointer           = 0x28:0xfffffe04271f0be0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 0 (thread taskq)
rdi: fffffe04333b6ad0 rsi: ffffffff81fffd98 rdx: fffff80105dfab60
rcx: 0000000000000000  r8: 0000000000000001  r9: fffffe04333b1ec0
rax: 0000000000000000 rbx: fffffe04333b6ad0 rbp: fffffe04271f0be0
r10: 0000006040000000 r11: fffff88dda355800 r12: fffff80105dfab40
r13: 000000000000002f r14: fffff80106e06780 r15: 000000000000002f
trap number             = 12
panic: page fault
cpuid = 32
time = 1785516113
KDB: stack backtrace:
#0 0xffffffff80bd69cd at kdb_backtrace+0x5d
#1 0xffffffff80b87526 at vpanic+0x136
#2 0xffffffff80b873e3 at panic+0x43
#3 0xffffffff81098fcd at trap_pfault+0x37d
#4 0xffffffff8106f4b8 at calltrap+0x8
#5 0xffffffff80bf06a8 at propagate_priority+0x58
#6 0xffffffff80bf12b2 at turnstile_wait+0x2d2
#7 0xffffffff80b627e5 at __mtx_lock_sleep+0x175
#8 0xffffffff80f1d282 at zone_release+0x272
#9 0xffffffff80f21f89 at bucket_drain+0xb9
#10 0xffffffff80f222ca at bucket_cache_reclaim_domain+0x2aa
#11 0xffffffff80f229eb at zone_timeout+0x2eb
#12 0xffffffff80f22688 at uma_timeout+0x58
#13 0xffffffff80bed4c2 at taskqueue_run_locked+0x182
#14 0xffffffff80bee692 at taskqueue_thread_loop+0xc2
#15 0xffffffff80b3d97b at fork_exit+0x7b
#16 0xffffffff810704de at fork_trampoline+0xe
Uptime: 10h56m11s



panic: VERIFY3U(hdr->b_type, ==, type) failed (1158217073 == 1)

cpuid = 16
time = 1785518044
KDB: stack backtrace:
#0 0xffffffff80bd69cd at kdb_backtrace+0x5d
#1 0xffffffff80b87526 at vpanic+0x136
#2 0xffffffff8260f6aa at spl_panic+0x3a
#3 0xffffffff8264e01b at arc_buf_destroy_impl+0x1bb
#4 0xffffffff8264dccf at arc_buf_destroy+0x6f
#5 0xffffffff826697a9 at dbuf_destroy+0x29
#6 0xffffffff8266f8cb at dbuf_evict_one+0x11b
#7 0xffffffff8266760d at dbuf_evict_thread+0x31d
#8 0xffffffff80b3d97b at fork_exit+0x7b
#9 0xffffffff810704de at fork_trampoline+0xe
Uptime: 28m11s



Fatal trap 9: general protection fault while in kernel mode
cpuid = 59; apic id = 3b
instruction pointer     = 0x20:0xffffffff8262da34
stack pointer           = 0x28:0xfffffe04382d9d70
frame pointer           = 0x28:0xfffffe04382d9ef0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 6 (l2arc_feed_thread)
rdi: fffff86043a09aa8 rsi: fffff870000073a0 rdx: 7dd33688f89d9229
rcx: 7dd33688f89d9291  r8: 3804bba0ece8a263  r9: db1c330e10fdf96e
rax: 7dd33688f89d9229 rbx: ffffffff828e0300 rbp: fffffe04382d9ef0
r10: 0000000000000011 r11: 0000000000030000 r12: fffff86048ece000
r13: fffffe085613b000 r14: 0000000007c09800 r15: 7dd33688f89d9229
trap number             = 9
panic: general protection fault
cpuid = 59
time = 1785519857
KDB: stack backtrace:
#0 0xffffffff80bd69cd at kdb_backtrace+0x5d
#1 0xffffffff80b87526 at vpanic+0x136
#2 0xffffffff80b873e3 at panic+0x43
#3 0xffffffff81098b88 at trap_fatal+0x68
#4 0xffffffff8106f4b8 at calltrap+0x8
#5 0xffffffff80b3d97b at fork_exit+0x7b
#6 0xffffffff810704de at fork_trampoline+0xe
Uptime: 26m20s



I noticed in another bug report talks about dedups and block_cloneing. We do not have dedup enabled, but apparently bclone is enabled:

# zpool get all zroot DATA FILUR07D1 |egrep 'block_clon|bclone|dedup'
DATA       dedupratio                     1.00x                          -
DATA       bcloneused                     1.92M                          -
DATA       bclonesaved                    1.92M                          -
DATA       bcloneratio                    2.00x                          -
DATA       dedup_table_size               0                              -
DATA       dedup_table_quota              auto                           default
DATA       feature@block_cloning          active                         local
DATA       feature@fast_dedup             enabled                        local
DATA       feature@block_cloning_endian   active                         local
FILUR07D1  dedupratio                     1.00x                          -
FILUR07D1  bcloneused                     0                              -
FILUR07D1  bclonesaved                    0                              -
FILUR07D1  bcloneratio                    1.00x                          -
FILUR07D1  dedup_table_size               0                              -
FILUR07D1  dedup_table_quota              auto                           default
FILUR07D1  feature@block_cloning          enabled                        local
FILUR07D1  feature@fast_dedup             enabled                        local
FILUR07D1  feature@block_cloning_endian   enabled                        local
zroot      dedupratio                     1.00x                          -
zroot      bcloneused                     0                              -
zroot      bclonesaved                    0                              -
zroot      bcloneratio                    1.00x                          -
zroot      dedup_table_size               0                              -
zroot      dedup_table_quota              auto                           default
zroot      feature@block_cloning          enabled                        local
zroot      feature@fast_dedup             enabled                        local
zroot      feature@block_cloning_endian   enabled                        local


DATA is where all the data is stored, FILUR07D1 are new disks where we are slowly moving data to. The same morning I replaced a broken drive and added a new one as a spare to DATA and FILUR07D1 - no pool errors seen though.

I have 10 other similar servers where I've yet to see any panics/crashes... But it's summer vacation time here so pretty slow now - but that'll probably change in the next week or so, so this is a bit worrying...