Bug 271772 - ZFS with L2ARC: panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed
Summary: ZFS with L2ARC: panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: Martin Matuska
URL:
Keywords: crash
Depends on:
Blocks: 14.0r
  Show dependency tree / graph
 
Reported: 2023-06-01 17:34 UTC by Graham Perrin
Modified: 2023-06-17 06:56 UTC (History)
3 users (show)

See Also:

Attachments
openzfs/zfs commit bcd5321 adapated for application to FreeBSD src (4.42 KB, patch)
2023-06-08 06:32 UTC, Graham Perrin 		no flags Details | Diff
Screenshot of Netdata for the period 10th–17th June (93.83 KB, image/png)
2023-06-17 03:18 UTC, Graham Perrin 		no flags Details
Show Obsolete (1) View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Graham Perrin freebsd_committer freebsd_triage 2023-06-01 17:34:05 UTC 
Five panics soon after attempting to use GENERIC main-n263189-c2c9ac88c2bb-dirty. 

No panic since I switched to GENERIC-NODEBUG. 

The first of five: 

Dumptime: 2023-05-27 03:17:16 +0100

----

Dump header from device: /dev/ada0p2
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 1840246784
  Blocksize: 512
  Compression: none
  Dumptime: 2023-05-27 03:17:16 +0100
  Hostname: mowa219-gjp4-8570p-freebsd
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 14.0-CURRENT #3 main-n263189-c2c9ac88c2bb-dirty: Sat May 27 01:04:23 BST 2023
    grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
  Panic String: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (14053755392 <= 14053740032)

  Dump Parity: 1896504401
  Bounds: 1
  Dump Status: good

----

…

Unread portion of the kernel message buffer:
panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (14053755392 <= 14053740032)

cpuid = 2
time = 1685153836
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c6bd8c90
vpanic() at vpanic+0x150/frame 0xfffffe00c6bd8ce0
spl_panic() at spl_panic+0x3a/frame 0xfffffe00c6bd8d40
l2arc_feed_thread() at l2arc_feed_thread+0x2401/frame 0xfffffe00c6bd8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00c6bd8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c6bd8f30
--- trap 0x2f2a2072, rip = 0x949504154415f4d, rsp = 0x7265747061644120, rbp = 0x52505f4253550965 ---
KDB: enter: panic
Uptime: 5m47s
Dumping 1754 out of 16260 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:59
59              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:59
#1  doadump (textdump=textdump@entry=1)
    at /usr/src/sys/kern/kern_shutdown.c:407
#2  0xffffffff80b5102c in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:528
#3  0xffffffff80b5154d in vpanic (fmt=<optimized out>, 
    ap=ap@entry=0xfffffe00c6bd8d20) at /usr/src/sys/kern/kern_shutdown.c:972
#4  0xffffffff81f9327a in spl_panic (file=<optimized out>, 
    func=<optimized out>, line=<unavailable>, fmt=<unavailable>)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_misc.c:103
#5  0xffffffff81fef1e1 in l2arc_write_buffers (spa=0xfffffe0110b5d000, 
    dev=0x0, target_sz=16777216)
    at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:9445
#6  l2arc_feed_thread (unused=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:9559
#7  0xffffffff80b077b0 in fork_exit (
    callout=0xffffffff81fecde0 <l2arc_feed_thread>, arg=0x0, 
    frame=0xfffffe00c6bd8f40) at /usr/src/sys/kern/kern_fork.c:1102
#8  <signal handler called>
#9  0x0949504154415f4d in ?? ()
Backtrace stopped: Cannot access memory at address 0x7265747061644120
(kgdb) 

------------------------------------------------------------------------
ps -axlww

UID   PID  PPID C PRI  NI    VSZ               RSS MWCHAN   STAT TT      TIME COMMAND
  0     0     0 0 -16   0      0 18005604586963200 swapin   DLs   -   0:45.86 [kernel]
…
Comment 1 Graham Perrin freebsd_committer freebsd_triage 2023-06-01 17:36:02 UTC 
Second of the five panics: 

Dumptime: 2023-05-27 03:41:03 +0100

----

Dump header from device: /dev/ada0p2
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 2013057024
  Blocksize: 512
  Compression: none
  Dumptime: 2023-05-27 03:41:03 +0100
  Hostname: mowa219-gjp4-8570p-freebsd
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 14.0-CURRENT #3 main-n263189-c2c9ac88c2bb-dirty: Sat May 27 01:04:23 BST 2023
    grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
  Panic String: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (6704865280 <= 6704857600)

  Dump Parity: 4196367481
  Bounds: 2
  Dump Status: good

----

…

Unread portion of the kernel message buffer:
.
<118>.
<118>Terminated
<118>May 27 03:40:26 mowa219-gjp4-8570p-freebsd syslogd: exiting on signal 15
<6>wlan0: link state changed to DOWN
<118>Enter full pathname of shell or RETURN for /bin/sh: # mount -uw /
<118># zfs mount -a
<118># zpool offli# zfs mount -a# zpool online august gpt/cache-august
<118># zpool online august gpt/cache2-august
<118># zpool online august gpt/cache3-august
panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (6704865280 <= 6704857600)

cpuid = 2
time = 1685155263
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c6bd8c90
vpanic() at vpanic+0x150/frame 0xfffffe00c6bd8ce0
spl_panic() at spl_panic+0x3a/frame 0xfffffe00c6bd8d40
l2arc_feed_thread() at l2arc_feed_thread+0x2401/frame 0xfffffe00c6bd8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00c6bd8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c6bd8f30
--- trap 0x2f2a2072, rip = 0x949504154415f4d, rsp = 0x7265747061644120, rbp = 0x52505f4253550965 ---
KDB: enter: panic
Uptime: 22m7s
Dumping 1919 out of 16260 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:59
59              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:59
#1  doadump (textdump=textdump@entry=1)
    at /usr/src/sys/kern/kern_shutdown.c:407
#2  0xffffffff80b5102c in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:528
#3  0xffffffff80b5154d in vpanic (fmt=<optimized out>, 
    ap=ap@entry=0xfffffe00c6bd8d20) at /usr/src/sys/kern/kern_shutdown.c:972
#4  0xffffffff8205727a in spl_panic (file=<optimized out>, 
    func=<optimized out>, line=<unavailable>, fmt=<unavailable>)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_misc.c:103
#5  0xffffffff820b31e1 in l2arc_write_buffers (spa=0xfffffe0110b77000, 
    dev=0x0, target_sz=16777216)
    at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:9445
#6  l2arc_feed_thread (unused=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:9559
#7  0xffffffff80b077b0 in fork_exit (
    callout=0xffffffff820b0de0 <l2arc_feed_thread>, arg=0x0, 
    frame=0xfffffe00c6bd8f40) at /usr/src/sys/kern/kern_fork.c:1102
#8  <signal handler called>
#9  0x0949504154415f4d in ?? ()
Backtrace stopped: Cannot access memory at address 0x7265747061644120
(kgdb) 

------------------------------------------------------------------------
ps -axlww

UID   PID  PPID C PRI NI   VSZ               RSS MWCHAN   STAT TT      TIME COMMAND
  0     0     0 3 -16  0     0 18005604586963200 swapin   DLs   -   3:57.46 [kernel]
…
Comment 2 Graham Perrin freebsd_committer freebsd_triage 2023-06-01 17:36:57 UTC 
Third of the five: 

Dumptime: 2023-05-27 14:03:32 +0100

----

Dump header from device: /dev/ada0p2
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 3069304832
  Blocksize: 512
  Compression: none
  Dumptime: 2023-05-27 14:03:32 +0100
  Hostname: mowa219-gjp4-8570p-freebsd
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 14.0-CURRENT #3 main-n263189-c2c9ac88c2bb-dirty: Sat May 27 01:04:23 BST 2023
    grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
  Panic String: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (3680984576 <= 3680983552)

  Dump Parity: 3867178103
  Bounds: 3
  Dump Status: good

----

…

Unread portion of the kernel message buffer:
panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (3680984576 <= 3680983552)

cpuid = 0
time = 1685192612
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c6bd8c90
vpanic() at vpanic+0x150/frame 0xfffffe00c6bd8ce0
spl_panic() at spl_panic+0x3a/frame 0xfffffe00c6bd8d40
l2arc_feed_thread() at l2arc_feed_thread+0x2401/frame 0xfffffe00c6bd8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00c6bd8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c6bd8f30
--- trap 0xd76fc700, rip = 0xa0243d076ea0a69f, rsp = 0xf8b58c250ce46217, rbp = 0x3c30052e1ef2463b ---
KDB: enter: panic
Uptime: 23m51s
Dumping 2927 out of 16260 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:59
59              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:59
#1  doadump (textdump=textdump@entry=1)
    at /usr/src/sys/kern/kern_shutdown.c:407
#2  0xffffffff80b5102c in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:528
#3  0xffffffff80b5154d in vpanic (fmt=<optimized out>, 
    ap=ap@entry=0xfffffe00c6bd8d20) at /usr/src/sys/kern/kern_shutdown.c:972
#4  0xffffffff82ca827a in spl_panic (file=<optimized out>, 
    func=<optimized out>, line=<unavailable>, fmt=<unavailable>)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_misc.c:103
#5  0xffffffff82d041e1 in l2arc_write_buffers (spa=0xfffffe0110b20000, 
    dev=0x0, target_sz=16777216)
    at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:9445
#6  l2arc_feed_thread (unused=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:9559
#7  0xffffffff80b077b0 in fork_exit (
    callout=0xffffffff82d01de0 <l2arc_feed_thread>, arg=0x0, 
    frame=0xfffffe00c6bd8f40) at /usr/src/sys/kern/kern_fork.c:1102
#8  <signal handler called>
#9  0xa0243d076ea0a69f in ?? ()
Backtrace stopped: Cannot access memory at address 0xf8b58c250ce46217
(kgdb) 

------------------------------------------------------------------------
ps -axlww

 UID   PID  PPID C PRI  NI      VSZ               RSS MWCHAN   STAT TT      TIME COMMAND
   0     0     0 0 -16   0        0 18005604586963200 swapin   DLs   -   1:14.97 [kernel]
…
Comment 3 Graham Perrin freebsd_committer freebsd_triage 2023-06-01 17:38:16 UTC 
Fourth of the five: 

Dump header from device: /dev/ada0p2
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 2233499648
  Blocksize: 512
  Compression: none
  Dumptime: 2023-05-27 17:00:10 +0100
  Hostname: mowa219-gjp4-8570p-freebsd
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 14.0-CURRENT #3 main-n263189-c2c9ac88c2bb-dirty: Sat May 27 01:04:23 BST 2023
    grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
  Panic String: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (5833055232 <= 5833052672)

  Dump Parity: 1230073968
  Bounds: 4
  Dump Status: good
Comment 4 Graham Perrin freebsd_committer freebsd_triage 2023-06-01 17:42:08 UTC 
Fifth. 

(This panic occurred so soon after the fourth, above, that I did not get a core.txt for the fourth.) 

Dumptime: 2023-05-27 17:03:59 +0100

----

Dump header from device: /dev/ada0p2
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 1065807872
  Blocksize: 512
  Compression: none
  Dumptime: 2023-05-27 17:03:59 +0100
  Hostname: mowa219-gjp4-8570p-freebsd
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 14.0-CURRENT #3 main-n263189-c2c9ac88c2bb-dirty: Sat May 27 01:04:23 BST 2023
    grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
  Panic String: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (5895209984 <= 5895196160)

  Dump Parity: 2874308210
  Bounds: 5
  Dump Status: good

----

…

Unread portion of the kernel message buffer:
panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (5895209984 <= 5895196160)

cpuid = 3
time = 1685203439
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c6bd8c90
vpanic() at vpanic+0x150/frame 0xfffffe00c6bd8ce0
spl_panic() at spl_panic+0x3a/frame 0xfffffe00c6bd8d40
l2arc_feed_thread() at l2arc_feed_thread+0x2401/frame 0xfffffe00c6bd8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00c6bd8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c6bd8f30
--- trap 0xd76fc700, rip = 0xa0243d076ea0a69f, rsp = 0xf8b58c250ce46217, rbp = 0x3c30052e1ef2463b ---
KDB: enter: panic
Uptime: 2m15s
Dumping 1016 out of 16260 MB:..2%..12%..21%..32%..41%..51%..62%..71%..81%..92%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:59
59              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:59
#1  doadump (textdump=textdump@entry=1)
    at /usr/src/sys/kern/kern_shutdown.c:407
#2  0xffffffff80b5102c in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:528
#3  0xffffffff80b5154d in vpanic (fmt=<optimized out>, 
    ap=ap@entry=0xfffffe00c6bd8d20) at /usr/src/sys/kern/kern_shutdown.c:972
#4  0xffffffff8207727a in spl_panic (file=<optimized out>, 
    func=<optimized out>, line=<unavailable>, fmt=<unavailable>)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_misc.c:103
#5  0xffffffff820d31e1 in l2arc_write_buffers (spa=0xfffffe0110b51000, 
    dev=0x0, target_sz=16777216)
    at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:9445
#6  l2arc_feed_thread (unused=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:9559
#7  0xffffffff80b077b0 in fork_exit (
    callout=0xffffffff820d0de0 <l2arc_feed_thread>, arg=0x0, 
    frame=0xfffffe00c6bd8f40) at /usr/src/sys/kern/kern_fork.c:1102
#8  <signal handler called>
#9  0xa0243d076ea0a69f in ?? ()
Backtrace stopped: Cannot access memory at address 0xf8b58c250ce46217
(kgdb) 

------------------------------------------------------------------------
ps -axlww

UID  PID PPID C PRI NI   VSZ               RSS MWCHAN   STAT TT     TIME COMMAND
  0    0    0 3 -16  0     0 18005604586963200 swapin   DLs   -  1:52.75 [kernel]
…
Comment 5 Graham Perrin freebsd_committer freebsd_triage 2023-06-01 17:44:07 UTC 
From <https://lists.freebsd.org/archives/freebsd-current/2023-May/003736.html> (2023-05-27): 

The first panic occurred during installworld, uptime five minutes.

The second was in single user mode, when I took the third of three L2ARC 
devices online, uptime twenty-two minutes.

The third panic was whilst using Plasma, without the device that 
featured in the second panic, uptime twenty-three minutes.

The result of a probe at 07:49 UTC: 
<https://bsd-hardware.info/?probe=65376d6b42>
Comment 6 Graham Perrin freebsd_committer freebsd_triage 2023-06-06 06:28:44 UTC 
Tentatively blocking tracking bug 271607, although I suspect that this panic will not occur for users of (non-debug) GENERIC in 14.0-RELEASE. 

Someone with proper knowledge might like to clear the block. 

The fix (pull request) for this confirmed bug is approved upstream at <https://github.com/openzfs/zfs/pull/14939#pullrequestreview-1463435242>.
Comment 7 Graham Perrin freebsd_committer freebsd_triage 2023-06-08 06:32:18 UTC 
Created attachment 242680 [details]
openzfs/zfs commit bcd5321 adapated for application to FreeBSD src

Origin: 

<https://github.com/openzfs/zfs/commit/bcd5321039c3de29c14eac1068d392c15ad7fe2c.patch>

Adaptations: 

* the path in the three a/b lines

* addition of the problem report (PR) number

* addition of a non-ambiguous URL for this bug report

* non-ambiguous https://github.com/openzfs/zfs/pull/14939 
  in lieu of #14939

----

From <https://github.com/openzfs/zfs/issues/14936#issuecomment-1581696959> a few minutes ago: 

> … assert just fired …
Comment 8 Gleb Smirnoff freebsd_committer freebsd_triage 2023-06-09 16:48:54 UTC 
The non-debug kernel also panics after latest OpenZFS merge, but not so often as INVARIANTS. The panic looks like:

--- trap 0x9, rip = 0xffffffff803ab94b, rsp = 0xfffffe022e45ed30, rbp = 0xfffffe022e45ed50 ---
buf_hash_insert() at buf_hash_insert+0xab/frame 0xfffffe022e45ed50
arc_write_done() at arc_write_done+0xfa/frame 0xfffffe022e45ed90
zio_done() at zio_done+0xf0b/frame 0xfffffe022e45ee00
zio_execute() at zio_execute+0x9f/frame 0xfffffe022e45ee40
taskqueue_run_locked() at taskqueue_run_locked+0x181/frame 0xfffffe022e45eec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xc3/frame 0xfffffe022e45eef0
fork_exit() at fork_exit+0x7d/frame 0xfffffe022e45ef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe022e45ef30
Comment 9 Gleb Smirnoff freebsd_committer freebsd_triage 2023-06-09 16:49:56 UTC 
Email from Martin:

https://lists.freebsd.org/archives/freebsd-current/2023-June/003833.html
Comment 10 Graham Perrin freebsd_committer freebsd_triage 2023-06-10 07:54:08 UTC 
(In reply to Gleb Smirnoff from comment #8)

Thanks, at a glance that might be a different bug/panic; and 

(In reply to Gleb Smirnoff from comment #9)

I'll follow up on the list.
Comment 11 Gleb Smirnoff freebsd_committer freebsd_triage 2023-06-10 08:14:03 UTC 
Both fixes were pushed to OpenZFS/master, but only the first one was merged to FreeBSD/main.
Comment 12 Graham Perrin freebsd_committer freebsd_triage 2023-06-10 12:14:38 UTC 
Comment on attachment 242680 [details]
openzfs/zfs commit bcd5321 adapated for application to FreeBSD src

I have been using this patch (alone) since yesterday morning, 9th June. No panic. 

% bectl list -c creation | grep 267411d164d4
n263447-267411d164d4-a -      -          37.6M 2023-06-09 04:50
n263447-267411d164d4-b NR     /          436G  2023-06-09 08:37
% uptime
 1:11p.m.  up 14:59, 7 users, load averages: 1.50, 1.37, 1.37
% uname -aKU
FreeBSD mowa219-gjp4-8570p-freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #5 main-n263447-267411d164d4-dirty: Thu Jun  8 14:26:44 BST 2023     grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 1400090 1400090
%
Comment 13 Graham Perrin freebsd_committer freebsd_triage 2023-06-10 12:20:15 UTC 
(In reply to Graham Perrin from comment #7)

> From <https://github.com/openzfs/zfs/issues/14936#issuecomment-1581696959> 
> a few minutes ago: 
> 
>> … assert just fired …

This morning: 

<https://github.com/openzfs/zfs/commit/feff9dfed3df1bbae5dd74959a6ad87d11f27ffb>
Comment 14 Graham Perrin freebsd_committer freebsd_triage 2023-06-10 12:23:29 UTC 
(In reply to Gleb Smirnoff from comment #11)

> … the first one was merged to FreeBSD/main.

Which one, please?
Comment 15 Martin Matuska freebsd_committer freebsd_triage 2023-06-10 17:34:19 UTC 
Please try with main 4e8d558c9d1cf3e7e424e3fb123b01979c3d57f2 or later
Comment 16 Graham Perrin freebsd_committer freebsd_triage 2023-06-10 17:49:03 UTC 
Comment on attachment 242680 [details]
openzfs/zfs commit bcd5321 adapated for application to FreeBSD src

Obsolete; superseded by <https://cgit.freebsd.org/src/commit/?id=4e8d558c9d1cf3e7e424e3fb123b01979c3d57f2> – thanks to mm@
Comment 17 Graham Perrin freebsd_committer freebsd_triage 2023-06-17 03:18:24 UTC 
Created attachment 242818 [details]
Screenshot of Netdata for the period 10th–17th June

(In reply to Martin Matuska from comment #15, 2023-06-10)

> Please try with main 4e8d558c9d1cf3e7e424e3fb123b01979c3d57f2 or later

Good for me. Thanks!

----

The one panic below was, I think, unrelated to L2ARC (bug 271945 – whilst using earlier 267411d164d4-dirty to build world for 4e8d558c9d1c).


% zgrep BOOT /var/log/messages.1.bz2
Jun  9 05:08:01 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
Jun  9 05:21:01 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
Jun  9 08:52:40 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
Jun 11 05:24:09 mowa219-gjp4-8570p-freebsd kernel: Dumping 7112 out of 16260 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%---<<BOOT>>---
Jun 11 05:48:01 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
Jun 11 08:47:38 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
Jun 11 09:04:48 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
Jun 11 09:09:18 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
% zgrep BOOT /var/log/messages.0.bz2
Jun 11 13:29:13 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
Jun 11 13:50:04 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
Jun 13 06:56:12 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
Jun 15 00:41:53 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
% grep BOOT /var/log/messages
Jun 15 01:12:04 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
Jun 16 02:44:28 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
% bectl list -c creation
BE                     Active Mountpoint Space Created
n262604-7ee4066db129-b -      -          12.4G 2023-04-29 21:29
n263189-c2c9ac88c2bb-b -      -          1.45G 2023-05-28 18:43
n263189-c2c9ac88c2bb-c -      -          544M  2023-06-01 06:32
n263189-c2c9ac88c2bb-d -      -          127M  2023-06-03 07:58
n263402-086e0149ae56-a -      -          168M  2023-06-04 17:40
n263402-086e0149ae56-b -      -          45.7M 2023-06-06 06:45
n263447-267411d164d4-a -      -          55.5M 2023-06-09 04:50
n263447-267411d164d4-b -      -          392M  2023-06-09 08:37
n263493-4e8d558c9d1c-a -      -          194M  2023-06-11 08:38
n263493-4e8d558c9d1c-b NR     /          407G  2023-06-15 00:20
% date ; uptime ; uname -aKU
Sat 17 Jun 2023 04:18:00 BST
 4:18a.m.  up 14:33, 8 users, load averages: 0.78, 0.99, 1.11
FreeBSD mowa219-gjp4-8570p-freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #6 main-n263493-4e8d558c9d1c-dirty: Sun Jun 11 06:22:01 BST 2023     grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 1400090 1400090
% 


Re: the attached screenshot, the sum of panic-free uptimes since I began running 4e8d558c9d1c might be around four days.
Comment 18 Martin Matuska freebsd_committer freebsd_triage 2023-06-17 06:56:40 UTC 
I consider this bug fixed.