FreeBSD Bugzilla – Attachment 182174 Details for
Bug 218954
[ZFS] Add a sysctl to toggle zfs_free_leak_on_eio
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
Remember
[x]
|
Forgot Password
Login:
[x]
[patch]
sys/cddl: Add a sysctl to toggle zfs_free_leak_on_eio
sys-cddl-Add-a-sysctl-to-toggle-zfs_free_leak_on_eio.diff (text/plain), 13.94 KB, created by
Fabian Keil
on 2017-04-29 12:19:07 UTC
(
hide
)
Description:
sys/cddl: Add a sysctl to toggle zfs_free_leak_on_eio
Filename:
MIME Type:
Creator:
Fabian Keil
Created:
2017-04-29 12:19:07 UTC
Size:
13.94 KB
patch
obsolete
>From 7259164b3bd630ba0130be17fcb3f2c1652d51d5 Mon Sep 17 00:00:00 2001 >From: Fabian Keil <fk@fabiankeil.de> >Date: Thu, 20 Apr 2017 20:07:58 +0200 >Subject: [PATCH] sys/cddl: Add a sysctl to toggle zfs_free_leak_on_eio > >Setting the sysctl allows to break a previously-endless cycle >of ZFS collecting checksum errors for metadata. > >Before setting vfs.zfs.free_leak_on_eio=1: > >fk@t520 ~ $zpool status cloudia2 > pool: cloudia2 > state: ONLINE >status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. >action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: https://illumos.org/msg/ZFS-8000-8A > scan: scrub repaired 308K in 53h23m with 3358 errors on Sun Apr 16 20:33:26 2017 >config: > > NAME STATE READ WRITE CKSUM > cloudia2 ONLINE 0 0 129 > label/cloudia2.eli ONLINE 0 0 516 > >errors: 3362 data errors, use '-v' for a list > >fk@t520 ~ $zpool status -v cloudia2 >[..] >errors: Permanent errors have been detected in the following files: > > <0x186>:<0x28> > <0x186>:<0x35> > <0xffffffffffffffff>:<0x28> > >Every five seconds the checksum counter got increased. > >zfsdbg-msg output: >2017 Apr 21 11:12:43: bptree index 0: traversing from min_txg=1 bookmark -1/40/0/5120 >2017 Apr 21 11:12:43: bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:43: bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:43: bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:43: bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:43: bptree index 5: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:43: bptree index 6: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:43: bptree index 7: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:48: bptree index 0: traversing from min_txg=1 bookmark -1/40/0/5120 >2017 Apr 21 11:12:48: bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:48: bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:48: bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:48: bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:48: bptree index 5: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:48: bptree index 6: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:48: bptree index 7: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:53: bptree index 0: traversing from min_txg=1 bookmark -1/40/0/5120 >2017 Apr 21 11:12:53: bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:53: bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:53: bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:53: bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:53: bptree index 5: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:53: bptree index 6: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:53: bptree index 7: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:58: bptree index 0: traversing from min_txg=1 bookmark -1/40/0/5120 >2017 Apr 21 11:12:58: bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:58: bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:58: bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:58: bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:58: bptree index 5: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:58: bptree index 6: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:12:58: bptree index 7: traversing from min_txg=-1 bookmark 0/0/0/0 > >fk@t520 ~ $zpool get all cloudia2 >NAME PROPERTY VALUE SOURCE >cloudia2 size 2.98T - >cloudia2 capacity 54% - >cloudia2 altroot - default >cloudia2 health ONLINE - >cloudia2 guid 4205907112567218706 default >cloudia2 version - default >cloudia2 bootfs - default >cloudia2 delegation on default >cloudia2 autoreplace off default >cloudia2 cachefile - default >cloudia2 failmode wait default >cloudia2 listsnapshots off default >cloudia2 autoexpand off default >cloudia2 dedupditto 0 default >cloudia2 dedupratio 1.00x - >cloudia2 free 1.37T - >cloudia2 allocated 1.62T - >cloudia2 readonly off - >cloudia2 comment - default >cloudia2 expandsize - - >cloudia2 freeing 24.2G default >cloudia2 fragmentation 32% - >cloudia2 leaked 0 default >[...] > >After setting vfs.zfs.free_leak_on_eio=1: > >zfsdbg-msg output: >2017 Apr 21 11:13:03: bptree index 0: traversing from min_txg=1 bookmark -1/40/0/5120 >2017 Apr 21 11:13:06: freed 100000 blocks in 3050ms from free_bpobj/bptree txg 17892; err=-1 >2017 Apr 21 11:13:07: bptree index 0: traversing from min_txg=1 bookmark -1/68/0/718 >2017 Apr 21 11:13:08: bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:13:08: bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:13:08: bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:13:08: bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:13:08: bptree index 5: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:13:08: bptree index 6: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:13:08: bptree index 7: traversing from min_txg=-1 bookmark 0/0/0/0 >2017 Apr 21 11:13:08: freed 96110 blocks in 1927ms from free_bpobj/bptree txg 17893; err=0 >2017 Apr 21 11:15:33: command: zpool clear cloudia2 > >The checksum error counters stopped incrementing, >"freeing" went to 0 and "leaked" from 0 to 256M. > >fk@t520 ~ $zpool get all cloudia2 >NAME PROPERTY VALUE SOURCE >cloudia2 size 2.98T - >cloudia2 capacity 53% - >cloudia2 altroot - default >cloudia2 health ONLINE - >cloudia2 guid 4205907112567218706 default >cloudia2 version - default >cloudia2 bootfs - default >cloudia2 delegation on default >cloudia2 autoreplace off default >cloudia2 cachefile - default >cloudia2 failmode wait default >cloudia2 listsnapshots off default >cloudia2 autoexpand off default >cloudia2 dedupditto 0 default >cloudia2 dedupratio 1.00x - >cloudia2 free 1.39T - >cloudia2 allocated 1.59T - >cloudia2 readonly off - >cloudia2 comment - default >cloudia2 expandsize - - >cloudia2 freeing 0 default >cloudia2 fragmentation 32% - >cloudia2 leaked 256M default >[...] > >The difference on the receiving side confirmed that some space had been recovered: > >[fk@kendra ~]$ zfs list -r -p -t all dpool/ggated/cloudia2 >NAME USED AVAIL REFER MOUNTPOINT >[...] >dpool/ggated/cloudia2@2017-04-21_10:37 9251840 - 1812645106176 - >dpool/ggated/cloudia2@2017-04-21_11:17 3950592 - 1800267106304 - > >It's not obvious to me if the 256M were really leaked >but either way it looks like a clear win. > >On another ZFS pool with the same issue but backed by an USB disk all >the space in "freeing" was supposedly "leaked" but it was a lot less >to begin with: > >Before setting vfs.zfs.free_leak_on_eio=1: > >fk@t520 /usr/src $zpool get all wde4 >NAME PROPERTY VALUE SOURCE >wde4 size 1.81T - >wde4 capacity 94% - >wde4 altroot - default >wde4 health ONLINE - >wde4 guid 14402430966328721211 default >wde4 version - default >wde4 bootfs - default >wde4 delegation on default >wde4 autoreplace off default >wde4 cachefile - default >wde4 failmode wait default >wde4 listsnapshots off default >wde4 autoexpand off default >wde4 dedupditto 0 default >wde4 dedupratio 1.00x - >wde4 free 107G - >wde4 allocated 1.71T - >wde4 readonly off - >wde4 comment - default >wde4 expandsize - - >wde4 freeing 1.18M default >wde4 fragmentation 23% - >wde4 leaked 0 default > >After setting vfs.zfs.free_leak_on_eio=1: > >fk@t520 /usr/src $zpool get all wde4 >NAME PROPERTY VALUE SOURCE >wde4 size 1.81T - >wde4 capacity 94% - >wde4 altroot - default >wde4 health ONLINE - >wde4 guid 14402430966328721211 default >wde4 version - default >wde4 bootfs - default >wde4 delegation on default >wde4 autoreplace off default >wde4 cachefile - default >wde4 failmode wait default >wde4 listsnapshots off default >wde4 autoexpand off default >wde4 dedupditto 0 default >wde4 dedupratio 1.00x - >wde4 free 107G - >wde4 allocated 1.71T - >wde4 readonly off - >wde4 comment - default >wde4 expandsize - - >wde4 freeing 0 default >wde4 fragmentation 23% - >wde4 leaked 1.18M default >[...] > >The pool was affected by the issue since 2015: >https://lists.freebsd.org/pipermail/freebsd-fs/2015-February/020845.html > >Obtained from: ElectroBSD >--- > sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_misc.c | 2 ++ > 1 file changed, 2 insertions(+) > >diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_misc.c b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_misc.c >index f54313844049..cdef92f88e15 100644 >--- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_misc.c >+++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_misc.c >@@ -367,6 +367,8 @@ SYSCTL_INT(_vfs_zfs, OID_AUTO, deadman_enabled, CTLFLAG_RDTUN, > &zfs_deadman_enabled, 0, "Kernel panic on stalled ZFS I/O"); > SYSCTL_INT(_vfs_zfs, OID_AUTO, spa_asize_inflation, CTLFLAG_RWTUN, > &spa_asize_inflation, 0, "Worst case inflation factor for single sector writes"); >+SYSCTL_INT(_vfs_zfs, OID_AUTO, free_leak_on_eio, CTLFLAG_RWTUN, >+ &zfs_free_leak_on_eio, 0, "Leak space from indirect blocks that can not be read"); > #endif > > #ifndef illumos >-- >2.12.1 >
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=182174">View the attachment on a separate page</a>.</b>
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
Attachments on
bug 218954
: 182174