Bug 192085

Summary: [zfs] panic on zvol resize
Product: Base System Reporter: Kristof Provost <kp>
Component: kernAssignee: freebsd-fs (Nobody) <fs>
Status: Closed FIXED    
Severity: Affects Some People CC: chloe.desoutter, smh
Priority: --- Keywords: patch
Version: CURRENT   
Hardware: Any   
OS: Any   
Description Flags
zvol resize patch none

Description Kristof Provost freebsd_committer 2014-07-24 07:41:38 UTC
On current r268263 with WITNESS and INVARIANTS enabled, do:

# zfs create tank/zvol
# zfs set mountpoint=none tank/zvol
# zfs create -V100G tank/zvol/disk0
# zfs set volsize=200G tank/zvol/disk0

This panics as follows:
panic: solaris assert: !rrw_held(&dp->dp_config_rwlock, RW_READER), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c, line: 1120
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01217d54b0
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe01217d5560
vpanic() at vpanic+0x126/frame 0xfffffe01217d55a0
panic() at panic+0x43/frame 0xfffffe01217d5600
assfail() at assfail+0x1d/frame 0xfffffe01217d5610
dsl_pool_hold() at dsl_pool_hold+0x67/frame 0xfffffe01217d5650
dmu_objset_hold() at dmu_objset_hold+0x21/frame 0xfffffe01217d5690
dsl_prop_get_integer() at dsl_prop_get_integer+0x28/frame 0xfffffe01217d56d0
zvol_set_volsize() at zvol_set_volsize+0x126/frame 0xfffffe01217d5760
zfs_prop_set_special() at zfs_prop_set_special+0x2e2/frame 0xfffffe01217d57f0
zfs_set_prop_nvlist() at zfs_set_prop_nvlist+0x23f/frame 0xfffffe01217d5880
zfs_ioc_set_prop() at zfs_ioc_set_prop+0x106/frame 0xfffffe01217d58e0
zfsdev_ioctl() at zfsdev_ioctl+0x6ee/frame 0xfffffe01217d5990
devfs_ioctl_f() at devfs_ioctl_f+0xfb/frame 0xfffffe01217d59f0
kern_ioctl() at kern_ioctl+0x22b/frame 0xfffffe01217d5a50
sys_ioctl() at sys_ioctl+0x13c/frame 0xfffffe01217d5aa0
amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe01217d5bb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe01217d5bb0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8019e89ba, rsp = 0x7fffffffb8c8, rbp = 0x7fffffffb940 ---
Comment 1 Kristof Provost freebsd_committer 2014-07-24 07:42:13 UTC
I've poked at this a bit more, and I think I understand the problem now.

zvol_set_volsize() takes a hold on the file system with dmu_objset_hold()
and then verifies that it's not marked as read-only.
It does this through dsl_prop_get_integer() which also tries to take a
hold on the file system with dmu_objset_hold(). That triggers the assert
in dsl_pool_hold().
Comment 2 Kristof Provost freebsd_committer 2014-07-24 07:43:36 UTC
The problem appear to be fixed in Illumos commit

Essentially they've changed from using dmu_objset_hold() to

 - https://www.illumos.org/issues/3464
 - https://github.com/illumos/illumos-gate/commit/3b2aab18808792cbd248a12f1edf139b89833c13
 - ZoL: https://github.com/zfsonlinux/zfs/pull/2048

I included the zvol resize bits of the patch in my tree, and can now
resize zvols without panicing the machine.
Comment 3 Kristof Provost freebsd_committer 2014-07-24 07:45:27 UTC
Created attachment 144931 [details]
zvol resize patch

This patch fixes the problem for me, but it might be better to import all of 3b2aab18808792cbd248a12f1edf139b89833c13 rather than just cherry-picking the zvol bits as I've done.

(PS: Despite what Bugzilla thinks I'm not actually a committer.)
Comment 4 ChloƩ Desoutter 2014-09-01 22:25:57 UTC
Affects me aswell on 11-CURRENT. Run on a machine that had just panicked before. Will panic when zfs set volsize=

root@cabron:~ # zpool status
  pool: zroot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
  scan: none requested

	NAME                                          STATE     READ WRITE CKSUM
	zroot                                         ONLINE       0     0     0
	  gptid/14121fe7-dc29-11e3-9178-84349711c070  ONLINE       0     0     0

errors: No known data errors
root@cabron:~ # zfs list
zroot                918G  1.74T   144K  none
zroot/ROOT          13.2G  1.74T   144K  none
zroot/ROOT/default  13.2G  1.74T  6.69G  /
zroot/centoracle    20.6G  1.76T  1.09G  -
zroot/chdesk         531G  2.24T  15.6G  -
zroot/factorie      41.3G  1.78T  2.37G  -
zroot/freebsdguest  13.1G  1.75T  2.76G  -
zroot/herpderp      12.4G  1.74T  12.0G  -
zroot/hostie        51.6G  1.78T  5.23G  -
zroot/jabba         10.3G  1.74T  2.65G  -
zroot/memailguest    206G  1.94T  2.72G  -
zroot/tmp           42.5M  1.74T  42.5M  /tmp
zroot/usr           5.51G  1.74T   144K  /usr
zroot/usr/home      2.86G  1.74T  2.86G  /usr/home
zroot/usr/ports     1.40G  1.74T   940M  /usr/ports
zroot/usr/src       1.25G  1.74T  1.14G  /usr/src
zroot/var            777M  1.74T   420M  /var
zroot/var/crash      159M  1.74T   159M  /var/crash
zroot/var/log       3.22M  1.74T  2.19M  /var/log
zroot/var/mail      3.74M  1.74T  3.50M  /var/mail
zroot/var/tmp       4.31M  1.74T  4.31M  /var/tmp
zroot/vmtest        12.0G  1.75T  1.73G  -

Dump header from device /dev/gpt/swap0
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 1353125888B (1290 MB)
  Blocksize: 512
  Dumptime: Mon Sep  1 22:12:29 2014
  Hostname: cabron.atasta.net
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 11.0-CURRENT #0 r269749: Sat Aug  9 17:05:00 CEST 2014
  Panic String: solaris assert: !rrw_held(&dp->dp_config_rwlock, RW_READER), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c, line: 1115
  Dump Parity: 531732301
  Bounds: 0
  Dump Status: good

cabron.atasta.net dumped core - see /var/crash/vmcore.0

Mon Sep  1 22:19:17 CEST 2014

FreeBSD cabron.atasta.net 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r269749: Sat Aug  9 17:05:00 CEST 2014     root@cabron.atasta.net:/usr/obj/usr/src/sys/GENERIC  amd64

panic: solaris assert: !rrw_held(&dp->dp_config_rwlock, RW_READER), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c, line: 1115

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: solaris assert: !rrw_held(&dp->dp_config_rwlock, RW_READER), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c, line: 1115
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe08568bd3b0
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe08568bd460
vpanic() at vpanic+0x126/frame 0xfffffe08568bd4a0
panic() at panic+0x43/frame 0xfffffe08568bd500
assfail() at assfail+0x1d/frame 0xfffffe08568bd510
dsl_pool_hold() at dsl_pool_hold+0x67/frame 0xfffffe08568bd550
dmu_objset_hold() at dmu_objset_hold+0x21/frame 0xfffffe08568bd590
dsl_prop_get_integer() at dsl_prop_get_integer+0x28/frame 0xfffffe08568bd5d0
zvol_set_volsize() at zvol_set_volsize+0x126/frame 0xfffffe08568bd660
zfs_prop_set_special() at zfs_prop_set_special+0x2e2/frame 0xfffffe08568bd6f0
zfs_set_prop_nvlist() at zfs_set_prop_nvlist+0x23f/frame 0xfffffe08568bd780
zfs_ioc_set_prop() at zfs_ioc_set_prop+0x106/frame 0xfffffe08568bd7e0
zfsdev_ioctl() at zfsdev_ioctl+0x6ee/frame 0xfffffe08568bd890
devfs_ioctl_f() at devfs_ioctl_f+0xfb/frame 0xfffffe08568bd8f0
kern_ioctl() at kern_ioctl+0x22b/frame 0xfffffe08568bd950
sys_ioctl() at sys_ioctl+0x13c/frame 0xfffffe08568bd9a0
amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe08568bdab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe08568bdab0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8019e9e3a, rsp = 0x7fffffffb828, rbp = 0x7fffffffb8a0 ---
KDB: enter: panic
Uptime: 2m25s
Dumping 1290 out of 32574 MB:..2% (CTRL-C to abort) ..12% (CTRL-C to abort)  (CTRL-C to abort) ..22%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/vmm.ko.symbols...done.
Loaded symbols for /boot/kernel/vmm.ko.symbols
Reading symbols from /boot/kernel/nmdm.ko.symbols...done.
Loaded symbols for /boot/kernel/nmdm.ko.symbols
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
Reading symbols from /boot/kernel/pflog.ko.symbols...done.
Loaded symbols for /boot/kernel/pflog.ko.symbols
Reading symbols from /boot/kernel/pf.ko.symbols...done.
Loaded symbols for /boot/kernel/pf.ko.symbols
Reading symbols from /boot/kernel/if_tap.ko.symbols...done.
Loaded symbols for /boot/kernel/if_tap.ko.symbols
#0  doadump (textdump=1) at pcpu.h:219
219     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) #0  doadump (textdump=1) at pcpu.h:219
#1  0xffffffff8090c287 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff8090c7c5 in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:746
#3  0xffffffff8090c813 in panic (fmt=0x0)
    at /usr/src/sys/kern/kern_shutdown.c:675
#4  0xffffffff81f4e21d in assfail (a=<value optimized out>, 
    f=<value optimized out>, l=<value optimized out>)
    at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81
#5  0xffffffff81cc6377 in dsl_pool_hold (name=<value optimized out>, 
    tag=0xffffffff81dba184, dp=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:1115
#6  0xffffffff81ca6471 in dmu_objset_hold (
    name=0xfffffe000b94a000 "zroot/herpderp", tag=0xffffffff81dba184, 
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:476
#7  0xffffffff81cca8e8 in dsl_prop_get_integer (ddname=0x0, 
    propname=0xffffffff81dd1f13 "readonly", valuep=0xfffffe08568bd5e8, 
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_prop.c:264
#8  0xffffffff81d55636 in zvol_set_volsize (
    name=0xfffffe000b94a000 "zroot/herpderp", maj=<value optimized out>, 
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c:916
#9  0xffffffff81d33042 in zfs_prop_set_special (
    dsname=0xfffffe000b94a000 "zroot/herpderp", source=<value optimized out>, 
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:2485
#10 0xffffffff81d3261f in zfs_set_prop_nvlist (
    dsname=0xfffffe000b94a000 "zroot/herpderp", source=ZPROP_SRC_LOCAL, 
    nvl=<value optimized out>, errlist=0xfffff800a97bd9e0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:2600
#11 0xffffffff81d3b066 in zfs_ioc_set_prop (zc=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:2776
#12 0xffffffff81d36c5e in zfsdev_ioctl (dev=<value optimized out>, 
    zcmd=<value optimized out>, arg=<value optimized out>, 
    flag=<value optimized out>, td=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:6136
#13 0xffffffff807f8d7b in devfs_ioctl_f (fp=0xfffff800171dd690, 
    com=3222821398, data=0xfffff80006e730a0, cred=<value optimized out>, 
    td=0xfffff800a9593490) at /usr/src/sys/fs/devfs/devfs_vnops.c:757
#14 0xffffffff80965a9b in kern_ioctl (td=<value optimized out>, 
    fd=<value optimized out>, com=<value optimized out>) at file.h:323
#15 0xffffffff8096581c in sys_ioctl (td=0xfffff800a9593490, 
    uap=0xfffffe08568bda40) at /usr/src/sys/kern/sys_generic.c:702
#16 0xffffffff80d2028a in amd64_syscall (td=0xfffff800a9593490, traced=0)
    at subr_syscall.c:133
#17 0xffffffff80d0220b in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:390
#18 0x00000008019e9e3a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal

Comment 5 Steven Hartland freebsd_committer 2015-03-15 11:33:02 UTC
This should be already fixed by r276069 in current and r277483 in stable/10
Comment 6 Kristof Provost freebsd_committer 2015-03-15 15:19:38 UTC
I've been unable to reproduce this on a current system (from March 8th), so this seems to be fixed.