Bug 227465 - [zfs][iscsi]repeatable panics on iSCSI target/zvol creation
Summary: [zfs][iscsi]repeatable panics on iSCSI target/zvol creation
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.1-STABLE
Hardware: Any Any
: --- Affects Only Me
Assignee: Eugene Grosbein
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2018-04-12 12:06 UTC by emz
Modified: 2022-10-23 07:48 UTC (History)
4 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description emz 2018-04-12 12:06:40 UTC 
I'm getting repeatable panics while massively creating zvols and iSCSI targets pointing to these zvols. At the time of the panicking volmode was default, and vfs.zfs.vol.mode was 1, thus the effective volmode was geom.

I hope that switching to volmode=dev will help me with getting rid of this, but I'm still reporting it just in case someone would want to have a look.

Backtrace from 11.1-RELEASE-p6 r329259:
===Cut===
# kgdb /boot/kernel.old/kernel vmcore.0 
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
stack pointer           = 0x28:0xfffffe10485e9300
frame pointer           = 0x28:0xfffffe10485e9300
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 761 (ctld)
trap number             = 12
panic: page fault
cpuid = 7
KDB: stack backtrace:
#0 0xffffffff80aadac7 at kdb_backtrace+0x67
#1 0xffffffff80a6bba6 at vpanic+0x186
#2 0xffffffff80a6ba13 at panic+0x43
#3 0xffffffff80edf832 at trap_fatal+0x322
#4 0xffffffff80edf889 at trap_pfault+0x49
#5 0xffffffff80edf0c6 at trap+0x286
#6 0xffffffff80ec36d1 at calltrap+0x8
#7 0xffffffff822492f6 at zvol_geom_start+0xf6
#8 0xffffffff809bc477 at g_io_request+0x297
#9 0xffffffff809bc195 at g_io_getattr+0x75
#10 0xffffffff809b80bf at g_dev_ioctl+0x33f
#11 0xffffffff8263afe6 at ctl_be_block_open_dev+0x626
#12 0xffffffff82639ddd at ctl_be_block_ioctl+0x92d
#13 0xffffffff82632bdc at ctl_ioctl+0x48c
#14 0xffffffff8093ae38 at devfs_ioctl_f+0x128
#15 0xffffffff80ac9415 at kern_ioctl+0x255
#16 0xffffffff80ac914f at sys_ioctl+0x16f
#17 0xffffffff80ee0394 at amd64_syscall+0x6c4
Uptime: 6d20h47m1s
Dumping 25781 out of 65386 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel.old/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/zfs.ko
Reading symbols from /boot/kernel.old/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/opensolaris.ko
Reading symbols from /boot/kernel.old/geom_mirror.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/geom_mirror.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/geom_mirror.ko
Reading symbols from /boot/kernel.old/pf.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/pf.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/pf.ko
Reading symbols from /boot/kernel.old/mlx5.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/mlx5.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/mlx5.ko
Reading symbols from /boot/kernel.old/linuxkpi.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/linuxkpi.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/linuxkpi.ko
Reading symbols from /boot/kernel.old/mlx5en.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/mlx5en.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/mlx5en.ko
Reading symbols from /boot/kernel.old/ichwd.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/ichwd.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/ichwd.ko
Reading symbols from /boot/kernel.old/ctl.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/ctl.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/ctl.ko
Reading symbols from /boot/kernel.old/cfiscsi.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/cfiscsi.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/cfiscsi.ko
Reading symbols from /boot/kernel.old/iscsi.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/iscsi.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/iscsi.ko
Reading symbols from /boot/kernel.old/ng_socket.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/ng_socket.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/ng_socket.ko
Reading symbols from /boot/kernel.old/netgraph.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/netgraph.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/netgraph.ko
Reading symbols from /boot/kernel.old/ng_netflow.ko...Reading symbols from /usr/lib/debug//boot/kernel.old/ng_netflow.ko.debug...done.
done.
Loaded symbols for /boot/kernel.old/ng_netflow.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:222
222     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump (textdump=<value optimized out>) at pcpu.h:222
#1  0xffffffff80a6b721 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80a6bbe0 in vpanic (fmt=<value optimized out>, ap=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80a6ba13 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:690
#4  0xffffffff80edf832 in trap_fatal (frame=0xfffffe10485e9240, eva=8) at /usr/src/sys/amd64/amd64/trap.c:801
#5  0xffffffff80edf889 in trap_pfault (frame=0xfffffe10485e9240, usermode=0) at pcpu.h:222
#6  0xffffffff80edf0c6 in trap (frame=0xfffffe10485e9240) at /usr/src/sys/amd64/amd64/trap.c:421
#7  0xffffffff80ec36d1 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236
#8  0xffffffff821ab494 in dmu_objset_spa (os=0x0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:86
#9  0xffffffff822492f6 in zvol_geom_start (bp=0xfffff80b1e45a000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c:2752
#10 0xffffffff809bc477 in g_io_request (bp=0xfffff80b1e45a000, cp=0xfffff801d9a4c080)
    at /usr/src/sys/geom/geom_io.c:626
#11 0xffffffff809bc195 in g_io_getattr (attr=0xfffffe10485e94e8 "GEOM::candelete", cp=<value optimized out>, 
    len=0xfffffe10485e9528, ptr=0xfffffe10485e9530) at /usr/src/sys/geom/geom_io.c:303
#12 0xffffffff809b80bf in g_dev_ioctl (dev=<value optimized out>, cmd=<value optimized out>, 
    data=0xfffffe10485e94e8 "GEOM::candelete", fflag=<value optimized out>, td=0xfffff80144281000) at atomic.h:188
#13 0xffffffff8263afe6 in ctl_be_block_open_dev (be_lun=0xfffff8097076b400, req=<value optimized out>)
    at /usr/src/sys/cam/ctl/ctl_backend_block.c:2072
#14 0xffffffff82639ddd in ctl_be_block_ioctl (dev=<value optimized out>, cmd=<value optimized out>, 
    addr=0xfffff8084df9e000 "block", flag=<value optimized out>, td=<value optimized out>)
    at /usr/src/sys/cam/ctl/ctl_backend_block.c:2610
#15 0xffffffff82632bdc in ctl_ioctl (dev=0xfffff80106088200, cmd=<value optimized out>, 
    addr=0xfffff8084df9e000 "block", flag=3, td=0xfffff80144281000) at /usr/src/sys/cam/ctl/ctl.c:3012
#16 0xffffffff8093ae38 in devfs_ioctl_f (fp=0xfffff8004a9f66e0, com=3243303201, data=0xfffff8084df9e000, 
    cred=0xfffff80019719000, td=0xfffff80144281000) at /usr/src/sys/fs/devfs/devfs_vnops.c:791
#17 0xffffffff80ac9415 in kern_ioctl (td=<value optimized out>, fd=4, com=<value optimized out>, 
    data=<value optimized out>) at file.h:323
#18 0xffffffff80ac914f in sys_ioctl (td=<value optimized out>, uap=0xfffffe10485e9a30)
    at /usr/src/sys/kern/sys_generic.c:745
#19 0xffffffff80ee0394 in amd64_syscall (td=0xfffff80144281000, traced=0) at subr_syscall.c:135
#20 0xffffffff80ec39bb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
#21 0x000000080163f56a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb)
===Cut===

Panic from 11.1-STABLE r332062:
===Cut===
# kgdb /boot/kernel/kernel vmcore.0     
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
trap number             = 12
panic: page fault
cpuid = 21
KDB: stack backtrace:
#0 0xffffffff80b3a367 at kdb_backtrace+0x67
#1 0xffffffff80af3f77 at vpanic+0x177
#2 0xffffffff80af3df3 at panic+0x43
#3 0xffffffff80f71d42 at trap_fatal+0x352
#4 0xffffffff80f71d99 at trap_pfault+0x49
#5 0xffffffff80f71617 at trap+0x2c7
#6 0xffffffff80f51cdc at calltrap+0x8
#7 0xffffffff822fd526 at zvol_geom_start+0xf6
#8 0xffffffff80a463b5 at g_io_schedule_down+0x1a5
#9 0xffffffff80a46cad at g_down_procbody+0x6d
#10 0xffffffff80ab7973 at fork_exit+0x83
#11 0xffffffff80f529fe at fork_trampoline+0xe
Uptime: 6d4h36m7s
Dumping 13242 out of 65385 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /usr/lib/debug//boot/kernel/pf.ko.debug...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /boot/kernel/mlx5.ko...Reading symbols from /usr/lib/debug//boot/kernel/mlx5.ko.debug...done.
done.
Loaded symbols for /boot/kernel/mlx5.ko
Reading symbols from /boot/kernel/linuxkpi.ko...Reading symbols from /usr/lib/debug//boot/kernel/linuxkpi.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linuxkpi.ko
Reading symbols from /boot/kernel/mlx5en.ko...Reading symbols from /usr/lib/debug//boot/kernel/mlx5en.ko.debug...done.
done.
Loaded symbols for /boot/kernel/mlx5en.ko
Reading symbols from /boot/kernel/ichwd.ko...Reading symbols from /usr/lib/debug//boot/kernel/ichwd.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ichwd.ko
Reading symbols from /boot/kernel/ctl.ko...Reading symbols from /usr/lib/debug//boot/kernel/ctl.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ctl.ko
Reading symbols from /boot/kernel/cfiscsi.ko...Reading symbols from /usr/lib/debug//boot/kernel/cfiscsi.ko.debug...done.
done.
Loaded symbols for /boot/kernel/cfiscsi.ko
Reading symbols from /boot/kernel/iscsi.ko...Reading symbols from /usr/lib/debug//boot/kernel/iscsi.ko.debug...done.
done.
Loaded symbols for /boot/kernel/iscsi.ko
Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_socket.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_socket.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /usr/lib/debug//boot/kernel/netgraph.ko.debug...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/ng_netflow.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_netflow.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_netflow.ko
Reading symbols from /boot/kernel/dtraceall.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtraceall.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtraceall.ko
Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtrace.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtrace.ko
Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtmalloc.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtmalloc.ko
Reading symbols from /boot/kernel/dtnfscl.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtnfscl.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtnfscl.ko
Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /usr/lib/debug//boot/kernel/fbt.ko.debug...done.
done.
Loaded symbols for /boot/kernel/fbt.ko
Reading symbols from /boot/kernel/fasttrap.ko...Reading symbols from /usr/lib/debug//boot/kernel/fasttrap.ko.debug...done.
done.
Loaded symbols for /boot/kernel/fasttrap.ko
Reading symbols from /boot/kernel/sdt.ko...Reading symbols from /usr/lib/debug//boot/kernel/sdt.ko.debug...done.
done.
Loaded symbols for /boot/kernel/sdt.ko
Reading symbols from /boot/kernel/systrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/systrace.ko.debug...done.
done.
Loaded symbols for /boot/kernel/systrace.ko
Reading symbols from /boot/kernel/systrace_freebsd32.ko...Reading symbols from /usr/lib/debug//boot/kernel/systrace_freebsd32.ko.debug...done.
done.
Loaded symbols for /boot/kernel/systrace_freebsd32.ko
Reading symbols from /boot/kernel/profile.ko...Reading symbols from /usr/lib/debug//boot/kernel/profile.ko.debug...done.
done.
Loaded symbols for /boot/kernel/profile.ko
Reading symbols from /boot/kernel/ksyms.ko...Reading symbols from /usr/lib/debug//boot/kernel/ksyms.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ksyms.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:229
229     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump (textdump=<value optimized out>) at pcpu.h:229
#1  0xffffffff80af3b8b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:383
#2  0xffffffff80af3fb1 in vpanic (fmt=<value optimized out>, ap=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:776
#3  0xffffffff80af3df3 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:707
#4  0xffffffff80f71d42 in trap_fatal (frame=0xfffffe10470bd910, eva=8) at /usr/src/sys/amd64/amd64/trap.c:817
#5  0xffffffff80f71d99 in trap_pfault (frame=0xfffffe10470bd910, usermode=0) at pcpu.h:229
#6  0xffffffff80f71617 in trap (frame=0xfffffe10470bd910) at /usr/src/sys/amd64/amd64/trap.c:410
#7  0xffffffff80f51cdc in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232
#8  0xffffffff82256ed4 in dmu_objset_spa (os=0x0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:94
#9  0xffffffff822fd526 in zvol_geom_start (bp=0xfffff80ac4bcd468)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c:2769
#10 0xffffffff80a463b5 in g_io_schedule_down (tp=<value optimized out>) at /usr/src/sys/geom/geom_io.c:881
#11 0xffffffff80a46cad in g_down_procbody (arg=<value optimized out>) at /usr/src/sys/geom/geom_kern.c:109
#12 0xffffffff80ab7973 in fork_exit (callout=0xffffffff80a46c40 <g_down_procbody>, arg=0x0, frame=0xfffffe10470bdac0)
    at /usr/src/sys/kern/kern_fork.c:1053
#13 0xffffffff80f529fe in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:843
#14 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb)
===Cut===
Comment 1 Andriy Gapon freebsd_committer freebsd_triage 2018-04-12 12:31:56 UTC 
Please try stable/11 after base r332095.
Comment 2 emz 2018-04-18 13:46:57 UTC 
Got panic on r332466:

Apr 18 17:52:39 san1 kernel: vputx: usecount not zero for vnode
Apr 18 17:52:39 san1 kernel: 0xfffff80f4d7d1760: tag devfs, type VCHR
Apr 18 17:52:39 san1 kernel: usecount -1, writecount -1, refcount 0

Only these lines in the messages, no crashdump this time (although I was getting decent crashdumps previously).

Cannot find much aboth this kind of panic, what does it mean ?
Comment 3 emz 2018-04-19 06:19:55 UTC 
Am I right assuming that the last panic is related to the https://svnweb.freebsd.org/base?view=revision&revision=331666 ?

(Is this PR a right place to continue the discussion ?)
Comment 4 Andriy Gapon freebsd_committer freebsd_triage 2018-04-19 06:49:00 UTC 
(In reply to emz from comment #3)
It's not impossible but it is not very probable.
Comment 5 emz 2018-04-19 08:26:19 UTC 
Created https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227631 since it seems to be another problem.
Comment 6 Andriy Gapon freebsd_committer freebsd_triage 2018-12-06 13:43:31 UTC 
Do you still see this issue (comment #0) ?
Comment 7 emz 2018-12-06 15:25:58 UTC 
After switching to volmode=dev on all the pools that serve iSCSI backends - no. Seems like this issue is more related to this option, rather than to FreeBSD revision.
Comment 8 Eugene Grosbein freebsd_committer freebsd_triage 2022-10-23 06:13:51 UTC 
volmode=dev is the key for stability for noted setup
Comment 9 Graham Perrin freebsd_committer freebsd_triage 2022-10-23 07:48:21 UTC 
^Triage: with closure, there should be assignment to a person.