Bug 236480 - [zfs] [panic] solaris assert: dn->dn_type != DMU_OT_NONE
Summary: [zfs] [panic] solaris assert: dn->dn_type != DMU_OT_NONE
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Andriy Gapon
URL:
Keywords:
: 236220 (view as bug list)
Depends on:
Blocks:
 
Reported: 2019-03-12 02:42 UTC by ncrogers
Modified: 2019-09-09 15:24 UTC (History)
10 users (show)

See Also:

Attachments
core dump text (326.70 KB, text/plain)
2019-03-14 19:55 UTC, ncrogers 		no flags Details
second core dump text (170.10 KB, text/plain)
2019-03-14 19:56 UTC, ncrogers 		no flags Details
Fix dn->dn_type != DMU_OT_NONE in dbuf_create() (2.83 KB, patch)
2019-03-15 21:20 UTC, ncrogers 		no flags Details | Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description ncrogers 2019-03-12 02:42:20 UTC 
This is related to bug #226130.

The following panic happens with INVARIANTS enabled. It has happened more than once but is difficult to reproduce reliably. The affected dataset is home to a PostgreSQL database.

courtland# kgdb /boot/kernel/kernel /var/crash/vmcore.0
GNU gdb (GDB) 8.2.1 [GDB v8.2.1 for FreeBSD]
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...done.
done.

Unread portion of the kernel message buffer:
panic: solaris assert: dn->dn_type != DMU_OT_NONE, file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c, line: 2540
cpuid = 1
time = 1552104873
KDB: stack backtrace:
#0 0xffffffff80d3bf77 at kdb_backtrace+0x67
#1 0xffffffff80cf4a53 at vpanic+0x1a3
#2 0xffffffff80cf4833 at panic+0x43
#3 0xffffffff803bbcba at assfail+0x1a
#4 0xffffffff803f094b at dbuf_create+0x9b
#5 0xffffffff803f08a1 at dbuf_create_bonus+0x71
#6 0xffffffff803f8d10 at dmu_bonus_hold+0xb0
#7 0xffffffff804cde29 at zfs_zget+0x69
#8 0xffffffff804bff37 at zfs_get_data+0x67
#9 0xffffffff804d3faf at zil_commit_impl+0x15af
#10 0xffffffff804c73bb at zfs_freebsd_fsync+0xcb
#11 0xffffffff813861be at VOP_FSYNC_APV+0x9e
#12 0xffffffff80dab2d4 at vop_stdfdatasync+0x34
#13 0xffffffff8138a84e at VOP_FDATASYNC_APV+0x9e
#14 0xffffffff80dc7bd3 at kern_fsync+0x1b3
#15 0xffffffff811fc532 at amd64_syscall+0x272
#16 0xffffffff811d65ad at fast_syscall_common+0x101
Uptime: 3d10h28m54s
Dumping 4867 out of 16241 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at ./machine/pcpu.h:230
230	./machine/pcpu.h: No such file or directory.
(kgdb) backtrace
#0  __curthread () at ./machine/pcpu.h:230
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80cf468b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:446
#3  0xffffffff80cf4ab3 in vpanic (fmt=<optimized out>, ap=0xfffffe00b77c54a0) at /usr/src/sys/kern/kern_shutdown.c:872
#4  0xffffffff80cf4833 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:799
#5  0xffffffff803bbcba in assfail (a=<unavailable>, f=<unavailable>, l=<unavailable>)
    at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81
#6  0xffffffff803f094b in dbuf_create (dn=0xfffff8015f77a770, level=0 '\000', blkid=18446744073709551615, parent=0xfffff80296803c60, blkptr=0x0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:2540
#7  0xffffffff803f08a1 in dbuf_create_bonus (dn=0xfffff8015f77a770) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:3085
#8  0xffffffff803f8d10 in dmu_bonus_hold (os=<optimized out>, object=<optimized out>, tag=0x0, dbp=0xfffffe00b77c5648)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:345
#9  0xffffffff804cde29 in zfs_zget (zfsvfs=0xfffffe008ea28000, obj_num=33262, zpp=0xfffffe00b77c56c0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1137
#10 0xffffffff804bff37 in zfs_get_data (arg=0xfffffe008ea28000, lr=0xfffffe00a6620300, buf=0x0, lwb=0xfffff8042cf828c0, zio=0xfffff80047eb5000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1297
#11 0xffffffff804d3faf in zil_lwb_commit (zilog=<optimized out>, lwb=<optimized out>, itx=<optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c:1493
#12 zil_process_commit_list (zilog=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c:2069
#13 zil_commit_writer (zilog=<optimized out>, zcw=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c:2201
#14 zil_commit_impl (zilog=<optimized out>, foid=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c:2712
#15 0xffffffff804c73bb in zfs_fsync (syncflag=0, vp=<optimized out>, cr=<optimized out>, ct=<optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2627
#16 zfs_freebsd_fsync (ap=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:5079
#17 0xffffffff813861be in VOP_FSYNC_APV (vop=0xffffffff81c64f70 <zfs_vnodeops>, a=0xfffffe00b77c58c0) at vnode_if.c:1331
#18 0xffffffff80dab2d4 in VOP_FSYNC (vp=<unavailable>, waitfor=1, td=<optimized out>) at ./vnode_if.h:549
#19 vop_stdfdatasync (ap=<optimized out>) at /usr/src/sys/kern/vfs_default.c:732
#20 0xffffffff8138a84e in VOP_FDATASYNC_APV (vop=0xffffffff81d3b338 <default_vnodeops>, a=0xfffffe00b77c5938) at vnode_if.c:4437
#21 0xffffffff80dc7bd3 in VOP_FDATASYNC (vp=<optimized out>, td=<optimized out>) at ./vnode_if.h:1849
#22 kern_fsync (td=0xfffff8000bb96580, fd=<optimized out>, fullsync=false) at /usr/src/sys/kern/vfs_syscalls.c:3374
#23 0xffffffff811fc532 in syscallenter (td=0xfffff8000bb96580) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135
#24 amd64_syscall (td=0xfffff8000bb96580, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1076
#25 <signal handler called>
#26 0x000000080131e8aa in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffffffd078
(kgdb) 
#0  __curthread () at ./machine/pcpu.h:230
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80cf468b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:446
#3  0xffffffff80cf4ab3 in vpanic (fmt=<optimized out>, ap=0xfffffe00b77c54a0) at /usr/src/sys/kern/kern_shutdown.c:872
#4  0xffffffff80cf4833 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:799
#5  0xffffffff803bbcba in assfail (a=<unavailable>, f=<unavailable>, l=<unavailable>)
    at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81
#6  0xffffffff803f094b in dbuf_create (dn=0xfffff8015f77a770, level=0 '\000', blkid=18446744073709551615, parent=0xfffff80296803c60, blkptr=0x0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:2540
#7  0xffffffff803f08a1 in dbuf_create_bonus (dn=0xfffff8015f77a770) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:3085
#8  0xffffffff803f8d10 in dmu_bonus_hold (os=<optimized out>, object=<optimized out>, tag=0x0, dbp=0xfffffe00b77c5648)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:345
#9  0xffffffff804cde29 in zfs_zget (zfsvfs=0xfffffe008ea28000, obj_num=33262, zpp=0xfffffe00b77c56c0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1137
#10 0xffffffff804bff37 in zfs_get_data (arg=0xfffffe008ea28000, lr=0xfffffe00a6620300, buf=0x0, lwb=0xfffff8042cf828c0, zio=0xfffff80047eb5000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1297
#11 0xffffffff804d3faf in zil_lwb_commit (zilog=<optimized out>, lwb=<optimized out>, itx=<optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c:1493
#12 zil_process_commit_list (zilog=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c:2069
#13 zil_commit_writer (zilog=<optimized out>, zcw=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c:2201
#14 zil_commit_impl (zilog=<optimized out>, foid=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c:2712
#15 0xffffffff804c73bb in zfs_fsync (syncflag=0, vp=<optimized out>, cr=<optimized out>, ct=<optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2627
#16 zfs_freebsd_fsync (ap=<optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:5079
#17 0xffffffff813861be in VOP_FSYNC_APV (vop=0xffffffff81c64f70 <zfs_vnodeops>, a=0xfffffe00b77c58c0) at vnode_if.c:1331
#18 0xffffffff80dab2d4 in VOP_FSYNC (vp=<unavailable>, waitfor=1, td=<optimized out>) at ./vnode_if.h:549
#19 vop_stdfdatasync (ap=<optimized out>) at /usr/src/sys/kern/vfs_default.c:732
#20 0xffffffff8138a84e in VOP_FDATASYNC_APV (vop=0xffffffff81d3b338 <default_vnodeops>, a=0xfffffe00b77c5938) at vnode_if.c:4437
#21 0xffffffff80dc7bd3 in VOP_FDATASYNC (vp=<optimized out>, td=<optimized out>) at ./vnode_if.h:1849
#22 kern_fsync (td=0xfffff8000bb96580, fd=<optimized out>, fullsync=false) at /usr/src/sys/kern/vfs_syscalls.c:3374
#23 0xffffffff811fc532 in syscallenter (td=0xfffff8000bb96580) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135
#24 amd64_syscall (td=0xfffff8000bb96580, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1076
#25 <signal handler called>
#26 0x000000080131e8aa in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffffffd078
(kgdb) 


(kgdb) fr 6
#6  0xffffffff803f094b in dbuf_create (dn=0xfffff8015f77a770, 
    level=0 '\000', blkid=18446744073709551615, 
    parent=0xfffff80296803c60, blkptr=0x0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:2540
2540		ASSERT(dn->dn_type != DMU_OT_NONE);


(kgdb) print *dn
$2 = {dn_struct_rwlock = {lock_object = {
      lo_name = 0xffffffff814ada70 "dn->dn_struct_rwlock", 
      lo_flags = 40960000, lo_data = 0, lo_witness = 0x0}, 
    sx_lock = 18446735277813228928}, dn_link = {
    list_next = 0xfffff8026c5973d8, list_prev = 0xfffff8018f9cbb48}, 
  dn_objset = 0xfffff8000b72c400, dn_object = 33262, 
  dn_dbuf = 0xfffff80296803c60, dn_handle = 0xfffff802edef9520, 
  dn_phys = 0xfffffe01c6fd1c00, dn_type = DMU_OT_NONE, dn_bonuslen = 168, 
  dn_bonustype = 44 ',', dn_nblkptr = 1 '\001', dn_checksum = 0 '\000', 
  dn_compress = 0 '\000', dn_nlevels = 1 '\001', 
  dn_indblkshift = 17 '\021', dn_datablkshift = 0 '\000', 
  dn_moved = 0 '\000', dn_datablkszsec = 7, dn_datablksz = 3584, 
  dn_maxblkid = 0, dn_next_type = "\000\000\000", 
  dn_num_slots = 1 '\001', dn_next_nblkptr = "\000\000\000", 
  dn_next_nlevels = "\000\000\000", dn_next_indblkshift = "\000\000\000", 
  dn_next_bonustype = "\000\000\000", dn_rm_spillblk = "\000\000\000", 
  dn_next_bonuslen = {0, 0, 0, 0}, dn_next_blksz = {0, 0, 0, 0}, 
  dn_dbufs_count = 0, dn_dirty_link = {{list_next = 0x0, 
      list_prev = 0x0}, {list_next = 0xfffff803bbf774e0, 
      list_prev = 0xfffff8034a965830}, {list_next = 0x0, 
      list_prev = 0x0}, {list_next = 0x0, list_prev = 0x0}}, dn_mtx = {
    lock_object = {lo_name = 0xffffffff81488d05 "dn->dn_mtx", 
      lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
  dn_dirty_records = {{list_size = 344, list_offset = 0, list_head = {
        list_next = 0xfffff8015f77a890, list_prev = 0xfffff8015f77a890}}, 
    {list_size = 344, list_offset = 0, list_head = {
        list_next = 0xfffff8015f77a8b0, list_prev = 0xfffff8015f77a8b0}}, 
    {list_size = 344, list_offset = 0, list_head = {
        list_next = 0xfffff8015f77a8d0, list_prev = 0xfffff8015f77a8d0}}, 
    {list_size = 344, list_offset = 0, list_head = {
        list_next = 0xfffff8015f77a8f0, 
        list_prev = 0xfffff8015f77a8f0}}}, dn_free_ranges = {0x0, 0x0, 
    0x0, 0x0}, dn_allocated_txg = 0, dn_free_txg = 0, 
  dn_assigned_txg = 0, dn_notxholds = {
    cv_description = 0xffffffff814ba0a4 "dn->dn_notxholds", 
    cv_waiters = 0}, dn_dirtyctx = DN_DIRTY_OPEN, 
  dn_dirtyctx_firstset = 0xfffff8038681d070 "\336\300\255\336\336\300\255\336\360G\305\201\377\377\377\377\336\300\255\336\336\300\255\336\360G\305\201\377\377\377\377\336\300\255\336\336\300\255\336\360G\305\201\377\377\377\377\336\300\255\336\336\300\255\336\360G\305\201\377\377\377\377\336\300\255\336\336\300\255\336\360G\305\201\377\377\377\377\336\300\255\336\336\300\255\336\360G\305\201\377\377\377\377\336\300\255\336\336\300\255\336\360G\305\201\377\377\377\377\336\300\255\336\336\300\255\336\360G\305\201\377\377\377\377\336\300\255\336\336\300\255\336\360G\305\201\377\377\377\377\336\300\255\336\336\300\255\336\360G\305\201\377\377\377\377\336\300\255\336\336\300\255\336\360G\305\201\377\377\377\377\336\300\255\336\336\300\255\336\3--Type <RET> for more, q to quit, c to continue without paging--
60G\305\201\377\377\377\377\336\300\255\336\336\300\255\336"..., 
  dn_tx_holds = {rc_mtx = {lock_object = {
        lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, 
        lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, 
    rc_list = {list_size = 40, list_offset = 0, list_head = {
        list_next = 0xfffff8015f77a990, list_prev = 0xfffff8015f77a990}}, 
    rc_removed = {list_size = 40, list_offset = 0, list_head = {
        list_next = 0xfffff8015f77a9b0, list_prev = 0xfffff8015f77a9b0}}, 
    rc_count = 0, rc_removed_count = 0}, dn_holds = {rc_mtx = {
      lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
        lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
      sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, 
      list_offset = 0, list_head = {list_next = 0xfffff8015f77aa08, 
        list_prev = 0xfffff8015f77aa08}}, rc_removed = {list_size = 40, 
      list_offset = 0, list_head = {list_next = 0xfffff8015f77aa28, 
        list_prev = 0xfffff8015f77aa28}}, rc_count = 2, 
    rc_removed_count = 0}, dn_dbufs_mtx = {lock_object = {
      lo_name = 0xffffffff814404f1 "dn->dn_dbufs_mtx", 
      lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
  dn_dbufs = {avl_root = 0x0, 
    avl_compar = 0xffffffff80414810 <dbuf_compare>, avl_offset = 288, 
    avl_numnodes = 0, avl_size = 352}, dn_bonus = 0x0, dn_have_spill = 0, 
  dn_zio = 0xfffff8024a314000, dn_oldused = 0, dn_oldflags = 0, 
  dn_olduid = 0, dn_oldgid = 0, dn_newuid = 70, dn_newgid = 70, 
  dn_id_flags = 13, dn_zfetch = {zf_rwlock = {lock_object = {
        lo_name = 0xffffffff813ec741 "zf->zf_rwlock", 
        lo_flags = 40960000, lo_data = 0, lo_witness = 0x0}, 
      sx_lock = 1}, zf_stream = {list_size = 80, list_offset = 64, 
      list_head = {list_next = 0xfffff8015f77ab10, 
        list_prev = 0xfffff8015f77ab10}}, zf_dnode = 0xfffff8015f77a770}}
(kgdb) print *parent
$3 = {db = {db_object = 0, db_offset = 17022976, db_size = 16384, 
    db_data = 0xfffffe01c6fd0000}, db_objset = 0xfffff8000b72c400, 
  db_dnode_handle = 0xfffff8000b72c420, db_parent = 0xfffff800743a9000, 
  db_hash_next = 0x0, db_blkid = 1039, db_blkptr = 0xfffffe017b400780, 
  db_level = 0 '\000', db_mtx = {lock_object = {
      lo_name = 0xffffffff81427def "db->db_mtx", lo_flags = 577830912, 
      lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, db_state = DB_CACHED, 
  db_holds = {rc_mtx = {lock_object = {
        lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, 
        lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, 
    rc_list = {list_size = 40, list_offset = 0, list_head = {
        list_next = 0xfffff80296803d18, list_prev = 0xfffff80296803d18}}, 
    rc_removed = {list_size = 40, list_offset = 0, list_head = {
        list_next = 0xfffff80296803d38, list_prev = 0xfffff80296803d38}}, 
    rc_count = 19, rc_removed_count = 0}, db_buf = 0xfffff80225c1de00, 
  db_changed = {cv_description = 0xffffffff8138c10b "db->db_changed", 
    cv_waiters = 0}, db_data_pending = 0xfffff80449928000, 
  db_last_dirty = 0xfffff80449928000, db_link = {avl_child = {
      0xfffff802abcc6ee0, 0xfffff8030efbfc20}, 
    avl_pcb = 18446735290778607909}, db_cache_link = {list_next = 0x0, 
    list_prev = 0x0}, db_caching_status = DB_NO_CACHE, 
  db_user = 0xfffff802edef9000, db_user_immediate_evict = 0 '\000', 
  db_freed_in_flight = 0 '\000', db_pending_evict = 0 '\000', 
  db_dirtycnt = 1 '\001'}


(kgdb) fr 9
#9  0xffffffff804cde29 in zfs_zget (zfsvfs=0xfffffe008ea28000, obj_num=33262, zpp=0xfffffe00b77c56c0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1137
1137		err = sa_buf_hold(zfsvfs->z_os, obj_num, NULL, &db);


(kgdb) print *zfsvfs
$4 = {z_vfs = 0xfffff800133fd000, z_parent = 0xfffffe008ea28000, z_os = 0xfffff8000b72c400, z_root = 4, z_rootvnode = 0xfffff8001340eb40, 
  z_rootvnodelock = {lock_object = {lo_name = 0xffffffff814594a9 "zfs root vnode lock", lo_flags = 50528256, lo_data = 0, lo_witness = 0x0}, 
    rm_writecpus = {__bits = {0, 0, 0, 0}}, rm_activeReaders = {lh_first = 0x0}, _rm_lock = {_rm_wlock_object = {
        lo_name = 0xffffffff814594a9 "zfs root vnode lock", lo_flags = 16842752, lo_data = 0, lo_witness = 0x0}, _rm_lock_mtx = {lock_object = {
          lo_name = 0xffffffff814594a9 "zfs root vnode lock", lo_flags = 16842752, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, _rm_lock_sx = {
        lock_object = {lo_name = 0xffffffff814594a9 "zfs root vnode lock", lo_flags = 16842752, lo_data = 0, lo_witness = 0x0}, sx_lock = 0}}}, 
  z_unlinkedobj = 3, z_max_blksz = 16384, z_fuid_obj = 0, z_fuid_size = 0, z_fuid_idx = {avl_root = 0x0, avl_compar = 0x0, avl_offset = 0, 
    avl_numnodes = 0, avl_size = 0}, z_fuid_domain = {avl_root = 0x0, avl_compar = 0x0, avl_offset = 0, avl_numnodes = 0, avl_size = 0}, 
  z_fuid_lock = {lock_object = {lo_name = 0xffffffff81496098 "zfsvfs->z_fuid_lock", lo_flags = 40960000, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
  z_fuid_loaded = 0, z_fuid_dirty = 0, z_fuid_replay = 0x0, z_log = 0xfffff80003b09800, z_acl_mode = 0, z_acl_inherit = 4, 
  z_case = ZFS_CASE_SENSITIVE, z_utf8 = 0, z_norm = 0, z_atime = 0, z_unmounted = 0, z_teardown_lock = {locks = {{rr_lock = {lock_object = {
            lo_name = 0xffffffff813bcb70 "rrl->rr_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {
          cv_description = 0xffffffff8141c850 "rrl->rr_cv", cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {
              lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, 
          rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea281c0, list_prev = 0xfffffe008ea281c0}}, rc_removed = {
            list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea281e0, list_prev = 0xfffffe008ea281e0}}, rc_count = 1, 
          rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, 
              lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea28238, list_prev = 0xfffffe008ea28238}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea28258, list_prev = 0xfffffe008ea28258}}, rc_count = 0, rc_removed_count = 0}, rr_writer_wanted = 0, 
        rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", lo_flags = 577830912, lo_data = 0, 
            lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", cv_waiters = 0}, rr_writer = 0x0, 
        rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
            sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea282f0, 
              list_prev = 0xfffffe008ea282f0}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28310, 
              list_prev = 0xfffffe008ea28310}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {lock_object = {
              lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, 
          rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28368, list_prev = 0xfffffe008ea28368}}, rc_removed = {
            list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28388, list_prev = 0xfffffe008ea28388}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea28420, list_prev = 0xfffffe008ea28420}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea28440, list_prev = 0xfffffe008ea28440}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28498, list_prev = 0xfffffe008ea28498}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea284b8, list_prev = 0xfffffe008ea284b8}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea28550, list_prev = 0xfffffe008ea28550}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
--Type <RET> for more, q to quit, c to continue without paging--
              list_next = 0xfffffe008ea28570, list_prev = 0xfffffe008ea28570}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea285c8, list_prev = 0xfffffe008ea285c8}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea285e8, list_prev = 0xfffffe008ea285e8}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea28680, list_prev = 0xfffffe008ea28680}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea286a0, list_prev = 0xfffffe008ea286a0}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea286f8, list_prev = 0xfffffe008ea286f8}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28718, list_prev = 0xfffffe008ea28718}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea287b0, list_prev = 0xfffffe008ea287b0}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea287d0, list_prev = 0xfffffe008ea287d0}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28828, list_prev = 0xfffffe008ea28828}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28848, list_prev = 0xfffffe008ea28848}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea288e0, list_prev = 0xfffffe008ea288e0}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea28900, list_prev = 0xfffffe008ea28900}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28958, list_prev = 0xfffffe008ea28958}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28978, list_prev = 0xfffffe008ea28978}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea28a10, list_prev = 0xfffffe008ea28a10}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea28a30, list_prev = 0xfffffe008ea28a30}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28a88, list_prev = 0xfffffe008ea28a88}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28aa8, list_prev = 0xfffffe008ea28aa8}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
--Type <RET> for more, q to quit, c to continue without paging--
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea28b40, list_prev = 0xfffffe008ea28b40}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea28b60, list_prev = 0xfffffe008ea28b60}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28bb8, list_prev = 0xfffffe008ea28bb8}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28bd8, list_prev = 0xfffffe008ea28bd8}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea28c70, list_prev = 0xfffffe008ea28c70}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea28c90, list_prev = 0xfffffe008ea28c90}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28ce8, list_prev = 0xfffffe008ea28ce8}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28d08, list_prev = 0xfffffe008ea28d08}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea28da0, list_prev = 0xfffffe008ea28da0}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea28dc0, list_prev = 0xfffffe008ea28dc0}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28e18, list_prev = 0xfffffe008ea28e18}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28e38, list_prev = 0xfffffe008ea28e38}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea28ed0, list_prev = 0xfffffe008ea28ed0}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea28ef0, list_prev = 0xfffffe008ea28ef0}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28f48, list_prev = 0xfffffe008ea28f48}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea28f68, list_prev = 0xfffffe008ea28f68}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea29000, list_prev = 0xfffffe008ea29000}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea29020, list_prev = 0xfffffe008ea29020}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea29078, list_prev = 0xfffffe008ea29078}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea29098, list_prev = 0xfffffe008ea29098}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
--Type <RET> for more, q to quit, c to continue without paging--
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea29130, list_prev = 0xfffffe008ea29130}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea29150, list_prev = 0xfffffe008ea29150}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea291a8, list_prev = 0xfffffe008ea291a8}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea291c8, list_prev = 0xfffffe008ea291c8}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea29260, list_prev = 0xfffffe008ea29260}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea29280, list_prev = 0xfffffe008ea29280}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea292d8, list_prev = 0xfffffe008ea292d8}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea292f8, list_prev = 0xfffffe008ea292f8}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea29390, list_prev = 0xfffffe008ea29390}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea293b0, list_prev = 0xfffffe008ea293b0}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea29408, list_prev = 0xfffffe008ea29408}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea29428, list_prev = 0xfffffe008ea29428}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}, {rr_lock = {lock_object = {lo_name = 0xffffffff813bcb70 "rrl->rr_lock", 
            lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff8141c850 "rrl->rr_cv", 
          cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", 
              lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, 
            list_head = {list_next = 0xfffffe008ea294c0, list_prev = 0xfffffe008ea294c0}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
              list_next = 0xfffffe008ea294e0, list_prev = 0xfffffe008ea294e0}}, rc_count = 0, rc_removed_count = 0}, rr_linked_rcount = {rc_mtx = {
            lock_object = {lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
          rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea29538, list_prev = 0xfffffe008ea29538}}, 
          rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe008ea29558, list_prev = 0xfffffe008ea29558}}, rc_count = 0, 
          rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 0}}}, z_teardown_inactive_lock = {lock_object = {
      lo_name = 0xffffffff81441592 "zfsvfs->z_teardown_inactive_lock", lo_flags = 40960000, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
  z_all_znodes = {list_size = 272, list_offset = 240, list_head = {list_next = 0xfffff8000ba58420, list_prev = 0xfffff8006add8ca0}}, z_znodes_lock = {
    lock_object = {lo_name = 0xffffffff81489faa "zfsvfs->z_znodes_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, 
  z_ctldir = 0xfffff8000ba05400, z_show_ctldir = 0, z_issnap = 0, z_vscan = 0, z_use_fuids = 1, z_replay = 0, z_use_sa = 1, z_use_namecache = 1, 
  z_version = 5, z_shares_dir = 7, z_lock = {lock_object = {lo_name = 0xffffffff8138ce8d "zfsvfs->z_lock", lo_flags = 577830912, lo_data = 0, 
      lo_witness = 0x0}, sx_lock = 1}, z_userquota_obj = 0, z_groupquota_obj = 0, z_replay_eof = 0, z_attr_table = 0xfffff80013351b00, z_hold_mtx = {{
      lock_object = {lo_name = 0xffffffff81429501 "zfsvfs->z_hold_mtx[i]", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
--Type <RET> for more, q to quit, c to continue without paging--
      sx_lock = 1} <repeats 46 times>, {lock_object = {lo_name = 0xffffffff81429501 "zfsvfs->z_hold_mtx[i]", lo_flags = 577830912, lo_data = 0, 
        lo_witness = 0x0}, sx_lock = 18446735277813228928}, {lock_object = {lo_name = 0xffffffff81429501 "zfsvfs->z_hold_mtx[i]", 
        lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1} <repeats 17 times>}, z_unlinked_drain_task = {ta_link = {stqe_next = 0x0}, 
    ta_pending = 0, ta_priority = 0, ta_func = 0xffffffff804bdcd0 <zfsvfs_task_unlinked_drain>, ta_context = 0xfffffe008ea28000}}


(kgdb) print *zfsvfs->z_vfs
$5 = {mnt_mtx = {lock_object = {lo_name = 0xffffffff81393043 "struct mount mtx", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, 
  mnt_gen = 1, mnt_list = {tqe_next = 0xfffff8001343a000, tqe_prev = 0xfffff800133fe028}, mnt_op = 0xffffffff81c64dc8 <zfs_vfsops>, 
  mnt_vfc = 0xffffffff81c64f18 <zfs_vfsconf>, mnt_vnodecovered = 0xfffff8001340ed20, mnt_syncer = 0xfffff8001340e960, mnt_ref = 896, 
  mnt_nvnodelist = {tqh_first = 0xfffff8001340eb40, tqh_last = 0xfffff80137ee83e0}, mnt_nvnodelistsize = 896, mnt_writeopcount = 0, 
  mnt_kern_flag = 1073742272, mnt_flag = 268439568, mnt_opt = 0xfffff8000ba57e10, mnt_optnew = 0x0, mnt_maxsymlinklen = 0, mnt_stat = {
    f_version = 538182936, f_type = 222, f_flags = 268439568, f_bsize = 512, f_iosize = 16384, f_blocks = 404148032, f_bfree = 390589344, 
    f_bavail = 390589344, f_files = 390593281, f_ffree = 390589344, f_syncwrites = 0, f_asyncwrites = 0, f_syncreads = 0, f_asyncreads = 0, 
    f_spare = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, f_namemax = 255, f_owner = 0, f_fsid = {val = {-901290623, -1775448610}}, 
    f_charspare = '\000' <repeats 79 times>, f_fstypename = "zfs", '\000' <repeats 12 times>, 
    f_mntfromname = "zroot/space/pgsql", '\000' <repeats 1006 times>, f_mntonname = "/space/pgsql", '\000' <repeats 1011 times>}, 
  mnt_cred = 0xfffff800133ef900, mnt_data = 0xfffffe008ea28000, mnt_time = 0, mnt_iosize_max = 65536, mnt_export = 0x0, mnt_label = 0x0, 
  mnt_hashseed = 127686091, mnt_lockref = 0, mnt_secondary_writes = 0, mnt_secondary_accwrites = 0, mnt_susp_owner = 0x0, mnt_gjprovider = 0x0, 
  mnt_listmtx = {lock_object = {lo_name = 0xffffffff8142f7e7 "struct mount vlist mtx", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, 
    mtx_lock = 0}, mnt_activevnodelist = {tqh_first = 0xfffff802882ebb40, tqh_last = 0xfffff8001340ec00}, mnt_activevnodelistsize = 438, 
  mnt_tmpfreevnodelist = {tqh_first = 0xfffff800244071e0, tqh_last = 0xfffff803c87a82a0}, mnt_tmpfreevnodelistsize = 11, mnt_explock = {
    lock_object = {lo_name = 0xffffffff81469cce "explock", lo_flags = 108199936, lo_data = 0, lo_witness = 0x0}, lk_lock = 1, lk_exslpfail = 0, 
    lk_timo = 0, lk_pri = 96}, mnt_upper_link = {tqe_next = 0x0, tqe_prev = 0x0}, mnt_uppers = {tqh_first = 0x0, tqh_last = 0xfffff800133fdaa8}}
(kgdb) 


courtland# zdb -dddd zroot/space/pgsql 33262
Dataset zroot/space/pgsql [ZPL], ID 184, cr_txg 37, 6.37G, 3905 objects, rootbp DVA[0]=<0:95f64f000:1000> DVA[1]=<0:11e4fbc000:1000> [L0 DMU objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P birth=925624L/925624P fill=3905 cksum=d3563dce4:eed0e01cca3:a72f616c79c87:5770ae867335855

    Object  lvl   iblk   dblk  dsize  dnsize lsize   %full  type
zdb: dmu_bonus_hold(33262) failed, errno 2
Comment 1 ncrogers 2019-03-12 02:47:15 UTC 
I meant to reference bug #236220
Comment 2 ncrogers 2019-03-14 19:55:51 UTC 
Created attachment 202858 [details]
core dump text

core.txt.0
Comment 3 ncrogers 2019-03-14 19:56:18 UTC 
Created attachment 202859 [details]
second core dump text

core.txt.1
Comment 4 Julien Cigar 2019-03-14 20:18:26 UTC 
I'm suffering the same problem on a freshly 12.0-RELEASE running PostgreSQL 11.1 in a jail with nullfs mounts:

chouffe% mount -t nullfs
/data/jails/pg11 on /jails/pg11/data (nullfs, local)
/data/jails/pg11/data11 on /jails/pg11/data/data11 (nullfs, local)
/data/jails/pg11/backups on /jails/pg11/data/backups (nullfs, local)

chouffe% jls
   JID  IP Address      Hostname                      Path
     1  192.168.10.39   pg11.prod.lan                 /jails/pg11

I could login to the system but a simple "ls" hangs and I had to login on the iLO console and issue an hard reset. The system is running the GENERIC kernel with the following:

chouffe% more /boot/loader.conf 
aesni_load="YES"
boot_multicons="YES"
boot_serial="YES"
comconsole_port="0x2F8"
console="comconsole,efi"
cpu_microcode_load="YES"
cpu_microcode_name="/boot/firmware/intel-ucode.bin"
geom_mirror_load="YES"
hw.vga.textmode="1"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
kern.racct.enable="1"
vfs.zfs.min_auto_ashift="12"
zfs_load="YES"

chouffe% more /etc/sysctl.conf 
vfs.zfs.min_auto_ashift=12
kern.ipc.somaxconn=2048
vfs.zfs.arc_max=25769803776
vfs.nfsd.server_max_nfsvers=4
vfs.nfsd.server_min_nfsvers=4

I couldn't capture any useful crash dump unfortunately ..
Comment 5 ncrogers 2019-03-14 20:26:21 UTC 
(In reply to Julien Cigar from comment #4)

Glad to hear I'm not the only one... I think this bug is a duplicate of #236220 but I was asked to create a separate issue because of the kernel panic. I think you are suffering from the "deadlock" / hang issue that I am experiencing. The reason, I believe, that I get an actual panic is because I am trying a custom kernel with INVARIANTS enabled. If you don't have INVARIANTS it eventually the filesystem deadlock.

FWIW I am not using nullfs, but I AM using postgresql 11.1.

It might be useful if you could report a `procstat -kka` output when this happens.
Comment 6 Julien Cigar 2019-03-14 21:35:40 UTC 
(In reply to ncrogers from comment #5)

Yes it is exactly the same symptoms as bug #236220
I'll try to procstat -kka next time it happens
Comment 7 Andriy Gapon freebsd_committer freebsd_triage 2019-03-14 22:14:02 UTC 
(In reply to Julien Cigar from comment #6)
Not an insightful comment by any measure, but maybe follow-up to the bug that matches your symptoms the best rather than to a bug that may or may not be its duplicate.
Comment 8 Andriy Gapon freebsd_committer freebsd_triage 2019-03-14 22:29:09 UTC 
(In reply to ncrogers from comment #0)
Could you please also print *dn->dn_phys ?
Comment 9 Andriy Gapon freebsd_committer freebsd_triage 2019-03-14 22:35:33 UTC 
And *dn->dn_dbuf
Comment 10 ncrogers 2019-03-14 22:42:02 UTC 
(kgdb) fr 6
#6  0xffffffff803f094b in dbuf_create (dn=0xfffff8015f77a770, level=0 '\000', 
    blkid=18446744073709551615, parent=0xfffff80296803c60, blkptr=0x0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:2540
2540		ASSERT(dn->dn_type != DMU_OT_NONE);

(kgdb) print *dn->dn_phys
$1 = {dn_type = 0 '\000', dn_indblkshift = 0 '\000', dn_nlevels = 0 '\000', 
  dn_nblkptr = 0 '\000', dn_bonustype = 0 '\000', dn_checksum = 0 '\000', 
  dn_compress = 0 '\000', dn_flags = 0 '\000', dn_datablkszsec = 0, 
  dn_bonuslen = 0, dn_extra_slots = 0 '\000', dn_pad2 = "\000\000", 
  dn_maxblkid = 0, dn_used = 0, dn_pad3 = {0, 0, 0, 0}, {dn_blkptr = {{
        blk_dva = {{dva_word = {0, 0}}, {dva_word = {0, 0}}, {dva_word = {0, 
              0}}}, blk_prop = 0, blk_pad = {0, 0}, blk_phys_birth = 0, 
        blk_birth = 0, blk_fill = 0, blk_cksum = {zc_word = {0, 0, 0, 0}}}, {
        blk_dva = {{dva_word = {0, 0}}, {dva_word = {0, 0}}, {dva_word = {0, 
              0}}}, blk_prop = 0, blk_pad = {0, 0}, blk_phys_birth = 0, 
        blk_birth = 0, blk_fill = 0, blk_cksum = {zc_word = {0, 0, 0, 0}}}, {
        blk_dva = {{dva_word = {0, 0}}, {dva_word = {0, 0}}, {dva_word = {0, 
              0}}}, blk_prop = 0, blk_pad = {0, 0}, blk_phys_birth = 0, 
        blk_birth = 0, blk_fill = 0, blk_cksum = {zc_word = {0, 0, 0, 0}}}}, {
      __dn_ignore1 = {blk_dva = {{dva_word = {0, 0}}, {dva_word = {0, 0}}, {
            dva_word = {0, 0}}}, blk_prop = 0, blk_pad = {0, 0}, 
        blk_phys_birth = 0, blk_birth = 0, blk_fill = 0, blk_cksum = {
          zc_word = {0, 0, 0, 0}}}, dn_bonus = '\000' <repeats 319 times>}, {
      __dn_ignore2 = {blk_dva = {{dva_word = {0, 0}}, {dva_word = {0, 0}}, {
            dva_word = {0, 0}}}, blk_prop = 0, blk_pad = {0, 0}, 
        blk_phys_birth = 0, blk_birth = 0, blk_fill = 0, blk_cksum = {
          zc_word = {0, 0, 0, 0}}}, __dn_ignore3 = '\000' <repeats 191 times>, 
      dn_spill = {blk_dva = {{dva_word = {0, 0}}, {dva_word = {0, 0}}, {
            dva_word = {0, 0}}}, blk_prop = 0, blk_pad = {0, 0}, 
        blk_phys_birth = 0, blk_birth = 0, blk_fill = 0, blk_cksum = {
          zc_word = {0, 0, 0, 0}}}}}}

(kgdb) print *dn->dn_dbuf
$2 = {db = {db_object = 0, db_offset = 17022976, db_size = 16384, 
    db_data = 0xfffffe01c6fd0000}, db_objset = 0xfffff8000b72c400, 
  db_dnode_handle = 0xfffff8000b72c420, db_parent = 0xfffff800743a9000, 
  db_hash_next = 0x0, db_blkid = 1039, db_blkptr = 0xfffffe017b400780, 
  db_level = 0 '\000', db_mtx = {lock_object = {
      lo_name = 0xffffffff81427def "db->db_mtx", lo_flags = 577830912, 
      lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, db_state = DB_CACHED, 
  db_holds = {rc_mtx = {lock_object = {
        lo_name = 0xffffffff814893e6 "rc->rc_mtx", lo_flags = 577830912, 
        lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, 
    rc_list = {list_size = 40, list_offset = 0, list_head = {
        list_next = 0xfffff80296803d18, list_prev = 0xfffff80296803d18}}, 
    rc_removed = {list_size = 40, list_offset = 0, list_head = {
        list_next = 0xfffff80296803d38, list_prev = 0xfffff80296803d38}}, 
    rc_count = 19, rc_removed_count = 0}, db_buf = 0xfffff80225c1de00, 
  db_changed = {cv_description = 0xffffffff8138c10b "db->db_changed", 
    cv_waiters = 0}, db_data_pending = 0xfffff80449928000, 
  db_last_dirty = 0xfffff80449928000, db_link = {avl_child = {
      0xfffff802abcc6ee0, 0xfffff8030efbfc20}, 
    avl_pcb = 18446735290778607909}, db_cache_link = {list_next = 0x0, 
    list_prev = 0x0}, db_caching_status = DB_NO_CACHE, 
  db_user = 0xfffff802edef9000, db_user_immediate_evict = 0 '\000', 
  db_freed_in_flight = 0 '\000', db_pending_evict = 0 '\000', 
  db_dirtycnt = 1 '\001'}
(kgdb)
Comment 11 Andriy Gapon freebsd_committer freebsd_triage 2019-03-14 23:04:59 UTC 
I believe that the problem could be a race condition between dnode_hold and dnode_sync_free possibly introduced in base r337669 which is the large dnode feature from ZoL.
Perhaps, the problem has already been fixed in ZoL...
Comment 12 Andriy Gapon freebsd_committer freebsd_triage 2019-03-14 23:16:42 UTC 
This looks like the fix that I had in mind:
https://github.com/behlendorf/zfs/commit/50f32ed74e42aa28522e9681fb8ae55239fa33a7
but note that it is from a personal branch, not ZoL.

Here is a relevant bug report:
https://github.com/zfsonlinux/zfs/issues/5396
and another one:
https://github.com/zfsonlinux/zfs/issues/6522

And this is what ultimately went into ZoL:
https://github.com/zfsonlinux/zfs/commit/4c5b89f59e4e5c8f5b4680040118ebde09598bbe
Much larger and much harder to understand.
Comment 13 ncrogers 2019-03-14 23:21:22 UTC 
(In reply to Andriy Gapon from comment #12)

I had the same thought when I found that but wasn’t sure where to even begin with trying to come up with a fix for FreeBSD...
Comment 14 ncrogers 2019-03-15 03:16:36 UTC 
(In reply to Andriy Gapon from comment #11)

Do you think this specific race is likely to cause the “deadlock” behavior where the fs “hangs”?
Comment 15 Andriy Gapon freebsd_committer freebsd_triage 2019-03-15 06:53:39 UTC 
(In reply to ncrogers from comment #14)
So far I cannot establish a firm link between those two problems.
On the other hand, instantiating a vnode/znode pair for a deleted file can lead to all kinds of unintended consequences down the road.
Could you please try to apply a patch based on the first ZoL link?
And then keep testing to see if both problems go away or if the first problem comes back? (Or some other less fortunate outcome.)
Comment 16 ncrogers 2019-03-15 21:18:57 UTC 
(In reply to Andriy Gapon from comment #15)
Yes, I have added the simpler fix from the private ZoL branch to my custom kernel and will start deploying that. It applied cleanly to 12.0-RELEASE. I am torn if it's worthwhile to continue testing with INVARIANTS enabled to see if it still panics, or without and see if it still hangs. I am still not able to replicate either case short of just waiting around for a system to do it - no amount of fs IO or CPU stressing seems to do anything, although I have not tried dbench like in the ZoL bug report. Fortunately I have access to quite a few production systems...

I am attaching the patch in case anyone else wants to try it.

I imagine there will need to be a larger effort from someone (mmacy?) to correctly bring in the fixes to the recent "large_dnode pool feature" from ZoL?
Comment 17 ncrogers 2019-03-15 21:20:29 UTC 
Created attachment 202890 [details]
Fix dn->dn_type != DMU_OT_NONE in dbuf_create()
Comment 18 Julien Cigar 2019-03-16 11:14:26 UTC 
(In reply to ncrogers from comment #16)

Thank you for testing, let us know if it solves the deadlock issue.

It's difficult to test here, as it is our main production PostgreSQL master, but if the problem reappears I will promote our slave (12.0-RELEASE, UFS) to master and vice-versa (and thus could also test some patches on the 12.0-RELEASE, ZFS machine)
Comment 19 Julien Cigar 2019-03-31 10:26:32 UTC 
The machine deadlocked again yesterday, I had to login to the HP iLO console (couldn't login through SSH) and issue a POWER RESET.

Any updates about the short fix in comment #17 ? I'd like to test it before switching our MASTER PostgreSQL to another machine

Thanks!
Comment 20 ncrogers 2019-04-01 12:47:38 UTC 
(In reply to Julien Cigar from comment #19)

I have not experienced a panic or hang/deadlock since deploying the patch to dbuf_create, however it has been only two weeks so it’s hard to say if that means anything or not because of how difficult this issue is to replicate. It is encouraging though, maybe you should try it. I would also suggest you post more information about the problem you are experiencing in the other bug report that is about the hanging since you are not experiencing the panic described in this report. Bug #236220

If you compile your kernel with invariants support you may be able to reproduce this panic.
Comment 21 Julien Cigar 2019-04-01 13:41:22 UTC 
Thank you for testing, I'll follow Bug #236220 and put my future comments there as I don't have INVARIANTS support.
Comment 22 Andriy Gapon freebsd_committer freebsd_triage 2019-06-06 11:05:48 UTC 
(In reply to ncrogers from comment #16)

Yes, I thought that mmacy would be more active on this issue.
I understand that he focuses on ZoF, but we still have in-tree ZFS to take care about and we are going to have it for a while in stable branches.
So, ignoring a bug like this is not good.

Now, I could commit the simpler fix, but I do not want to interfere with any possible on-going work to bring in the full fix (or the large dnode support as was committed to illumos).
Comment 23 Dennis Noordsij 2019-06-06 15:18:44 UTC 
I would like to confirm that after applying this patch my database-server uptime is now 36 days without any issue (previously I would make it 2 weeks or so).

It's certainly better than nothing.
Comment 24 Julien Cigar 2019-06-06 15:36:27 UTC 
I would add a "+1", I've applied the patch on our "dev" machine (small HPE Proliant Microserver) 1,5 months ago and it hasn't deadlocked so far.
Comment 25 Julien Cigar 2019-06-25 11:39:25 UTC 
With the upcoming 11.3-RELEASE, just to be sure: none of the the large_dnode pool features have been MFC'ed, right? In other words: this bug should not trigger on 11.3, right?
Comment 26 cedric 2019-06-28 07:51:50 UTC 
I've the same question as Julien: is this bug present in 11.3-RELEASE?
Since the simple patch seems to be working well, will it be finally integrated 1n 12.0 and in an official EN errata?
Thanks
Cédric
Comment 27 ncrogers 2019-06-28 18:19:15 UTC 
I am guessing this bug is not present in 11.3 since the large dnode feature doesn't appear to be in it, at least not that I can see in the SVN changelog? I have no official affiliation with FreeBSD, but based on my experience there is unlikely to be a 12.0 errata for this fix since it doesn't affect everyone. I would assume the problem still exists in CURRENT (13.0) so someone needs to decide what the correct approach is to fix this in CURRENT, and that can hopefully get MFCd to 12-STABLE and it will be fixed in 12.1. At least that is what I am hoping for...
Comment 28 Alexander Ushakov 2019-07-11 09:32:04 UTC 
I have the same problem with PostgreSQL 9.6 on 2 servers. Patch helps to solve problem. But I want to now when it'll be fixed in FreeBSD itself?
Comment 29 Dennis Noordsij 2019-07-16 13:43:16 UTC 
My mailserver just locked up in the same fashion, login prompt moved but then just hung trying to log in. I don't know if it is the exact same issue, but these machines are all set up almost identically and symptoms are identical.

There is however no postgresql running, but uptime was at least 3 months.
Comment 30 commit-hook freebsd_committer freebsd_triage 2019-08-12 10:30:40 UTC 
A commit references this bug:

Author: avg
Date: Mon Aug 12 10:30:01 UTC 2019
New revision: 350894
URL: https://svnweb.freebsd.org/changeset/base/350894

Log:
  a stop gap fix for a race between dnode_hold and dnode_sync_free

  The race was introduced in r337669, the large dnode feature import from
  ZoL.  The problem was debugged by ZoL developers and then,
  independently, on FreeBSD.

  The fix is an early proposal by Brian Behlendorf:
  https://github.com/behlendorf/zfs/commit/50f32ed74e42aa28522e9681fb8ae55239fa33a7
  This fix never went into ZoL.  A larger change that was committed later
  included a different solution because of the re-worked code.

  Ideally, we want to revert this fix and re-synchronize FreeBSD large
  dnode code with that in illumos (or newer ZoL).  illumos has a later
  import of the feature from ZoL that does not have the bug.

  PR:		236480
  Obtained from:	Brian Behlendorf <behlendorf1@llnl.gov>
  Submitted by:	ncrogers@gmail.com (patch adaptation)
  Reported by:	ncrogers@gmail.com
  Tested by:	ncrogers@gmail.com,
  		Dennis Noordsij <dennis.noordsij@alumni.helsinki.fi>,
  		Julien Cigar <julien@perdition.city>
  MFC after:	10 days

Changes:
  head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c
Comment 31 ncrogers 2019-08-21 17:30:59 UTC 
*** Bug 236220 has been marked as a duplicate of this bug. ***
Comment 32 commit-hook freebsd_committer freebsd_triage 2019-08-22 07:18:01 UTC 
A commit references this bug:

Author: avg
Date: Thu Aug 22 07:17:49 UTC 2019
New revision: 351383
URL: https://svnweb.freebsd.org/changeset/base/351383

Log:
  MFC r350894: a stop gap fix for a race between dnode_hold and dnode_sync_free

  The race was introduced in r337669, the large dnode feature import from
  ZoL.  The problem was debugged by ZoL developers and then,
  independently, on FreeBSD.

  The fix is an early proposal by Brian Behlendorf:
  https://github.com/behlendorf/zfs/commit/50f32ed74e42aa28522e9681fb8ae55239fa33a7
  This fix never went into ZoL.  A larger change that was committed later
  included a different solution because of the re-worked code.

  Ideally, we want to revert this fix and re-synchronize FreeBSD large
  dnode code with that in illumos (or newer ZoL).  illumos has a later
  import of the feature from ZoL that does not have the bug.

  PR:		236480

Changes:
_U  stable/12/
  stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c
Comment 33 albert.shih 2019-08-29 04:00:20 UTC 
(In reply to Dennis Noordsij from comment #29)

I concur, I was running my mailbox server with 11.2 with cyrus imap
The server run two zpool, one for the index, one for the mailbox himself.
After upgrading to 12.0 I experience random not responding from the first zpool (The one for the index). The second zpool will continue to work properly. The system himself still work too. 

I just apply the patch. I will wait and report to you. 

I'm not sure it's the same problem, but because cyrus imap index are likely a SGDB I will say it's seem very likely it's the same. 

So it will be fine if the patch can be apply to 12.0-RELEASE-Pxx.
Comment 34 albert.shih 2019-09-09 15:24:59 UTC 
(In reply to albert.shih from comment #33)

So after few time I can say with the patch everything work fine. 

So the bug not just affect postgresql but other software as well. 

A patch inside 12.0-RELEASE would be awesome.

Regards