Bug 240831 - zfs: Panic during snapshot on 12.1-STABLE r352648 (solaris assert: txg_list_empty)
Summary: zfs: Panic during snapshot on 12.1-STABLE r352648 (solaris assert: txg_list_e...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.1-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-fs mailing list
URL:
Keywords: crash, needs-qa
Depends on:
Blocks:
 
Reported: 2019-09-26 06:33 UTC by hlh
Modified: 2019-10-08 14:53 UTC (History)
1 user (show)

See Also:
koobs: mfc-stable11?
koobs: mfc-stable12?


Attachments
zfs list (1.38 KB, text/plain)
2019-09-26 12:55 UTC, hlh
no flags Details
dmesg.boot (9.84 KB, text/plain)
2019-09-26 12:56 UTC, hlh
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description hlh 2019-09-26 06:33:54 UTC
FreeBSD morzine.restart.bel 12.1-STABLE FreeBSD 12.1-STABLE r352648 MORZINE  amd64

Panic during a zfs snapshot -r:

panic: solaris assert: txg_list_empty (&dp->dp_dirty_dirs, txg)

/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c line 8135
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2019-09-26 08:36:36 UTC
Could you provide more information, including:

- zfs list (as an attachment)
- /var/run/dmesg.boot output (as an attachment)
- Backtrace of the panic
Comment 2 hlh 2019-09-26 12:55:34 UTC
Created attachment 207850 [details]
zfs list
Comment 3 hlh 2019-09-26 12:56:20 UTC
Created attachment 207851 [details]
dmesg.boot
Comment 4 hlh 2019-09-26 13:00:49 UTC
zfs snapshot -r on rpool work as expected
zfs snapshot -r in dspool panic.

The backtrace (transcrit by hand):

spa_sync() at spa_sync+0x1a0d/frame ...
txg_sync_thread() at txg_sync_thread+0x307/frame...
fork_exit() ...
Comment 5 Andriy Gapon freebsd_committer 2019-09-26 14:01:08 UTC
You need to configure a dump device and get a crash dump (vmcore file).
Comment 6 hlh 2019-09-26 14:26:51 UTC
The crash dump can be found at:

http://tignes.restart.be/Xfer/
Comment 7 Andriy Gapon freebsd_committer 2019-09-27 11:32:57 UTC
(In reply to hlh from comment #6)
Since you are running a custom build, it's better to analyze the core in your environment. Alternatively, you will need to share your kernel and relevant modules.
Comment 8 hlh 2019-09-27 11:46:03 UTC
I put the kernel and the modules used in /Xfer/kernel/

If you want I can do some analysis on the core file if you give me the command to run.

By the way I also put core.txt.0 in /Xfer/
Comment 9 Andriy Gapon freebsd_committer 2019-09-27 20:25:15 UTC
(In reply to hlh from comment #8)
It would be more convenient if kernel/ was packaged as an archive (e.g., tar).
Also, the vmcore link gives me 403.

Some kgdb commands that you could run meanwhile:
frame 13
info local
frame 12
info local
Comment 10 hlh 2019-09-28 08:01:47 UTC
(kgdb) frame 13
#13 0xffffffff81b0098d in spa_sync (spa=<optimized out>, txg=2864864)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:8135
8135					ASSERT(txg_list_empty(&dp->dp_dirty_dirs, txg));
(kgdb) info local
pass = 1
dp = 0xfffff80004b72800
mos = 0xfffff80008af3400
rvd = 0xfffff80008c44000
vd = <optimized out>
tx = 0xfffff8024ff55700
slots_per_allocator = <optimized out>
max_queue_depth = <optimized out>
mc = <optimized out>
free_bpl = 0xfffffe004ebe8540
error = <optimized out>
(kgdb) frame 12
#12 0xffffffff81eac21a in assfail (a=0x80 <error: Cannot access memory at address 0x80>, f=0xfffffe004f894460 "", l=128)
    at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81
81		panic("solaris assert: %s, file: %s, line: %d", a, f, l);
(kgdb) info local
No locals.
Comment 11 Andriy Gapon freebsd_committer 2019-09-28 20:09:11 UTC
(In reply to hlh from comment #10)
In frame 13:
p *dp
p *dp->dp_spa
The output will be rather large.
Comment 12 hlh 2019-09-29 07:15:53 UTC
kgdb) p *dp
$1 = {dp_spa = 0xfffffe004ebe8000, dp_meta_objset = 0xfffff80008af3400, dp_root_dir = 0xfffff80004e75c00, 
  dp_mos_dir = 0xfffff80004e75800, dp_free_dir = 0xfffff80004a68c00, dp_leak_dir = 0x0, dp_origin_snap = 0xfffff80004a6d000, 
  dp_root_dir_obj = 32, dp_vnrele_taskq = 0xfffff80004bc7c20, dp_meta_rootbp = {blk_dva = {{dva_word = {8, 447810216}}, {
        dva_word = {8, 738256336}}, {dva_word = {8, 1208077200}}}, blk_prop = 9226475966770118659, blk_pad = {0, 0}, 
    blk_phys_birth = 0, blk_birth = 2864862, blk_fill = 2024, blk_cksum = {zc_word = {16400598605, 7775230236364, 
        1847786449934068, 293504487061283720}}}, dp_tmp_userrefs_obj = 0, dp_free_bpobj = {bpo_lock = {lock_object = {
        lo_name = 0xffffffff81bd0df4 <.L.str.19+1> "bpo->bpo_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
      sx_lock = 1}, bpo_os = 0xfffff80008af3400, bpo_object = 41, bpo_epb = 1024, bpo_havecomp = 1 '\001', 
    bpo_havesubobj = 1 '\001', bpo_phys = 0xfffff80008977a00, bpo_dbuf = 0xfffff8000845d420, bpo_cached_dbuf = 0x0}, 
  dp_bptree_obj = 0, dp_empty_bpobj = 20, dp_obsolete_bpobj = {bpo_lock = {lock_object = {lo_name = 0x0, lo_flags = 0, 
        lo_data = 0, lo_witness = 0x0}, sx_lock = 0}, bpo_os = 0x0, bpo_object = 0, bpo_epb = 0, bpo_havecomp = 0 '\000', 
    bpo_havesubobj = 0 '\000', bpo_phys = 0x0, bpo_dbuf = 0x0, bpo_cached_dbuf = 0x0}, dp_scan = 0xfffff80004e73c00, dp_lock = {
    lock_object = {lo_name = 0xffffffff81be58bf <.L.str.71+1> "dp->dp_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
    sx_lock = 1}, dp_spaceavail_cv = {cv_description = 0xffffffff81bb59a8 <.L.str.72+1> "dp->dp_spaceavail_cv", cv_waiters = 0}, 
  dp_dirty_pertxg = {0, 0, 0, 0}, dp_dirty_total = 0, dp_long_free_dirty_pertxg = {0, 0, 0, 0}, dp_mos_used_delta = 0, 
  dp_mos_compressed_delta = 0, dp_mos_uncompressed_delta = 0, dp_last_wakeup = 0, dp_tx = {tx_cpu = 0xfffff800033cf800, 
    tx_sync_lock = {lock_object = {lo_name = 0xffffffff81bd38ff <.L.str.7+1> "tx->tx_sync_lock", lo_flags = 577830912, 
        lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tx_open_txg = 2864865, tx_quiescing_txg = 0, tx_quiesced_txg = 0, 
    tx_syncing_txg = 2864864, tx_synced_txg = 2864863, tx_open_time = 5149924275093, tx_sync_txg_waiting = 2864864, 
    tx_quiesce_txg_waiting = 2864865, tx_sync_more_cv = {cv_description = 0xffffffff81bdade6 <.L.str.8+1> "tx->tx_sync_more_cv", 
      cv_waiters = 0}, tx_sync_done_cv = {cv_description = 0xffffffff81beb698 <.L.str.9+1> "tx->tx_sync_done_cv", cv_waiters = 1}, 
    tx_quiesce_more_cv = {cv_description = 0xffffffff81bb6249 <.L.str.10+1> "tx->tx_quiesce_more_cv", cv_waiters = 1}, 
    tx_quiesce_done_cv = {cv_description = 0xffffffff81bd53c7 <.L.str.11+1> "tx->tx_quiesce_done_cv", cv_waiters = 0}, 
    tx_timeout_cv = {cv_description = 0x0, cv_waiters = 0}, tx_exit_cv = {
      cv_description = 0xffffffff81bc31f1 <.L.str.12+1> "tx->tx_exit_cv", cv_waiters = 0}, tx_threads = 2 '\002', 
    tx_exiting = 0 '\000', tx_sync_thread = 0xfffff80008e9d000, tx_quiesce_thread = 0xfffff80008e9c000, tx_commit_cb_taskq = 0x0}, 
  dp_dirty_datasets = {tl_lock = {lock_object = {lo_name = 0xffffffff81be221e <.L.str.28+1> "tl->tl_lock", lo_flags = 577830912, 
        lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tl_offset = 928, tl_spa = 0xfffffe004ebe8000, tl_head = {0x0, 0x0, 0x0, 
      0x0}}, dp_dirty_zilogs = {tl_lock = {lock_object = {lo_name = 0xffffffff81be221e <.L.str.28+1> "tl->tl_lock", 
        lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tl_offset = 840, tl_spa = 0xfffffe004ebe8000, 
    tl_head = {0x0, 0x0, 0x0, 0x0}}, dp_dirty_dirs = {tl_lock = {lock_object = {
        lo_name = 0xffffffff81be221e <.L.str.28+1> "tl->tl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
      sx_lock = 1}, tl_offset = 96, tl_spa = 0xfffffe004ebe8000, tl_head = {0xfffff80008fb2c60, 0x0, 0x0, 0x0}}, dp_sync_tasks = {
    tl_lock = {lock_object = {lo_name = 0xffffffff81be221e <.L.str.28+1> "tl->tl_lock", lo_flags = 577830912, lo_data = 0, 
        lo_witness = 0x0}, sx_lock = 1}, tl_offset = 0, tl_spa = 0xfffffe004ebe8000, tl_head = {0x0, 0x0, 0x0, 0x0}}, 
  dp_early_sync_tasks = {tl_lock = {lock_object = {lo_name = 0xffffffff81be221e <.L.str.28+1> "tl->tl_lock", lo_flags = 577830912, 
        lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tl_offset = 0, tl_spa = 0xfffffe004ebe8000, tl_head = {0x0, 0x0, 0x0, 0x0}}, 
  dp_sync_taskq = 0xfffff80004bc7c40, dp_zil_clean_taskq = 0xfffff80004bc7c30, dp_config_rwlock = {rr_lock = {lock_object = {
        lo_name = 0xffffffff81bb98ae <.L.str.1+1> "rrl->rr_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
      sx_lock = 1}, rr_cv = {cv_description = 0xffffffff81bc8820 <.L.str.2+1> "rrl->rr_cv", cv_waiters = 0}, rr_writer = 0x0, 
    rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, 
          lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {
          list_next = 0xfffff80004b72d18, list_prev = 0xfffff80004b72d18}}, rc_removed = {list_size = 40, list_offset = 0, 
        list_head = {list_next = 0xfffff80004b72d38, list_prev = 0xfffff80004b72d38}}, rc_count = 0, rc_removed_count = 0}, 
--Type <RET> for more, q to quit, c to continue without paging--
    rr_linked_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, 
          lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {
          list_next = 0xfffff80004b72d90, list_prev = 0xfffff80004b72d90}}, rc_removed = {list_size = 40, list_offset = 0, 
        list_head = {list_next = 0xfffff80004b72db0, list_prev = 0xfffff80004b72db0}}, rc_count = 0, rc_removed_count = 0}, 
    rr_writer_wanted = 0, rr_track_all = 1}, dp_blkstats = 0x0}
(kgdb) p *dp->dp_spa
$2 = {spa_name = "dpool", '\000' <repeats 250 times>, spa_comment = 0x0, spa_avl = {avl_child = {0x0, 0x0}, 
    avl_pcb = 18446741875996868873}, spa_config = 0xfffff80008c45800, spa_config_syncing = 0x0, spa_config_splitting = 0x0, 
  spa_load_info = 0xfffff80004c94220, spa_config_txg = 2029544, spa_sync_pass = 1, spa_state = POOL_STATE_ACTIVE, 
  spa_inject_ref = 0, spa_sync_on = 1 '\001', spa_load_state = SPA_LOAD_NONE, spa_indirect_vdevs_loaded = 1, spa_trust_config = 1, 
  spa_config_source = SPA_CONFIG_SRC_MOS, spa_import_flags = 0, spa_zio_taskq = {{{stqs_count = 1, 
        stqs_taskq = 0xfffff80004deff00}, {stqs_count = 0, stqs_taskq = 0x0}, {stqs_count = 1, stqs_taskq = 0xfffff8000496b3c0}, {
        stqs_count = 0, stqs_taskq = 0x0}}, {{stqs_count = 1, stqs_taskq = 0xfffff8000496b3e0}, {stqs_count = 0, 
        stqs_taskq = 0x0}, {stqs_count = 8, stqs_taskq = 0xfffff800087e38c0}, {stqs_count = 0, stqs_taskq = 0x0}}, {{
        stqs_count = 1, stqs_taskq = 0xfffff80004bc7e10}, {stqs_count = 1, stqs_taskq = 0xfffff80004bc7df0}, {stqs_count = 1, 
        stqs_taskq = 0xfffff80004bc7dd0}, {stqs_count = 1, stqs_taskq = 0xfffff80004bc7db0}}, {{stqs_count = 8, 
        stqs_taskq = 0xfffff800087e3180}, {stqs_count = 0, stqs_taskq = 0x0}, {stqs_count = 1, stqs_taskq = 0xfffff80004bc7d10}, {
        stqs_count = 0, stqs_taskq = 0x0}}, {{stqs_count = 1, stqs_taskq = 0xfffff80004bc7cf0}, {stqs_count = 0, 
        stqs_taskq = 0x0}, {stqs_count = 1, stqs_taskq = 0xfffff80004bc7cd0}, {stqs_count = 0, stqs_taskq = 0x0}}, {{
        stqs_count = 1, stqs_taskq = 0xfffff80004bc7cb0}, {stqs_count = 0, stqs_taskq = 0x0}, {stqs_count = 1, 
        stqs_taskq = 0xfffff80004bc7c90}, {stqs_count = 0, stqs_taskq = 0x0}}}, spa_dsl_pool = 0xfffff80004b72800, 
  spa_is_initializing = 0, spa_normal_class = 0xfffff80004e74000, spa_log_class = 0xfffff80004a68400, spa_first_txg = 2863817, 
  spa_final_txg = 18446744073709551615, spa_freeze_txg = 18446744073709551615, spa_load_max_txg = 18446744073709551615, 
  spa_claim_max_txg = 2863817, spa_loaded_ts = {tv_sec = 1569501932, tv_nsec = 655368601}, spa_meta_objset = 0xfffff80008af3400, 
  spa_evicting_os_lock = {lock_object = {lo_name = 0xffffffff81bce36c <.L.str.16+1> "spa->spa_evicting_os_lock", 
      lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_evicting_os_list = {list_size = 896, 
    list_offset = 304, list_head = {list_next = 0xfffffe004ebe8380, list_prev = 0xfffffe004ebe8380}}, spa_evicting_os_cv = {
    cv_description = 0xffffffff81bd1a69 <.L.str.26+1> "spa->spa_evicting_os_cv", cv_waiters = 0}, spa_vdev_txg_list = {tl_lock = {
      lock_object = {lo_name = 0xffffffff81be221e <.L.str.28+1> "tl->tl_lock", lo_flags = 577830912, lo_data = 0, 
        lo_witness = 0x0}, sx_lock = 1}, tl_offset = 688, tl_spa = 0xfffffe004ebe8000, tl_head = {0x0, 0x0, 0x0, 
      0xfffff80008c432b0}}, spa_root_vdev = 0xfffff80008c44000, spa_min_ashift = 12, spa_max_ashift = 12, 
  spa_config_guid = 11714503416111511200, spa_load_guid = 6215109065070704231, spa_last_synced_guid = 11714503416111511200, 
  spa_config_dirty_list = {list_size = 2280, list_offset = 736, list_head = {list_next = 0xfffffe004ebe8428, 
      list_prev = 0xfffffe004ebe8428}}, spa_state_dirty_list = {list_size = 2280, list_offset = 752, list_head = {
      list_next = 0xfffffe004ebe8448, list_prev = 0xfffffe004ebe8448}}, spa_alloc_locks = 0xfffff80008180380, 
  spa_alloc_trees = 0xfffff80004bf7300, spa_alloc_count = 4, spa_spares = {sav_object = 0, sav_config = 0x0, sav_vdevs = 0x0, 
    sav_count = 0, sav_sync = 0, sav_pending = 0x0, sav_npending = 0}, spa_l2cache = {sav_object = 275, 
    sav_config = 0xfffff80004dee0a0, sav_vdevs = 0xfffff8000496bb70, sav_count = 1, sav_sync = 0, sav_pending = 0x0, 
    sav_npending = 0}, spa_label_features = 0xfffff80004c94200, spa_config_object = 57, spa_config_generation = 0, 
  spa_syncing_txg = 2864864, spa_deferred_bpobj = {bpo_lock = {lock_object = {
        lo_name = 0xffffffff81bd0df4 <.L.str.19+1> "bpo->bpo_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
      sx_lock = 1}, bpo_os = 0xfffff80008af3400, bpo_object = 61, bpo_epb = 128, bpo_havecomp = 1 '\001', 
    bpo_havesubobj = 1 '\001', bpo_phys = 0xfffff80008977800, bpo_dbuf = 0xfffff800089709a0, 
    bpo_cached_dbuf = 0xfffff8000d450000}, spa_free_bplist = {{bpl_lock = {lock_object = {
          lo_name = 0xffffffff81bc03ee <.L.str.1+1> "bpl->bpl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
        sx_lock = 1}, bpl_list = {list_size = 144, list_offset = 128, list_head = {list_next = 0xfffffe004ebe8570, 
          list_prev = 0xfffffe004ebe8570}}}, {bpl_lock = {lock_object = {
          lo_name = 0xffffffff81bc03ee <.L.str.1+1> "bpl->bpl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
        sx_lock = 1}, bpl_list = {list_size = 144, list_offset = 128, list_head = {list_next = 0xfffffe004ebe85b0, 
          list_prev = 0xfffffe004ebe85b0}}}, {bpl_lock = {lock_object = {
--Type <RET> for more, q to quit, c to continue without paging--
          lo_name = 0xffffffff81bc03ee <.L.str.1+1> "bpl->bpl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
        sx_lock = 1}, bpl_list = {list_size = 144, list_offset = 128, list_head = {list_next = 0xfffffe004ebe85f0, 
          list_prev = 0xfffffe004ebe85f0}}}, {bpl_lock = {lock_object = {
          lo_name = 0xffffffff81bc03ee <.L.str.1+1> "bpl->bpl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
        sx_lock = 1}, bpl_list = {list_size = 144, list_offset = 128, list_head = {list_next = 0xfffffe004ebe8630, 
          list_prev = 0xfffffe004ebe8630}}}}, spa_cksum_salt = {
    zcs_bytes = "Þ8\235\tÂýWú\224\200ÀÐÞàW\264Àu\203Wõ5[s\226-Ç\276\002\367?{"}, spa_cksum_tmpls_lock = {lock_object = {
      lo_name = 0xffffffff81bc30ff <.L.str.20+1> "spa->spa_cksum_tmpls_lock", lo_flags = 577830912, lo_data = 0, 
      lo_witness = 0x0}, sx_lock = 1}, spa_cksum_tmpls = {0x0 <repeats 13 times>}, spa_ubsync = {ub_magic = 12235020, 
    ub_version = 5000, ub_txg = 2864863, ub_guid_sum = 1822098653521677264, ub_timestamp = 1569507067, ub_rootbp = {blk_dva = {{
          dva_word = {8, 447810216}}, {dva_word = {8, 738256336}}, {dva_word = {8, 1208077200}}}, blk_prop = 9226475966770118659, 
      blk_pad = {0, 0}, blk_phys_birth = 0, blk_birth = 2864862, blk_fill = 2024, blk_cksum = {zc_word = {16400598605, 
          7775230236364, 1847786449934068, 293504487061283720}}}, ub_software_version = 5000, ub_mmp_magic = 0, ub_mmp_delay = 0, 
    ub_mmp_seq = 0, ub_checkpoint_txg = 0}, spa_uberblock = {ub_magic = 12235020, ub_version = 5000, ub_txg = 2864863, 
    ub_guid_sum = 1822098653521677264, ub_timestamp = 1569507067, ub_rootbp = {blk_dva = {{dva_word = {8, 447810216}}, {
          dva_word = {8, 738256336}}, {dva_word = {8, 1208077200}}}, blk_prop = 9226475966770118659, blk_pad = {0, 0}, 
      blk_phys_birth = 0, blk_birth = 2864862, blk_fill = 2024, blk_cksum = {zc_word = {16400598605, 7775230236364, 
          1847786449934068, 293504487061283720}}}, ub_software_version = 5000, ub_mmp_magic = 0, ub_mmp_delay = 0, ub_mmp_seq = 0, 
    ub_checkpoint_txg = 0}, spa_extreme_rewind = 0, spa_last_io = 5106579, spa_scrub_lock = {lock_object = {
      lo_name = 0xffffffff81bb612d <.L.str.21+1> "spa->spa_scrub_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
    sx_lock = 1}, spa_scrub_inflight = 0, spa_load_verify_ios = 0, spa_scrub_io_cv = {
    cv_description = 0xffffffff81be7b03 <.L.str.28+1> "spa->spa_scrub_io_cv", cv_waiters = 0}, spa_scrub_active = 0 '\000', 
  spa_scrub_type = 0 '\000', spa_scrub_finished = 0 '\000', spa_scrub_started = 0 '\000', spa_scrub_reopen = 0 '\000', 
  spa_scan_pass_start = 1569501933, spa_scan_pass_scrub_pause = 0, spa_scan_pass_scrub_spent_paused = 0, spa_scan_pass_exam = 0, 
  spa_scan_pass_issued = 0, spa_async_lock = {lock_object = {lo_name = 0xffffffff81bbf0d9 <.L.str.13+1> "spa->spa_async_lock", 
      lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_async_thread = 0x0, spa_async_thread_vd = 0x0, 
  spa_async_suspended = 0, spa_async_cv = {cv_description = 0xffffffff81bce39b <.L.str.25+1> "spa->spa_async_cv", cv_waiters = 0}, 
  spa_async_tasks = 0, spa_missing_tvds = 0, spa_missing_tvds_allowed = 0, spa_removing_phys = {sr_state = 0, 
    sr_removing_vdev = 18446744073709551615, sr_prev_indirect_vdev = 18446744073709551615, sr_start_time = 0, sr_end_time = 0, 
    sr_to_copy = 0, sr_copied = 0}, spa_vdev_removal = 0x0, spa_condensing_indirect_phys = {scip_vdev = 0, 
    scip_prev_obsolete_sm_object = 0, scip_next_mapping_object = 0}, spa_condensing_indirect = 0x0, 
  spa_condense_zthr = 0xfffff80008533900, spa_checkpoint_txg = 0, spa_checkpoint_info = {sci_timestamp = 0, sci_dspace = 0}, 
  spa_checkpoint_discard_zthr = 0xfffff80008533880, spa_root = 0x0, spa_ena = 0, spa_last_open_failed = 0, 
  spa_last_ubsync_txg = 0, spa_last_ubsync_txg_ts = 0, spa_load_txg = 0, spa_load_txg_ts = 1569501768, spa_load_meta_errors = 0, 
  spa_load_data_errors = 0, spa_verify_min_txg = 2863813, spa_errlog_lock = {lock_object = {
      lo_name = 0xffffffff81bc8a75 <.L.str.15+1> "spa->spa_errlog_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
    sx_lock = 1}, spa_errlog_last = 0, spa_errlog_scrub = 0, spa_errlist_lock = {lock_object = {
      lo_name = 0xffffffff81be958d <.L.str.14+1> "spa->spa_errlist_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
    sx_lock = 1}, spa_errlist_last = {avl_root = 0x0, avl_compar = 0xffffffff81af53f0 <spa_error_entry_compare>, avl_offset = 40, 
    avl_numnodes = 0, avl_size = 64}, spa_errlist_scrub = {avl_root = 0x0, 
    avl_compar = 0xffffffff81af53f0 <spa_error_entry_compare>, avl_offset = 40, avl_numnodes = 0, avl_size = 64}, spa_deflate = 1, 
  spa_history = 62, spa_history_lock = {lock_object = {lo_name = 0xffffffff81bd7331 <.L.str.17+1> "spa->spa_history_lock", 
      lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_pending_vdev = 0x0, spa_props_lock = {lock_object = {
      lo_name = 0xffffffff81be5e02 <.L.str.19+1> "spa->spa_props_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
--Type <RET> for more, q to quit, c to continue without paging--
    sx_lock = 1}, spa_pool_props_object = 0, spa_bootfs = 0, spa_failmode = 0, spa_delegation = 1, spa_config_list = {
    list_size = 24, list_offset = 0, list_head = {list_next = 0xfffff80004c94260, list_prev = 0xfffff80004c94260}}, 
  spa_async_zio_root = 0xfffff80004ded0c0, spa_suspend_zio_root = 0x0, spa_txg_zio = {0xfffff800818cc418, 0xfffff80004c6b418, 
    0xfffff800818dd418, 0xfffff80081848000}, spa_suspend_lock = {lock_object = {
      lo_name = 0xffffffff81bdccac <.L.str.22+1> "spa->spa_suspend_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
    sx_lock = 1}, spa_suspend_cv = {cv_description = 0xffffffff81bb6142 <.L.str.29+1> "spa->spa_suspend_cv", cv_waiters = 0}, 
  spa_suspended = 0 '\000', spa_claiming = 0 '\000', spa_is_root = 0, spa_minref = 11, spa_mode = 3, spa_log_state = SPA_LOG_GOOD, 
  spa_autoexpand = 0, spa_bootsize = 0, spa_ddt = {0xfffffe004bd98000, 0xfffffe004bda0000, 0xfffffe004bda8000, 0xfffffe004bdb0000, 
    0xfffffe004bdb8000, 0xfffffe004bdc0000, 0xfffffe004bdc8000, 0xfffffe004bdd0000, 0xfffffe004bdd8000, 0xfffffe004bde0000, 
    0xfffffe004fa00000, 0xfffffe004fa08000, 0xfffffe004fa10000}, spa_ddt_stat_object = 0, spa_dedup_ditto = 0, 
  spa_dedup_checksum = 8, spa_dspace = 987842478080, spa_vdev_top_lock = {lock_object = {
      lo_name = 0xffffffff81be0b46 <.L.str.23+1> "spa->spa_vdev_top_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
    sx_lock = 1}, spa_proc_lock = {lock_object = {lo_name = 0xffffffff81bce387 <.L.str.18+1> "spa->spa_proc_lock", 
      lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_proc_cv = {
    cv_description = 0xffffffff81beb620 <.L.str.27+1> "spa->spa_proc_cv", cv_waiters = 0}, spa_proc_state = SPA_PROC_NONE, 
  spa_proc = 0xffffffff814e6fb0 <proc0>, spa_did = 0, spa_trim_thread = 0xfffff80008af85e0, spa_trim_lock = {lock_object = {
      lo_name = 0xffffffff81be1317 <.L.str.8+1> "spa->spa_trim_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
    sx_lock = 1}, spa_trim_cv = {cv_description = 0xffffffff81bd7d34 <.L.str.9+1> "spa->spa_trim_cv", cv_waiters = 4795}, 
  spa_autoreplace = 0, spa_vdev_locks = 0, spa_creation_version = 5000, spa_prev_software_version = 5000, 
  spa_feat_for_write_obj = 59, spa_feat_for_read_obj = 58, spa_feat_desc_obj = 60, spa_feat_enabled_txg_obj = 63, 
  spa_feat_stats_lock = {lock_object = {lo_name = 0xffffffff81bd7348 <.L.str.24+1> "spa->spa_feat_stats_lock", 
      lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_feat_stats = 0xfffff800087d1ca0, 
  spa_feat_refcount_cache = {0, 3399, 1, 0, 116, 14, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1}, spa_deadman_cycid = {c_links = {le = {
        le_next = 0x0, le_prev = 0xfffffe00031b3f60}, sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, 
        tqe_prev = 0xfffffe00031b3f60}}, c_time = 26413723334338, c_precision = 268435437500, c_arg = 0xfffffe004ebe8000, 
    c_func = 0xffffffff81b08630 <spa_deadman_timeout>, c_lock = 0x0, c_flags = 2, c_iflags = 20, c_cpu = 0}, spa_deadman_task = {
    ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xffffffff81b08560 <spa_deadman>, 
    ta_context = 0xfffffe004ebe8000}, spa_deadman_calls = 0, spa_sync_starttime = 5149924275093, 
  spa_deadman_synctime = 1000000000000, spa_all_vdev_zaps = 64, spa_avz_action = AVZ_ACTION_NONE, spa_lowmem_page_load = 0, 
  spa_lowmem_last_txg = 0, spa_ccw_fail_time = 0, spa_config_lock = {{scl_lock = {lock_object = {
          lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
        sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = {
        cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = {
            lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
          sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe8f38, 
            list_prev = 0xfffffe004ebe8f38}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
            list_next = 0xfffffe004ebe8f58, list_prev = 0xfffffe004ebe8f58}}, rc_count = 1, rc_removed_count = 0}}, {scl_lock = {
        lock_object = {lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, 
          lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = {
        cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = {
            lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
          sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe8ff0, 
            list_prev = 0xfffffe004ebe8ff0}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
            list_next = 0xfffffe004ebe9010, list_prev = 0xfffffe004ebe9010}}, rc_count = 0, rc_removed_count = 0}}, {scl_lock = {
--Type <RET> for more, q to quit, c to continue without paging--
        lock_object = {lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, 
          lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = {
        cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = {
            lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
          sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe90a8, 
            list_prev = 0xfffffe004ebe90a8}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
            list_next = 0xfffffe004ebe90c8, list_prev = 0xfffffe004ebe90c8}}, rc_count = 0, rc_removed_count = 0}}, {scl_lock = {
        lock_object = {lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, 
          lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = {
        cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = {
            lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
          sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe9160, 
            list_prev = 0xfffffe004ebe9160}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
            list_next = 0xfffffe004ebe9180, list_prev = 0xfffffe004ebe9180}}, rc_count = 0, rc_removed_count = 0}}, {scl_lock = {
        lock_object = {lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, 
          lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = {
        cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = {
            lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
          sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe9218, 
            list_prev = 0xfffffe004ebe9218}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
            list_next = 0xfffffe004ebe9238, list_prev = 0xfffffe004ebe9238}}, rc_count = 0, rc_removed_count = 0}}, {scl_lock = {
        lock_object = {lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, 
          lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = {
        cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = {
            lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
          sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe92d0, 
            list_prev = 0xfffffe004ebe92d0}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
            list_next = 0xfffffe004ebe92f0, list_prev = 0xfffffe004ebe92f0}}, rc_count = 0, rc_removed_count = 0}}, {scl_lock = {
        lock_object = {lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, 
          lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = {
        cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = {
            lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, 
          sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe9388, 
            list_prev = 0xfffffe004ebe9388}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {
            list_next = 0xfffffe004ebe93a8, list_prev = 0xfffffe004ebe93a8}}, rc_count = 0, rc_removed_count = 0}}}, 
  spa_refcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, 
        lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {
        list_next = 0xfffffe004ebe9400, list_prev = 0xfffffe004ebe9400}}, rc_removed = {list_size = 40, list_offset = 0, 
      list_head = {list_next = 0xfffffe004ebe9420, list_prev = 0xfffffe004ebe9420}}, rc_count = 64, rc_removed_count = 0}, 
  spa_splitting_newspa = 0}
(kgdb)