FreeBSD morzine.restart.bel 12.1-STABLE FreeBSD 12.1-STABLE r352648 MORZINE amd64 Panic during a zfs snapshot -r: panic: solaris assert: txg_list_empty (&dp->dp_dirty_dirs, txg) /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c line 8135
Could you provide more information, including: - zfs list (as an attachment) - /var/run/dmesg.boot output (as an attachment) - Backtrace of the panic
Created attachment 207850 [details] zfs list
Created attachment 207851 [details] dmesg.boot
zfs snapshot -r on rpool work as expected zfs snapshot -r in dspool panic. The backtrace (transcrit by hand): spa_sync() at spa_sync+0x1a0d/frame ... txg_sync_thread() at txg_sync_thread+0x307/frame... fork_exit() ...
You need to configure a dump device and get a crash dump (vmcore file).
The crash dump can be found at: http://tignes.restart.be/Xfer/
(In reply to hlh from comment #6) Since you are running a custom build, it's better to analyze the core in your environment. Alternatively, you will need to share your kernel and relevant modules.
I put the kernel and the modules used in /Xfer/kernel/ If you want I can do some analysis on the core file if you give me the command to run. By the way I also put core.txt.0 in /Xfer/
(In reply to hlh from comment #8) It would be more convenient if kernel/ was packaged as an archive (e.g., tar). Also, the vmcore link gives me 403. Some kgdb commands that you could run meanwhile: frame 13 info local frame 12 info local
(kgdb) frame 13 #13 0xffffffff81b0098d in spa_sync (spa=<optimized out>, txg=2864864) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:8135 8135 ASSERT(txg_list_empty(&dp->dp_dirty_dirs, txg)); (kgdb) info local pass = 1 dp = 0xfffff80004b72800 mos = 0xfffff80008af3400 rvd = 0xfffff80008c44000 vd = <optimized out> tx = 0xfffff8024ff55700 slots_per_allocator = <optimized out> max_queue_depth = <optimized out> mc = <optimized out> free_bpl = 0xfffffe004ebe8540 error = <optimized out> (kgdb) frame 12 #12 0xffffffff81eac21a in assfail (a=0x80 <error: Cannot access memory at address 0x80>, f=0xfffffe004f894460 "", l=128) at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81 81 panic("solaris assert: %s, file: %s, line: %d", a, f, l); (kgdb) info local No locals.
(In reply to hlh from comment #10) In frame 13: p *dp p *dp->dp_spa The output will be rather large.
kgdb) p *dp $1 = {dp_spa = 0xfffffe004ebe8000, dp_meta_objset = 0xfffff80008af3400, dp_root_dir = 0xfffff80004e75c00, dp_mos_dir = 0xfffff80004e75800, dp_free_dir = 0xfffff80004a68c00, dp_leak_dir = 0x0, dp_origin_snap = 0xfffff80004a6d000, dp_root_dir_obj = 32, dp_vnrele_taskq = 0xfffff80004bc7c20, dp_meta_rootbp = {blk_dva = {{dva_word = {8, 447810216}}, { dva_word = {8, 738256336}}, {dva_word = {8, 1208077200}}}, blk_prop = 9226475966770118659, blk_pad = {0, 0}, blk_phys_birth = 0, blk_birth = 2864862, blk_fill = 2024, blk_cksum = {zc_word = {16400598605, 7775230236364, 1847786449934068, 293504487061283720}}}, dp_tmp_userrefs_obj = 0, dp_free_bpobj = {bpo_lock = {lock_object = { lo_name = 0xffffffff81bd0df4 <.L.str.19+1> "bpo->bpo_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, bpo_os = 0xfffff80008af3400, bpo_object = 41, bpo_epb = 1024, bpo_havecomp = 1 '\001', bpo_havesubobj = 1 '\001', bpo_phys = 0xfffff80008977a00, bpo_dbuf = 0xfffff8000845d420, bpo_cached_dbuf = 0x0}, dp_bptree_obj = 0, dp_empty_bpobj = 20, dp_obsolete_bpobj = {bpo_lock = {lock_object = {lo_name = 0x0, lo_flags = 0, lo_data = 0, lo_witness = 0x0}, sx_lock = 0}, bpo_os = 0x0, bpo_object = 0, bpo_epb = 0, bpo_havecomp = 0 '\000', bpo_havesubobj = 0 '\000', bpo_phys = 0x0, bpo_dbuf = 0x0, bpo_cached_dbuf = 0x0}, dp_scan = 0xfffff80004e73c00, dp_lock = { lock_object = {lo_name = 0xffffffff81be58bf <.L.str.71+1> "dp->dp_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, dp_spaceavail_cv = {cv_description = 0xffffffff81bb59a8 <.L.str.72+1> "dp->dp_spaceavail_cv", cv_waiters = 0}, dp_dirty_pertxg = {0, 0, 0, 0}, dp_dirty_total = 0, dp_long_free_dirty_pertxg = {0, 0, 0, 0}, dp_mos_used_delta = 0, dp_mos_compressed_delta = 0, dp_mos_uncompressed_delta = 0, dp_last_wakeup = 0, dp_tx = {tx_cpu = 0xfffff800033cf800, tx_sync_lock = {lock_object = {lo_name = 0xffffffff81bd38ff <.L.str.7+1> "tx->tx_sync_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tx_open_txg = 2864865, tx_quiescing_txg = 0, tx_quiesced_txg = 0, tx_syncing_txg = 2864864, tx_synced_txg = 2864863, tx_open_time = 5149924275093, tx_sync_txg_waiting = 2864864, tx_quiesce_txg_waiting = 2864865, tx_sync_more_cv = {cv_description = 0xffffffff81bdade6 <.L.str.8+1> "tx->tx_sync_more_cv", cv_waiters = 0}, tx_sync_done_cv = {cv_description = 0xffffffff81beb698 <.L.str.9+1> "tx->tx_sync_done_cv", cv_waiters = 1}, tx_quiesce_more_cv = {cv_description = 0xffffffff81bb6249 <.L.str.10+1> "tx->tx_quiesce_more_cv", cv_waiters = 1}, tx_quiesce_done_cv = {cv_description = 0xffffffff81bd53c7 <.L.str.11+1> "tx->tx_quiesce_done_cv", cv_waiters = 0}, tx_timeout_cv = {cv_description = 0x0, cv_waiters = 0}, tx_exit_cv = { cv_description = 0xffffffff81bc31f1 <.L.str.12+1> "tx->tx_exit_cv", cv_waiters = 0}, tx_threads = 2 '\002', tx_exiting = 0 '\000', tx_sync_thread = 0xfffff80008e9d000, tx_quiesce_thread = 0xfffff80008e9c000, tx_commit_cb_taskq = 0x0}, dp_dirty_datasets = {tl_lock = {lock_object = {lo_name = 0xffffffff81be221e <.L.str.28+1> "tl->tl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tl_offset = 928, tl_spa = 0xfffffe004ebe8000, tl_head = {0x0, 0x0, 0x0, 0x0}}, dp_dirty_zilogs = {tl_lock = {lock_object = {lo_name = 0xffffffff81be221e <.L.str.28+1> "tl->tl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tl_offset = 840, tl_spa = 0xfffffe004ebe8000, tl_head = {0x0, 0x0, 0x0, 0x0}}, dp_dirty_dirs = {tl_lock = {lock_object = { lo_name = 0xffffffff81be221e <.L.str.28+1> "tl->tl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tl_offset = 96, tl_spa = 0xfffffe004ebe8000, tl_head = {0xfffff80008fb2c60, 0x0, 0x0, 0x0}}, dp_sync_tasks = { tl_lock = {lock_object = {lo_name = 0xffffffff81be221e <.L.str.28+1> "tl->tl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tl_offset = 0, tl_spa = 0xfffffe004ebe8000, tl_head = {0x0, 0x0, 0x0, 0x0}}, dp_early_sync_tasks = {tl_lock = {lock_object = {lo_name = 0xffffffff81be221e <.L.str.28+1> "tl->tl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tl_offset = 0, tl_spa = 0xfffffe004ebe8000, tl_head = {0x0, 0x0, 0x0, 0x0}}, dp_sync_taskq = 0xfffff80004bc7c40, dp_zil_clean_taskq = 0xfffff80004bc7c30, dp_config_rwlock = {rr_lock = {lock_object = { lo_name = 0xffffffff81bb98ae <.L.str.1+1> "rrl->rr_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rr_cv = {cv_description = 0xffffffff81bc8820 <.L.str.2+1> "rrl->rr_cv", cv_waiters = 0}, rr_writer = 0x0, rr_anon_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = { list_next = 0xfffff80004b72d18, list_prev = 0xfffff80004b72d18}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffff80004b72d38, list_prev = 0xfffff80004b72d38}}, rc_count = 0, rc_removed_count = 0}, --Type <RET> for more, q to quit, c to continue without paging-- rr_linked_rcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = { list_next = 0xfffff80004b72d90, list_prev = 0xfffff80004b72d90}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffff80004b72db0, list_prev = 0xfffff80004b72db0}}, rc_count = 0, rc_removed_count = 0}, rr_writer_wanted = 0, rr_track_all = 1}, dp_blkstats = 0x0} (kgdb) p *dp->dp_spa $2 = {spa_name = "dpool", '\000' <repeats 250 times>, spa_comment = 0x0, spa_avl = {avl_child = {0x0, 0x0}, avl_pcb = 18446741875996868873}, spa_config = 0xfffff80008c45800, spa_config_syncing = 0x0, spa_config_splitting = 0x0, spa_load_info = 0xfffff80004c94220, spa_config_txg = 2029544, spa_sync_pass = 1, spa_state = POOL_STATE_ACTIVE, spa_inject_ref = 0, spa_sync_on = 1 '\001', spa_load_state = SPA_LOAD_NONE, spa_indirect_vdevs_loaded = 1, spa_trust_config = 1, spa_config_source = SPA_CONFIG_SRC_MOS, spa_import_flags = 0, spa_zio_taskq = {{{stqs_count = 1, stqs_taskq = 0xfffff80004deff00}, {stqs_count = 0, stqs_taskq = 0x0}, {stqs_count = 1, stqs_taskq = 0xfffff8000496b3c0}, { stqs_count = 0, stqs_taskq = 0x0}}, {{stqs_count = 1, stqs_taskq = 0xfffff8000496b3e0}, {stqs_count = 0, stqs_taskq = 0x0}, {stqs_count = 8, stqs_taskq = 0xfffff800087e38c0}, {stqs_count = 0, stqs_taskq = 0x0}}, {{ stqs_count = 1, stqs_taskq = 0xfffff80004bc7e10}, {stqs_count = 1, stqs_taskq = 0xfffff80004bc7df0}, {stqs_count = 1, stqs_taskq = 0xfffff80004bc7dd0}, {stqs_count = 1, stqs_taskq = 0xfffff80004bc7db0}}, {{stqs_count = 8, stqs_taskq = 0xfffff800087e3180}, {stqs_count = 0, stqs_taskq = 0x0}, {stqs_count = 1, stqs_taskq = 0xfffff80004bc7d10}, { stqs_count = 0, stqs_taskq = 0x0}}, {{stqs_count = 1, stqs_taskq = 0xfffff80004bc7cf0}, {stqs_count = 0, stqs_taskq = 0x0}, {stqs_count = 1, stqs_taskq = 0xfffff80004bc7cd0}, {stqs_count = 0, stqs_taskq = 0x0}}, {{ stqs_count = 1, stqs_taskq = 0xfffff80004bc7cb0}, {stqs_count = 0, stqs_taskq = 0x0}, {stqs_count = 1, stqs_taskq = 0xfffff80004bc7c90}, {stqs_count = 0, stqs_taskq = 0x0}}}, spa_dsl_pool = 0xfffff80004b72800, spa_is_initializing = 0, spa_normal_class = 0xfffff80004e74000, spa_log_class = 0xfffff80004a68400, spa_first_txg = 2863817, spa_final_txg = 18446744073709551615, spa_freeze_txg = 18446744073709551615, spa_load_max_txg = 18446744073709551615, spa_claim_max_txg = 2863817, spa_loaded_ts = {tv_sec = 1569501932, tv_nsec = 655368601}, spa_meta_objset = 0xfffff80008af3400, spa_evicting_os_lock = {lock_object = {lo_name = 0xffffffff81bce36c <.L.str.16+1> "spa->spa_evicting_os_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_evicting_os_list = {list_size = 896, list_offset = 304, list_head = {list_next = 0xfffffe004ebe8380, list_prev = 0xfffffe004ebe8380}}, spa_evicting_os_cv = { cv_description = 0xffffffff81bd1a69 <.L.str.26+1> "spa->spa_evicting_os_cv", cv_waiters = 0}, spa_vdev_txg_list = {tl_lock = { lock_object = {lo_name = 0xffffffff81be221e <.L.str.28+1> "tl->tl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, tl_offset = 688, tl_spa = 0xfffffe004ebe8000, tl_head = {0x0, 0x0, 0x0, 0xfffff80008c432b0}}, spa_root_vdev = 0xfffff80008c44000, spa_min_ashift = 12, spa_max_ashift = 12, spa_config_guid = 11714503416111511200, spa_load_guid = 6215109065070704231, spa_last_synced_guid = 11714503416111511200, spa_config_dirty_list = {list_size = 2280, list_offset = 736, list_head = {list_next = 0xfffffe004ebe8428, list_prev = 0xfffffe004ebe8428}}, spa_state_dirty_list = {list_size = 2280, list_offset = 752, list_head = { list_next = 0xfffffe004ebe8448, list_prev = 0xfffffe004ebe8448}}, spa_alloc_locks = 0xfffff80008180380, spa_alloc_trees = 0xfffff80004bf7300, spa_alloc_count = 4, spa_spares = {sav_object = 0, sav_config = 0x0, sav_vdevs = 0x0, sav_count = 0, sav_sync = 0, sav_pending = 0x0, sav_npending = 0}, spa_l2cache = {sav_object = 275, sav_config = 0xfffff80004dee0a0, sav_vdevs = 0xfffff8000496bb70, sav_count = 1, sav_sync = 0, sav_pending = 0x0, sav_npending = 0}, spa_label_features = 0xfffff80004c94200, spa_config_object = 57, spa_config_generation = 0, spa_syncing_txg = 2864864, spa_deferred_bpobj = {bpo_lock = {lock_object = { lo_name = 0xffffffff81bd0df4 <.L.str.19+1> "bpo->bpo_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, bpo_os = 0xfffff80008af3400, bpo_object = 61, bpo_epb = 128, bpo_havecomp = 1 '\001', bpo_havesubobj = 1 '\001', bpo_phys = 0xfffff80008977800, bpo_dbuf = 0xfffff800089709a0, bpo_cached_dbuf = 0xfffff8000d450000}, spa_free_bplist = {{bpl_lock = {lock_object = { lo_name = 0xffffffff81bc03ee <.L.str.1+1> "bpl->bpl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, bpl_list = {list_size = 144, list_offset = 128, list_head = {list_next = 0xfffffe004ebe8570, list_prev = 0xfffffe004ebe8570}}}, {bpl_lock = {lock_object = { lo_name = 0xffffffff81bc03ee <.L.str.1+1> "bpl->bpl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, bpl_list = {list_size = 144, list_offset = 128, list_head = {list_next = 0xfffffe004ebe85b0, list_prev = 0xfffffe004ebe85b0}}}, {bpl_lock = {lock_object = { --Type <RET> for more, q to quit, c to continue without paging-- lo_name = 0xffffffff81bc03ee <.L.str.1+1> "bpl->bpl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, bpl_list = {list_size = 144, list_offset = 128, list_head = {list_next = 0xfffffe004ebe85f0, list_prev = 0xfffffe004ebe85f0}}}, {bpl_lock = {lock_object = { lo_name = 0xffffffff81bc03ee <.L.str.1+1> "bpl->bpl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, bpl_list = {list_size = 144, list_offset = 128, list_head = {list_next = 0xfffffe004ebe8630, list_prev = 0xfffffe004ebe8630}}}}, spa_cksum_salt = { zcs_bytes = "Þ8\235\tÂýWú\224\200ÀÐÞàW\264Àu\203Wõ5[s\226-Ç\276\002\367?{"}, spa_cksum_tmpls_lock = {lock_object = { lo_name = 0xffffffff81bc30ff <.L.str.20+1> "spa->spa_cksum_tmpls_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_cksum_tmpls = {0x0 <repeats 13 times>}, spa_ubsync = {ub_magic = 12235020, ub_version = 5000, ub_txg = 2864863, ub_guid_sum = 1822098653521677264, ub_timestamp = 1569507067, ub_rootbp = {blk_dva = {{ dva_word = {8, 447810216}}, {dva_word = {8, 738256336}}, {dva_word = {8, 1208077200}}}, blk_prop = 9226475966770118659, blk_pad = {0, 0}, blk_phys_birth = 0, blk_birth = 2864862, blk_fill = 2024, blk_cksum = {zc_word = {16400598605, 7775230236364, 1847786449934068, 293504487061283720}}}, ub_software_version = 5000, ub_mmp_magic = 0, ub_mmp_delay = 0, ub_mmp_seq = 0, ub_checkpoint_txg = 0}, spa_uberblock = {ub_magic = 12235020, ub_version = 5000, ub_txg = 2864863, ub_guid_sum = 1822098653521677264, ub_timestamp = 1569507067, ub_rootbp = {blk_dva = {{dva_word = {8, 447810216}}, { dva_word = {8, 738256336}}, {dva_word = {8, 1208077200}}}, blk_prop = 9226475966770118659, blk_pad = {0, 0}, blk_phys_birth = 0, blk_birth = 2864862, blk_fill = 2024, blk_cksum = {zc_word = {16400598605, 7775230236364, 1847786449934068, 293504487061283720}}}, ub_software_version = 5000, ub_mmp_magic = 0, ub_mmp_delay = 0, ub_mmp_seq = 0, ub_checkpoint_txg = 0}, spa_extreme_rewind = 0, spa_last_io = 5106579, spa_scrub_lock = {lock_object = { lo_name = 0xffffffff81bb612d <.L.str.21+1> "spa->spa_scrub_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_scrub_inflight = 0, spa_load_verify_ios = 0, spa_scrub_io_cv = { cv_description = 0xffffffff81be7b03 <.L.str.28+1> "spa->spa_scrub_io_cv", cv_waiters = 0}, spa_scrub_active = 0 '\000', spa_scrub_type = 0 '\000', spa_scrub_finished = 0 '\000', spa_scrub_started = 0 '\000', spa_scrub_reopen = 0 '\000', spa_scan_pass_start = 1569501933, spa_scan_pass_scrub_pause = 0, spa_scan_pass_scrub_spent_paused = 0, spa_scan_pass_exam = 0, spa_scan_pass_issued = 0, spa_async_lock = {lock_object = {lo_name = 0xffffffff81bbf0d9 <.L.str.13+1> "spa->spa_async_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_async_thread = 0x0, spa_async_thread_vd = 0x0, spa_async_suspended = 0, spa_async_cv = {cv_description = 0xffffffff81bce39b <.L.str.25+1> "spa->spa_async_cv", cv_waiters = 0}, spa_async_tasks = 0, spa_missing_tvds = 0, spa_missing_tvds_allowed = 0, spa_removing_phys = {sr_state = 0, sr_removing_vdev = 18446744073709551615, sr_prev_indirect_vdev = 18446744073709551615, sr_start_time = 0, sr_end_time = 0, sr_to_copy = 0, sr_copied = 0}, spa_vdev_removal = 0x0, spa_condensing_indirect_phys = {scip_vdev = 0, scip_prev_obsolete_sm_object = 0, scip_next_mapping_object = 0}, spa_condensing_indirect = 0x0, spa_condense_zthr = 0xfffff80008533900, spa_checkpoint_txg = 0, spa_checkpoint_info = {sci_timestamp = 0, sci_dspace = 0}, spa_checkpoint_discard_zthr = 0xfffff80008533880, spa_root = 0x0, spa_ena = 0, spa_last_open_failed = 0, spa_last_ubsync_txg = 0, spa_last_ubsync_txg_ts = 0, spa_load_txg = 0, spa_load_txg_ts = 1569501768, spa_load_meta_errors = 0, spa_load_data_errors = 0, spa_verify_min_txg = 2863813, spa_errlog_lock = {lock_object = { lo_name = 0xffffffff81bc8a75 <.L.str.15+1> "spa->spa_errlog_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_errlog_last = 0, spa_errlog_scrub = 0, spa_errlist_lock = {lock_object = { lo_name = 0xffffffff81be958d <.L.str.14+1> "spa->spa_errlist_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_errlist_last = {avl_root = 0x0, avl_compar = 0xffffffff81af53f0 <spa_error_entry_compare>, avl_offset = 40, avl_numnodes = 0, avl_size = 64}, spa_errlist_scrub = {avl_root = 0x0, avl_compar = 0xffffffff81af53f0 <spa_error_entry_compare>, avl_offset = 40, avl_numnodes = 0, avl_size = 64}, spa_deflate = 1, spa_history = 62, spa_history_lock = {lock_object = {lo_name = 0xffffffff81bd7331 <.L.str.17+1> "spa->spa_history_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_pending_vdev = 0x0, spa_props_lock = {lock_object = { lo_name = 0xffffffff81be5e02 <.L.str.19+1> "spa->spa_props_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, --Type <RET> for more, q to quit, c to continue without paging-- sx_lock = 1}, spa_pool_props_object = 0, spa_bootfs = 0, spa_failmode = 0, spa_delegation = 1, spa_config_list = { list_size = 24, list_offset = 0, list_head = {list_next = 0xfffff80004c94260, list_prev = 0xfffff80004c94260}}, spa_async_zio_root = 0xfffff80004ded0c0, spa_suspend_zio_root = 0x0, spa_txg_zio = {0xfffff800818cc418, 0xfffff80004c6b418, 0xfffff800818dd418, 0xfffff80081848000}, spa_suspend_lock = {lock_object = { lo_name = 0xffffffff81bdccac <.L.str.22+1> "spa->spa_suspend_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_suspend_cv = {cv_description = 0xffffffff81bb6142 <.L.str.29+1> "spa->spa_suspend_cv", cv_waiters = 0}, spa_suspended = 0 '\000', spa_claiming = 0 '\000', spa_is_root = 0, spa_minref = 11, spa_mode = 3, spa_log_state = SPA_LOG_GOOD, spa_autoexpand = 0, spa_bootsize = 0, spa_ddt = {0xfffffe004bd98000, 0xfffffe004bda0000, 0xfffffe004bda8000, 0xfffffe004bdb0000, 0xfffffe004bdb8000, 0xfffffe004bdc0000, 0xfffffe004bdc8000, 0xfffffe004bdd0000, 0xfffffe004bdd8000, 0xfffffe004bde0000, 0xfffffe004fa00000, 0xfffffe004fa08000, 0xfffffe004fa10000}, spa_ddt_stat_object = 0, spa_dedup_ditto = 0, spa_dedup_checksum = 8, spa_dspace = 987842478080, spa_vdev_top_lock = {lock_object = { lo_name = 0xffffffff81be0b46 <.L.str.23+1> "spa->spa_vdev_top_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_proc_lock = {lock_object = {lo_name = 0xffffffff81bce387 <.L.str.18+1> "spa->spa_proc_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_proc_cv = { cv_description = 0xffffffff81beb620 <.L.str.27+1> "spa->spa_proc_cv", cv_waiters = 0}, spa_proc_state = SPA_PROC_NONE, spa_proc = 0xffffffff814e6fb0 <proc0>, spa_did = 0, spa_trim_thread = 0xfffff80008af85e0, spa_trim_lock = {lock_object = { lo_name = 0xffffffff81be1317 <.L.str.8+1> "spa->spa_trim_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_trim_cv = {cv_description = 0xffffffff81bd7d34 <.L.str.9+1> "spa->spa_trim_cv", cv_waiters = 4795}, spa_autoreplace = 0, spa_vdev_locks = 0, spa_creation_version = 5000, spa_prev_software_version = 5000, spa_feat_for_write_obj = 59, spa_feat_for_read_obj = 58, spa_feat_desc_obj = 60, spa_feat_enabled_txg_obj = 63, spa_feat_stats_lock = {lock_object = {lo_name = 0xffffffff81bd7348 <.L.str.24+1> "spa->spa_feat_stats_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, spa_feat_stats = 0xfffff800087d1ca0, spa_feat_refcount_cache = {0, 3399, 1, 0, 116, 14, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1}, spa_deadman_cycid = {c_links = {le = { le_next = 0x0, le_prev = 0xfffffe00031b3f60}, sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0xfffffe00031b3f60}}, c_time = 26413723334338, c_precision = 268435437500, c_arg = 0xfffffe004ebe8000, c_func = 0xffffffff81b08630 <spa_deadman_timeout>, c_lock = 0x0, c_flags = 2, c_iflags = 20, c_cpu = 0}, spa_deadman_task = { ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xffffffff81b08560 <spa_deadman>, ta_context = 0xfffffe004ebe8000}, spa_deadman_calls = 0, spa_sync_starttime = 5149924275093, spa_deadman_synctime = 1000000000000, spa_all_vdev_zaps = 64, spa_avz_action = AVZ_ACTION_NONE, spa_lowmem_page_load = 0, spa_lowmem_last_txg = 0, spa_ccw_fail_time = 0, spa_config_lock = {{scl_lock = {lock_object = { lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = { cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = { lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe8f38, list_prev = 0xfffffe004ebe8f38}}, rc_removed = {list_size = 40, list_offset = 0, list_head = { list_next = 0xfffffe004ebe8f58, list_prev = 0xfffffe004ebe8f58}}, rc_count = 1, rc_removed_count = 0}}, {scl_lock = { lock_object = {lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = { cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = { lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe8ff0, list_prev = 0xfffffe004ebe8ff0}}, rc_removed = {list_size = 40, list_offset = 0, list_head = { list_next = 0xfffffe004ebe9010, list_prev = 0xfffffe004ebe9010}}, rc_count = 0, rc_removed_count = 0}}, {scl_lock = { --Type <RET> for more, q to quit, c to continue without paging-- lock_object = {lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = { cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = { lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe90a8, list_prev = 0xfffffe004ebe90a8}}, rc_removed = {list_size = 40, list_offset = 0, list_head = { list_next = 0xfffffe004ebe90c8, list_prev = 0xfffffe004ebe90c8}}, rc_count = 0, rc_removed_count = 0}}, {scl_lock = { lock_object = {lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = { cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = { lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe9160, list_prev = 0xfffffe004ebe9160}}, rc_removed = {list_size = 40, list_offset = 0, list_head = { list_next = 0xfffffe004ebe9180, list_prev = 0xfffffe004ebe9180}}, rc_count = 0, rc_removed_count = 0}}, {scl_lock = { lock_object = {lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = { cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = { lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe9218, list_prev = 0xfffffe004ebe9218}}, rc_removed = {list_size = 40, list_offset = 0, list_head = { list_next = 0xfffffe004ebe9238, list_prev = 0xfffffe004ebe9238}}, rc_count = 0, rc_removed_count = 0}}, {scl_lock = { lock_object = {lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = { cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = { lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe92d0, list_prev = 0xfffffe004ebe92d0}}, rc_removed = {list_size = 40, list_offset = 0, list_head = { list_next = 0xfffffe004ebe92f0, list_prev = 0xfffffe004ebe92f0}}, rc_count = 0, rc_removed_count = 0}}, {scl_lock = { lock_object = {lo_name = 0xffffffff81bc4b2f <.L.str.106+1> "scl->scl_lock", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, scl_writer = 0x0, scl_write_wanted = 0, scl_cv = { cv_description = 0xffffffff81bdadbd <.L.str.107+1> "scl->scl_cv", cv_waiters = 0}, scl_count = {rc_mtx = {lock_object = { lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe9388, list_prev = 0xfffffe004ebe9388}}, rc_removed = {list_size = 40, list_offset = 0, list_head = { list_next = 0xfffffe004ebe93a8, list_prev = 0xfffffe004ebe93a8}}, rc_count = 0, rc_removed_count = 0}}}, spa_refcount = {rc_mtx = {lock_object = {lo_name = 0xffffffff81bd9017 <.L.str.3+1> "rc->rc_mtx", lo_flags = 577830912, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, rc_tracked = 0, rc_list = {list_size = 40, list_offset = 0, list_head = { list_next = 0xfffffe004ebe9400, list_prev = 0xfffffe004ebe9400}}, rc_removed = {list_size = 40, list_offset = 0, list_head = {list_next = 0xfffffe004ebe9420, list_prev = 0xfffffe004ebe9420}}, rc_count = 64, rc_removed_count = 0}, spa_splitting_newspa = 0} (kgdb)
I update to FreeBSD 12.1-STABLE #0 r353094 The problem is still there. I destroy dpool/VirtualBox/ubuntu (I need space, only 4GB left) And now `zfs snapshot -r dpool@snapxxx` works as expected...
I discover that `zfs snapshot -r dpool@snapxxx`work fine BUT if I run the _same_ command a second time the panic occurs. So the problem occurs when the snapshots already exist.
This is amongst bug reports that need special attention; see <https://lists.freebsd.org/archives/freebsd-fs/2023-April/002047.html>. (In reply to Henri Hennebert from comment #14) Please, do you know whether this behaviour is reproducible with ZFS in FreeBSD 12.4-RELEASE-p2? ---- I should assume that it's not an issue with OpenZFS in 13.2-RC6 and greater.
(In reply to Graham Perrin from comment #15) I'm running 13.2-STABLE and since OpenZFS the problem vanish :-)
Closing, with an assumption that a fix, if any, for stable/12 will not occur before the 2023-12-31 end of life. fs@ if you disagree, please reopen. (In reply to Henri Hennebert from comment #16) Thanks for the feedback.