cannot receive incremental updates from 12.1 to 12.2, zfs on the receiving side crashes. Non-incremental updates and received just fine. Console output from the sending (12.1) side: ===Cut=== [root@san1:~]# zfs send -vI @ver2_12322 data/reference@ver2_12329 | ssh emz@10.0.7.5 sudo zfs receive data send from @ver2_12322 to data/reference@ver2_12323 estimated size is 1,25G send from @ver2_12323 to data/reference@ver2_12324 estimated size is 3,21G send from @ver2_12324 to data/reference@ver2_12325 estimated size is 18,9G send from @ver2_12325 to data/reference@ver2_12326 estimated size is 28,3G send from @ver2_12326 to data/reference@ver2_12327 estimated size is 1,16G send from @ver2_12327 to data/reference@ver2_12328 estimated size is 11,9G send from @ver2_12328 to data/reference@ver2_12329 estimated size is 171G total estimated size is 236G send from @ver2_12322 to data/reference@ver2_12323 estimated size is 1,25G TIME SENT SNAPSHOT 14:09:23 2,74K data/reference@ver2_12323 14:09:24 2,74K data/reference@ver2_12323 14:09:25 2,74K data/reference@ver2_12323 14:09:26 2,74K data/reference@ver2_12323 14:09:27 2,74K data/reference@ver2_12323 14:09:28 2,74K data/reference@ver2_12323 14:09:29 2,74K data/reference@ver2_12323 internal error: Unknown error: 1029 warning: cannot send 'data/reference@ver2_12323': signal received ===Cut=== Receiving side dmesg: pid 1128 (zfs), jid 0, uid 0: exited on signal 6 (core dumped) pid 1127 (sudo), jid 0, uid 0: exited on signal 6 I have zfs.core, but not sure what tool to use to get the backtrace from it: /usr/libexec/gdb complains on it with the following: [...] Error while reading shared library symbols: Dwarf Error: wrong version in compilation unit header (is 4, should be 2) [in module /usr/lib/debug/lib/libjail.so.1.debug] [...and so on ...]
Created attachment 221233 [details] /sbin/zfs binary core Added the zfs binary core.
Created attachment 221234 [details] /sbin/zfs non-stripped binary Added /sbin/zfs non-stripped binary from receiving side (too).
Check zpool-features on both sides: zpool get all ZPOOL/NAME | grep feature@
Sending side misses feature@allocation_classes, but it's the situation when receiving side lacks features makes receiving impossible, right ? Furthermore, 12.2 still can receive non-incremental streams.
Backtrace: [root@san2:~]# gdb zfs.full zfs.core GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD] Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd12.2". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from zfs.full... warning: core file may not match specified executable file. [New LWP 102649] Core was generated by `zfs receive data'. Program terminated with signal SIGABRT, Aborted. #0 thr_kill () at thr_kill.S:3 3 RSYSCALL(thr_kill) (gdb) bt #0 thr_kill () at thr_kill.S:3 #1 0x00000008004a4084 in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:52 #2 0x000000080041a279 in abort () at /usr/src/lib/libc/stdlib/abort.c:67 #3 0x00000008002c9e70 in zfs_verror (hdl=0x800843000, error=2082, fmt=0x8002abb07 "%s", ap=0x7fffffff76e0) at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_util.c:309 #4 zfs_standard_error_fmt (hdl=0x800843000, error=<optimized out>, fmt=0x8002abb07 "%s") at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_util.c:438 #5 0x00000008002c9aa5 in zfs_standard_error (hdl=0x190f9, error=6, msg=0x8004a5c4a <thr_self+10> "\017\202\264G") at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_util.c:373 #6 0x00000008002c0592 in zfs_receive_one (hdl=<optimized out>, infd=0, tosnap=<optimized out>, originsnap=<optimized out>, flags=<optimized out>, drr=<optimized out>, drr_noswap=<optimized out>, sendfs=<optimized out>, stream_nv=<optimized out>, stream_avl=<optimized out>, top_zfs=<optimized out>, cleanup_fd=<optimized out>, action_handlep=<optimized out>, finalsnap=<optimized out>) at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c:3682 #7 zfs_receive_impl (hdl=<optimized out>, tosnap=<optimized out>, originsnap=<optimized out>, flags=<optimized out>, infd=0, sendfs=<optimized out>, stream_nv=0x0, stream_avl=0x0, top_zfs=0x7fffffffe1d8, cleanup_fd=8, action_handlep=0x7fffffffe1d0, finalsnap=0x7fffffffc66f "ver2_12329") at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c:3859 #8 0x00000008002c0112 in zfs_receive_package (hdl=<optimized out>, fd=0, destname=<optimized out>, flags=<optimized out>, drr=<optimized out>, zc=<optimized out>, top_zfs=<optimized out>, cleanup_fd=<optimized out>, action_handlep=<optimized out>) at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c:2957 #9 zfs_receive_impl (hdl=<optimized out>, tosnap=<optimized out>, originsnap=<optimized out>, flags=<optimized out>, infd=0, sendfs=<optimized out>, stream_nv=0x0, stream_avl=0x0, top_zfs=0x7fffffffe1d8, cleanup_fd=8, action_handlep=0x7fffffffe1d0, finalsnap=0x0) at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c:3865 #10 0x00000008002be8d1 in zfs_receive (hdl=0x800843000, tosnap=0x7fffffffee74 "data", props=<optimized out>, flags=0x7fffffffe230, infd=0, stream_avl=0x0) at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c:3895 #11 0x0000000000212582 in zfs_do_receive (argc=<optimized out>, argv=0x7fffffffec38) at /usr/src/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c:4189 #12 0x000000000020d505 in main (argc=<optimized out>, argv=0x7fffffffec30) at /usr/src/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c:7561 (gdb)
Full backtrace: (gdb) bt full #0 thr_kill () at thr_kill.S:3 No locals. #1 0x00000008004a4084 in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:52 id = 102649 #2 0x000000080041a279 in abort () at /usr/src/lib/libc/stdlib/abort.c:67 act = {__sigaction_u = {__sa_handler = 0x24, __sa_sigaction = 0x24}, sa_flags = 36, sa_mask = {__bits = {4294967263, 4294967295, 4294967295, 4294967295}}} #3 0x00000008002c9e70 in zfs_verror (hdl=0x800843000, error=2082, fmt=0x8002abb07 "%s", ap=0x7fffffff76e0) at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_util.c:309 No locals. #4 zfs_standard_error_fmt (hdl=0x800843000, error=<optimized out>, fmt=0x8002abb07 "%s") at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_util.c:438 ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fffffff7730, reg_save_area = 0x7fffffff7630}} #5 0x00000008002c9aa5 in zfs_standard_error (hdl=0x190f9, error=6, msg=0x8004a5c4a <thr_self+10> "\017\202\264G") at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_util.c:373 No locals. #6 0x00000008002c0592 in zfs_receive_one (hdl=<optimized out>, infd=0, tosnap=<optimized out>, originsnap=<optimized out>, flags=<optimized out>, drr=<optimized out>, drr_noswap=<optimized out>, sendfs=<optimized out>, stream_nv=<optimized out>, stream_avl=<optimized out>, top_zfs=<optimized out>, cleanup_fd=<optimized out>, action_handlep=<optimized out>, finalsnap=<optimized out>) at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c:3682 zc = {zc_name = "data\000ver2_12323", '\000' <repeats 1008 times>, zc_nvlist_src = 0, zc_nvlist_src_size = 0, zc_nvlist_dst = 0, zc_nvlist_dst_size = 0, zc_nvlist_dst_filled = B_FALSE, zc_pad2 = 0, zc_history = 0, zc_value = "data@ver2_12323", '\000' <repeats 2032 times>, zc_string = '\000' <repeats 255 times>, zc_guid = 0, zc_nvlist_conf = 0, zc_nvlist_conf_size = 0, zc_cookie = 0, zc_objset_type = 0, zc_perm_action = 0, zc_history_len = 0, zc_history_offset = 0, zc_obj = 0, zc_iflags = 0, zc_share = { z_exportdata = 0, z_sharedata = 0, z_sharetype = 0, z_sharemax = 0}, zc_jailid = 0, zc_objset_stats = {dds_num_clones = 0, dds_creation_txg = 0, dds_guid = 0, dds_type = DMU_OST_NONE, dds_is_snapshot = 0 '\000', dds_inconsistent = 0 '\000', dds_origin = '\000' <repeats 255 times>}, zc_begin_record = {drr_type = DRR_BEGIN, drr_payloadlen = 0, drr_u = {drr_begin = {drr_magic = 12712594348, drr_versioninfo = 1, drr_creation_time = 1609397741, drr_type = DMU_OST_ZVOL, drr_flags = 4, drr_toguid = 674406883349338168, drr_fromguid = 17010876038533095460, drr_toname = "data/reference@ver2_12323", '\000' <repeats 230 times>}, drr_end = { drr_checksum = {zc_word = {12712594348, 1, 1609397741, 17179869187}}, drr_toguid = 674406883349338168}, drr_object = {drr_object = 12712594348, drr_type = DMU_OT_OBJECT_DIRECTORY, drr_bonustype = DMU_OT_NONE, drr_blksz = 1609397741, drr_bonuslen = 0, drr_checksumtype = 3 '\003', drr_compress = 0 '\000', drr_dn_slots = 0 '\000', drr_pad = "\000\004\000\000", --Type <RET> for more, q to quit, c to continue without paging-- drr_toguid = 674406883349338168}, drr_freeobjects = {drr_firstobj = 12712594348, drr_numobjs = 1, drr_toguid = 1609397741}, drr_write = {drr_object = 12712594348, drr_type = DMU_OT_OBJECT_DIRECTORY, drr_pad = 0, drr_offset = 1609397741, drr_logical_size = 17179869187, drr_toguid = 674406883349338168, drr_checksumtype = 36 '$', drr_checksumflags = 32 ' ', drr_compressiontype = 248 '\370', drr_pad2 = "*\241\305", <incomplete sequence \354>, drr_key = {ddk_cksum = {zc_word = {7378429112340472164, 8520921972713288293, 3617290113340568165, 51}}, ddk_prop = 0}, drr_compressed_size = 0}, drr_free = { drr_object = 12712594348, drr_offset = 1, drr_length = 1609397741, drr_toguid = 17179869187}, drr_write_byref = {drr_object = 12712594348, drr_offset = 1, drr_length = 1609397741, drr_toguid = 17179869187, drr_refguid = 674406883349338168, drr_refobject = 17010876038533095460, drr_refoffset = 7378429112340472164, drr_checksumtype = 101 'e', drr_checksumflags = 114 'r', drr_pad2 = "ence@v", drr_key = {ddk_cksum = {zc_word = { 3617290113340568165, 51, 0, 0}}, ddk_prop = 0}}, drr_spill = { drr_object = 12712594348, drr_length = 1, drr_toguid = 1609397741, drr_pad = { 17179869187, 674406883349338168, 17010876038533095460, 7378429112340472164}}, drr_write_embedded = {drr_object = 12712594348, drr_offset = 1, drr_length = 1609397741, drr_toguid = 17179869187, drr_compression = 56 '8', drr_etype = 152 '\230', drr_pad = "\035\274|\371[\t", drr_lsize = 720904228, drr_psize = 3960653217}, drr_checksum = {drr_pad = {12712594348, 1, 1609397741, 17179869187, 674406883349338168, 17010876038533095460, 7378429112340472164, 8520921972713288293, 3617290113340568165, 51, 0 <repeats 24 times>}, drr_checksum = {zc_word = {0, 0, 0, 0}}}}}, zc_inject_record = {zi_objset = 0, zi_object = 0, zi_start = 0, zi_end = 0, zi_guid = 0, zi_level = 0, zi_error = 0, zi_type = 0, zi_freq = 0, zi_failfast = 0, zi_func = '\000' <repeats 255 times>, zi_iotype = 0, zi_duration = 0, zi_timer = 0, zi_nlanes = 0, zi_cmd = 0, zi_pad = 0}, zc_defer_destroy = 0, zc_flags = 0, zc_action_handle = 0, zc_cleanup_fd = 8, zc_simple = 0 '\000', zc_pad3 = "\000\000", zc_resumable = B_FALSE, zc_pad4 = 0, zc_sendobj = 0, zc_fromobj = 0, zc_createtxg = 0, zc_stat = {zs_gen = 0, zs_mode = 0, zs_links = 0, zs_ctime = {0, 0}}} errbuf = "cannot receive incremental stream", '\000' <repeats 990 times> prop_errbuf = '\000' <repeats 1023 times> parent_snapguid = <optimized out> newfs = <optimized out> drrb = <optimized out> snapprops_nvlist = <optimized out> clp = <optimized out> snapname = <optimized out> begin_time = <optimized out> err = <optimized out> cp = <optimized out> chopprefix = <optimized out> stream_resumingnewfs = <optimized out> ioctl_err = <optimized out> ioctl_errno = <optimized out> prop_errflags = <optimized out> stream_wantsnewfs = <optimized out> recursive = <optimized out> resuming = <optimized out> fs = <optimized out> props = <optimized out> ret = <optimized out> len = <optimized out> snap = {<optimized out> <repeats 256 times>} suffix = {<optimized out> <repeats 256 times>} zhp = <optimized out> is_volume = <optimized out> zhp = <optimized out> prop_errors = <optimized out> prop_err = <optimized out> tbuf = {<optimized out> <repeats 1024 times>} intval = <optimized out> prop = <optimized out> zc2 = {zc_name = {<optimized out> <repeats 1024 times>}, zc_nvlist_src = <optimized out>, zc_nvlist_src_size = <optimized out>, zc_nvlist_dst = <optimized out>, zc_nvlist_dst_size = <optimized out>, zc_nvlist_dst_filled = <optimized out>, zc_pad2 = <optimized out>, zc_history = <optimized out>, zc_value = { <optimized out> <repeats 2048 times>}, zc_string = {<optimized out> <repeats 256 times>}, zc_guid = <optimized out>, zc_nvlist_conf = <optimized out>, zc_nvlist_conf_size = <optimized out>, zc_cookie = <optimized out>, zc_objset_type = <optimized out>, zc_perm_action = <optimized out>, zc_history_len = <optimized out>, zc_history_offset = <optimized out>, zc_obj = <optimized out>, zc_iflags = <optimized out>, zc_share = { z_exportdata = <optimized out>, z_sharedata = <optimized out>, z_sharetype = <optimized out>, z_sharemax = <optimized out>}, zc_jailid = <optimized out>, zc_objset_stats = {dds_num_clones = <optimized out>, dds_creation_txg = <optimized out>, dds_guid = <optimized out>, dds_type = <optimized out>, dds_is_snapshot = <optimized out>, dds_inconsistent = <optimized out>, dds_origin = {<optimized out> <repeats 256 times>}}, zc_begin_record = {drr_type = <optimized out>, drr_payloadlen = <optimized out>, drr_u = { drr_begin = {drr_magic = <optimized out>, drr_versioninfo = <optimized out>, drr_creation_time = <optimized out>, drr_type = <optimized out>, drr_flags = <optimized out>, drr_toguid = <optimized out>, drr_fromguid = <optimized out>, drr_toname = {<optimized out> <repeats 256 times>}}, drr_end = {drr_checksum = {zc_word = {<optimized out>, <optimized out>, <optimized out>, <optimized out>}}, drr_toguid = <optimized out>}, drr_object = { drr_object = <optimized out>, drr_type = <optimized out>, drr_bonustype = <optimized out>, drr_blksz = <optimized out>, drr_bonuslen = <optimized out>, drr_checksumtype = <optimized out>, drr_compress = <optimized out>, drr_dn_slots = <optimized out>, drr_pad = { <optimized out>, <optimized out>, <optimized out>, <optimized out>, <optimized out>}, drr_toguid = <optimized out>}, drr_freeobjects = { drr_firstobj = <optimized out>, drr_numobjs = <optimized out>, drr_toguid = <optimized out>}, drr_write = {drr_object = <optimized out>, drr_type = <optimized out>, drr_pad = <optimized out>, drr_offset = <optimized out>, drr_logical_size = <optimized out>, drr_toguid = <optimized out>, drr_checksumtype = <optimized out>, drr_checksumflags = <optimized out>, drr_compressiontype = <optimized out>, drr_pad2 = {<optimized out>, <optimized out>, <optimized out>, <optimized out>, <optimized out>}, drr_key = {ddk_cksum = { zc_word = {<optimized out>, <optimized out>, <optimized out>, <optimized out>}}, ddk_prop = <optimized out>}, drr_compressed_size = <optimized out>}, drr_free = { drr_object = <optimized out>, drr_offset = <optimized out>, drr_length = <optimized out>, drr_toguid = <optimized out>}, drr_write_byref = { drr_object = <optimized out>, drr_offset = <optimized out>, drr_length = <optimized out>, drr_toguid = <optimized out>, drr_refguid = <optimized out>, drr_refobject = <optimized out>, drr_refoffset = <optimized out>, drr_checksumtype = <optimized out>, drr_checksumflags = <optimized out>, drr_pad2 = {<optimized out>, <optimized out>, <optimized out>, <optimized out>, <optimized out>, <optimized out>}, drr_key = { ddk_cksum = {zc_word = {<optimized out>, <optimized out>, <optimized out>, <optimized out>}}, ddk_prop = <optimized out>}}, drr_spill = { drr_object = <optimized out>, drr_length = <optimized out>, drr_toguid = <optimized out>, drr_pad = {<optimized out>, <optimized out>, <optimized out>, <optimized out>}}, drr_write_embedded = { drr_object = <optimized out>, drr_offset = <optimized out>, drr_length = <optimized out>, drr_toguid = <optimized out>, drr_compression = <optimized out>, drr_etype = <optimized out>, drr_pad = { <optimized out>, <optimized out>, <optimized out>, <optimized out>, <optimized out>, <optimized out>}, drr_lsize = <optimized out>, drr_psize = <optimized out>}, drr_checksum = {drr_pad = { <optimized out> <repeats 34 times>}, drr_checksum = {zc_word = {<optimized out>, <optimized out>, <optimized out>, <optimized out>}}}}}, zc_inject_record = { zi_objset = <optimized out>, zi_object = <optimized out>, zi_start = <optimized out>, zi_end = <optimized out>, zi_guid = <optimized out>, zi_level = <optimized out>, zi_error = <optimized out>, zi_type = <optimized out>, zi_freq = <optimized out>, zi_failfast = <optimized out>, zi_func = {<optimized out> <repeats 256 times>}, zi_iotype = <optimized out>, zi_duration = <optimized out>, zi_timer = <optimized out>, zi_nlanes = <optimized out>, zi_cmd = <optimized out>, zi_pad = <optimized out>}, zc_defer_destroy = <optimized out>, zc_flags = <optimized out>, zc_action_handle = <optimized out>, zc_cleanup_fd = <optimized out>, zc_simple = <optimized out>, zc_pad3 = {<optimized out>, <optimized out>, <optimized out>}, zc_resumable = <optimized out>, zc_pad4 = <optimized out>, zc_sendobj = <optimized out>, zc_fromobj = <optimized out>, zc_createtxg = <optimized out>, zc_stat = { zs_gen = <optimized out>, zs_mode = <optimized out>, zs_links = <optimized out>, zs_ctime = {<optimized out>, <optimized out>}}} local_nv = <optimized out> local_avl = <optimized out> fs = <optimized out> h = <optimized out> buf1 = {<optimized out> <repeats 64 times>} buf2 = {<optimized out> <repeats 64 times>} bytes = <optimized out> delta = <optimized out> #7 zfs_receive_impl (hdl=<optimized out>, tosnap=<optimized out>, originsnap=<optimized out>, flags=<optimized out>, infd=0, sendfs=<optimized out>, stream_nv=0x0, stream_avl=0x0, top_zfs=0x7fffffffe1d8, cleanup_fd=8, action_handlep=0x7fffffffe1d0, finalsnap=0x7fffffffc66f "ver2_12329") at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c:3859 nonpackage_sendfs = '\000' <repeats 255 times> drr = {drr_type = DRR_BEGIN, drr_payloadlen = 0, drr_u = {drr_begin = { drr_magic = 12712594348, drr_versioninfo = 1, drr_creation_time = 1609397741, drr_type = DMU_OST_ZVOL, drr_flags = 4, drr_toguid = 674406883349338168, drr_fromguid = 17010876038533095460, drr_toname = "data/reference@ver2_12323", '\000' <repeats 230 times>}, drr_end = { drr_checksum = {zc_word = {12712594348, 1, 1609397741, 17179869187}}, drr_toguid = 674406883349338168}, drr_object = {drr_object = 12712594348, drr_type = DMU_OT_OBJECT_DIRECTORY, drr_bonustype = DMU_OT_NONE, drr_blksz = 1609397741, drr_bonuslen = 0, drr_checksumtype = 3 '\003', drr_compress = 0 '\000', drr_dn_slots = 0 '\000', drr_pad = "\000\004\000\000", drr_toguid = 674406883349338168}, drr_freeobjects = {drr_firstobj = 12712594348, drr_numobjs = 1, drr_toguid = 1609397741}, drr_write = {drr_object = 12712594348, drr_type = DMU_OT_OBJECT_DIRECTORY, drr_pad = 0, drr_offset = 1609397741, drr_logical_size = 17179869187, drr_toguid = 674406883349338168, drr_checksumtype = 36 '$', drr_checksumflags = 32 ' ', drr_compressiontype = 248 '\370', drr_pad2 = "*\241\305", <incomplete sequence \354>, drr_key = {ddk_cksum = {zc_word = {7378429112340472164, 8520921972713288293, 3617290113340568165, 51}}, ddk_prop = 0}, drr_compressed_size = 0}, drr_free = { drr_object = 12712594348, drr_offset = 1, drr_length = 1609397741, drr_toguid = 17179869187}, drr_write_byref = {drr_object = 12712594348, drr_offset = 1, drr_length = 1609397741, drr_toguid = 17179869187, drr_refguid = 674406883349338168, drr_refobject = 17010876038533095460, drr_refoffset = 7378429112340472164, drr_checksumtype = 101 'e', drr_checksumflags = 114 'r', drr_pad2 = "ence@v", drr_key = {ddk_cksum = {zc_word = {3617290113340568165, 51, 0, 0}}, ddk_prop = 0}}, drr_spill = {drr_object = 12712594348, drr_length = 1, drr_toguid = 1609397741, drr_pad = {17179869187, 674406883349338168, 17010876038533095460, 7378429112340472164}}, drr_write_embedded = {drr_object = 12712594348, drr_offset = 1, drr_length = 1609397741, drr_toguid = 17179869187, drr_compression = 56 '8', drr_etype = 152 '\230', drr_pad = "\035\274|\371[\t", drr_lsize = 720904228, drr_psize = 3960653217}, drr_checksum = {drr_pad = {12712594348, 1, 1609397741, 17179869187, 674406883349338168, 17010876038533095460, 7378429112340472164, 8520921972713288293, 3617290113340568165, 51, 0 <repeats 24 times>}, drr_checksum = {zc_word = {0, 0, 0, 0}}}}} drr_noswap = {drr_type = DRR_BEGIN, drr_payloadlen = 0, drr_u = {drr_begin = { drr_magic = 12712594348, drr_versioninfo = 1, drr_creation_time = 1609397741, drr_type = DMU_OST_ZVOL, drr_flags = 4, drr_toguid = 674406883349338168, drr_fromguid = 17010876038533095460, drr_toname = "data/reference@ver2_12323", '\000' <repeats 230 times>}, drr_end = { drr_checksum = {zc_word = {12712594348, 1, 1609397741, 17179869187}}, drr_toguid = 674406883349338168}, drr_object = {drr_object = 12712594348, drr_type = DMU_OT_OBJECT_DIRECTORY, drr_bonustype = DMU_OT_NONE, drr_blksz = 1609397741, drr_bonuslen = 0, drr_checksumtype = 3 '\003', drr_compress = 0 '\000', drr_dn_slots = 0 '\000', drr_pad = "\000\004\000\000", drr_toguid = 674406883349338168}, drr_freeobjects = {drr_firstobj = 12712594348, drr_numobjs = 1, drr_toguid = 1609397741}, drr_write = {drr_object = 12712594348, drr_type = DMU_OT_OBJECT_DIRECTORY, drr_pad = 0, drr_offset = 1609397741, drr_logical_size = 17179869187, drr_toguid = 674406883349338168, drr_checksumtype = 36 '$', drr_checksumflags = 32 ' ', drr_compressiontype = 248 '\370', drr_pad2 = "*\241\305", <incomplete sequence \354>, drr_key = {ddk_cksum = {zc_word = {7378429112340472164, 8520921972713288293, 3617290113340568165, 51}}, ddk_prop = 0}, drr_compressed_size = 0}, drr_free = { drr_object = 12712594348, drr_offset = 1, drr_length = 1609397741, drr_toguid = 17179869187}, drr_write_byref = {drr_object = 12712594348, drr_offset = 1, drr_length = 1609397741, drr_toguid = 17179869187, drr_refguid = 674406883349338168, drr_refobject = 17010876038533095460, drr_refoffset = 7378429112340472164, drr_checksumtype = 101 'e', drr_checksumflags = 114 'r', drr_pad2 = "ence@v", drr_key = {ddk_cksum = {zc_word = {3617290113340568165, 51, 0, 0}}, ddk_prop = 0}}, drr_spill = {drr_object = 12712594348, drr_length = 1, drr_toguid = 1609397741, drr_pad = {17179869187, 674406883349338168, 17010876038533095460, 7378429112340472164}}, drr_write_embedded = {drr_object = 12712594348, drr_offset = 1, drr_length = 1609397741, drr_toguid = 17179869187, drr_compression = 56 '8', drr_etype = 152 '\230', drr_pad = "\035\274|\371[\t", drr_lsize = 720904228, drr_psize = 3960653217}, drr_checksum = {drr_pad = {12712594348, 1, 1609397741, 17179869187, 674406883349338168, 17010876038533095460, 7378429112340472164, 8520921972713288293, 3617290113340568165, 51, 0 <repeats 24 times>}, drr_checksum = {zc_word = {0, 0, 0, 0}}}}} errbuf = "cannot receive", '\000' <repeats 1009 times> zcksum = {zc_word = {23355066688, 1553595470366, 52790236450666, 1221340679914176}} drrb = <optimized out> err = <optimized out> hdrtype = <optimized out> featureflags = <optimized out> #8 0x00000008002c0112 in zfs_receive_package (hdl=<optimized out>, fd=0, destname=<optimized out>, flags=<optimized out>, drr=<optimized out>, zc=<optimized out>, top_zfs=<optimized out>, cleanup_fd=<optimized out>, action_handlep=<optimized out>) at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c:2957 drre = {drr_type = 3972928, drr_payloadlen = 8, drr_u = {drr_begin = { drr_magic = 34362115045, drr_versioninfo = 34363711296, drr_creation_time = 34362949041, drr_type = 4294954768, drr_flags = 32767, drr_toguid = 34362001379, drr_fromguid = 5, drr_toname = "\332\033\240\063\003\000\000\000\271\065\rO\323\000\000\000\as\261\027\356\033\000\000\217\373\231\352\342\205\002", '\000' <repeats 224 times>}, drr_end = {drr_checksum = { zc_word = {34362115045, 34363711296, 34362949041, 140737488342800}}, drr_toguid = 34362001379}, drr_object = {drr_object = 34362115045, drr_type = 3972928, drr_bonustype = DMU_OT_SPACE_MAP, drr_blksz = 3210673, drr_bonuslen = 8, drr_checksumtype = 16 '\020', drr_compress = 207 '\317', drr_dn_slots = 255 '\377', drr_pad = "\377\377\177\000", drr_toguid = 34362001379}, drr_freeobjects = { drr_firstobj = 34362115045, drr_numobjs = 34363711296, drr_toguid = 34362949041}, drr_write = {drr_object = 34362115045, drr_type = 3972928, drr_pad = 8, drr_offset = 34362949041, drr_logical_size = 140737488342800, drr_toguid = 34362001379, drr_checksumtype = 5 '\005', drr_checksumflags = 0 '\000', drr_compressiontype = 0 '\000', drr_pad2 = "\000\000\000\000", drr_key = {ddk_cksum = { zc_word = {13751032794, 907564365241, 30709413671687, 710159598484367}}, ddk_prop = 0}, drr_compressed_size = 0}, drr_free = {drr_object = 34362115045, drr_offset = 34363711296, drr_length = 34362949041, drr_toguid = 140737488342800}, drr_write_byref = {drr_object = 34362115045, drr_offset = 34363711296, drr_length = 34362949041, drr_toguid = 140737488342800, drr_refguid = 34362001379, drr_refobject = 5, drr_refoffset = 13751032794, drr_checksumtype = 185 '\271', drr_checksumflags = 53 '5', drr_pad2 = "\rO\323\000\000", drr_key = {ddk_cksum = { zc_word = {30709413671687, 710159598484367, 0, 0}}, ddk_prop = 0}}, drr_spill = { drr_object = 34362115045, drr_length = 34363711296, drr_toguid = 34362949041, drr_pad = {140737488342800, 34362001379, 5, 13751032794}}, drr_write_embedded = { drr_object = 34362115045, drr_offset = 34363711296, drr_length = 34362949041, drr_toguid = 140737488342800, drr_compression = 227 '\343', drr_etype = 135 '\207', drr_pad = "\"\000\b\000\000", drr_lsize = 5, drr_psize = 0}, drr_checksum = {drr_pad = { 34362115045, 34363711296, 34362949041, 140737488342800, 34362001379, 5, 13751032794, 907564365241, 30709413671687, 710159598484367, 0 <repeats 24 times>}, drr_checksum = { zc_word = {0, 0, 0, 0}}}}} tofs = "\001\000\000\000\000\000\000\000\000h$\000\b\000\000\000\000\340\200\000\b\000\000\000\001\000\000\000\000\000\000\000\214\341\200\000\b\000\000\000\031\000\000\000\000\000\000\000@\340\377\377\377\177\000\000\\\252{\000\b\000\000\000\260\340\377\377\377\177\000\000щ{\000\b\000\000\000p\340\377\377\377\177\000\000^\022|\000\b\000\000\000\220\354'\000\b", '\000' <repeats 11 times>, "\310\341\377\377\377\177\000\000\000\000\000\000\000\000\000\000\060\341\377\377\377\177\000\000\343\207\"\000\b\000\000\000\001\000\000\000\b\000\000\000\200\321!\000\000\000\000\000\002\000\000\000\000\000\000\000\--Type <RET> for more, q to quit, c to continue without paging-- 060\342\377\377\377\177\000\000t\356\377\377\377\177\000\000\000\000\000\000\000\000\000\000\240\341\377\377\377\177\000\000m"... errbuf = '\000' <repeats 48 times>, "cannot receive", '\000' <repeats 961 times> sendfs = '\000' <repeats 48 times>, "data/reference\000ver2_12329", '\000' <repeats 182 times> stream_nv = <optimized out> fromsnap = <optimized out> stream_avl = <optimized out> softerr = B_FALSE anyerr = B_FALSE sendsnap = 0x7fffffffc66f "ver2_12329" recursive = <optimized out> error = <optimized out> cp = <optimized out> stream_fss = <optimized out> renamed = <optimized out> pair = <optimized out> drrb = <optimized out> i = <optimized out> clp = <optimized out> zhp = <optimized out> #9 zfs_receive_impl (hdl=<optimized out>, tosnap=<optimized out>, originsnap=<optimized out>, flags=<optimized out>, infd=0, sendfs=<optimized out>, stream_nv=0x0, stream_avl=0x0, top_zfs=0x7fffffffe1d8, cleanup_fd=8, action_handlep=0x7fffffffe1d0, finalsnap=0x0) at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c:3865 drr = {drr_type = DRR_BEGIN, drr_payloadlen = 0, drr_u = {drr_begin = {drr_magic = 0, drr_versioninfo = 0, drr_creation_time = 0, drr_type = DMU_OST_NONE, drr_flags = 0, drr_toguid = 0, drr_fromguid = 0, drr_toname = "\254˺\365\002\000\000\000\002", '\000' <repeats 39 times>, "data/reference@ver2_12329", '\000' <repeats 182 times>}, drr_end = {drr_checksum = {zc_word = {0, 0, 0, 0}}, drr_toguid = 0}, drr_object = {drr_object = 0, drr_type = DMU_OT_NONE, drr_bonustype = DMU_OT_NONE, drr_blksz = 0, drr_bonuslen = 0, drr_checksumtype = 0 '\000', drr_compress = 0 '\000', drr_dn_slots = 0 '\000', drr_pad = "\000\000\000\000", drr_toguid = 0}, drr_freeobjects = {drr_firstobj = 0, drr_numobjs = 0, drr_toguid = 0}, drr_write = {drr_object = 0, drr_type = DMU_OT_NONE, drr_pad = 0, drr_offset = 0, drr_logical_size = 0, drr_toguid = 0, drr_checksumtype = 0 '\000', drr_checksumflags = 0 '\000', drr_compressiontype = 0 '\000', drr_pad2 = "\000\000\000\000", drr_key = {ddk_cksum = { zc_word = {12712594348, 2, 0, 0}}, ddk_prop = 0}, drr_compressed_size = 0}, drr_free = {drr_object = 0, drr_offset = 0, drr_length = 0, drr_toguid = 0}, drr_write_byref = {drr_object = 0, drr_offset = 0, drr_length = 0, drr_toguid = 0, drr_refguid = 0, drr_refobject = 0, drr_refoffset = 12712594348, drr_checksumtype = 2 '\002', drr_checksumflags = 0 '\000', drr_pad2 = "\000\000\000\000\000", drr_key = {ddk_cksum = {zc_word = {0, 0, 0, 0}}, ddk_prop = 7378429112340472164}}, drr_spill = {drr_object = 0, drr_length = 0, drr_toguid = 0, drr_pad = {0, 0, 0, 12712594348}}, drr_write_embedded = { drr_object = 0, drr_offset = 0, drr_length = 0, drr_toguid = 0, drr_compression = 0 '\000', drr_etype = 0 '\000', drr_pad = "\000\000\000\000\000", drr_lsize = 0, drr_psize = 0}, drr_checksum = {drr_pad = {0, 0, 0, 0, 0, 0, 12712594348, 2, 0, 0, 0, 0, 7378429112340472164, 8520921972713288293, 3617290113340568165, 57, 0 <repeats 18 times>}, drr_checksum = {zc_word = {0, 0, 0, 0}}}}} drr_noswap = {drr_type = DRR_BEGIN, drr_payloadlen = 0, drr_u = {drr_begin = {drr_magic = 0, drr_versioninfo = 0, drr_creation_time = 0, drr_type = DMU_OST_NONE, drr_flags = 0, drr_toguid = 0, drr_fromguid = 0, drr_toname = "\254˺\365\002\000\000\000\002", '\000' <repeats 39 times>, "data/reference@ver2_12329", '\000' <repeats 182 times>}, drr_end = {drr_checksum = {zc_word = {0, 0, 0, 0}}, drr_toguid = 0}, drr_object = {drr_object = 0, drr_type = DMU_OT_NONE, drr_bonustype = DMU_OT_NONE, drr_blksz = 0, drr_bonuslen = 0, drr_checksumtype = 0 '\000', drr_compress = 0 '\000', drr_dn_slots = 0 '\000', drr_pad = "\000\000\000\000", drr_toguid = 0}, drr_freeobjects = {drr_firstobj = 0, drr_numobjs = 0, drr_toguid = 0}, drr_write = {drr_object = 0, drr_type = DMU_OT_NONE, drr_pad = 0, drr_offset = 0, drr_logical_size = 0, drr_toguid = 0, drr_checksumtype = 0 '\000', drr_checksumflags = 0 '\000', drr_compressiontype = 0 '\000', drr_pad2 = "\000\000\000\000", drr_key = {ddk_cksum = { zc_word = {12712594348, 2, 0, 0}}, ddk_prop = 0}, drr_compressed_size = 0}, drr_free = {drr_object = 0, drr_offset = 0, drr_length = 0, drr_toguid = 0}, drr_write_byref = {drr_object = 0, drr_offset = 0, drr_length = 0, drr_toguid = 0, drr_refguid = 0, drr_refobject = 0, drr_refoffset = 12712594348, drr_checksumtype = 2 '\002', drr_checksumflags = 0 '\000', drr_pad2 = "\000\000\000\000\000", drr_key = {ddk_cksum = {zc_word = {0, 0, 0, 0}}, ddk_prop = 7378429112340472164}}, drr_spill = {drr_object = 0, drr_length = 0, drr_toguid = 0, drr_pad = {0, 0, 0, 12712594348}}, drr_write_embedded = { drr_object = 0, drr_offset = 0, drr_length = 0, drr_toguid = 0, drr_compression = 0 '\000', drr_etype = 0 '\000', drr_pad = "\000\000\000\000\000", drr_lsize = 0, drr_psize = 0}, drr_checksum = {drr_pad = {0, 0, 0, 0, 0, 0, 12712594348, 2, 0, 0, 0, 0, 7378429112340472164, 8520921972713288293, 3617290113340568165, 57, 0 <repeats 18 times>}, drr_checksum = {zc_word = {0, 0, 0, 0}}}}} errbuf = '\000' <repeats 48 times>, "cannot receive", '\000' <repeats 961 times> zcksum = {zc_word = {0, 22, 0, 0}} drrb = <optimized out> err = <optimized out> hdrtype = <optimized out> featureflags = <optimized out> #10 0x00000008002be8d1 in zfs_receive (hdl=0x800843000, tosnap=0x7fffffffee74 "data", props=<optimized out>, flags=0x7fffffffe230, infd=0, stream_avl=0x0) at /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c:3895 top_zfs = 0x0 action_handle = 0 originsnap = 0x0 err = <optimized out> cleanup_fd = 8 #11 0x0000000000212582 in zfs_do_receive (argc=<optimized out>, argv=0x7fffffffec38) at /usr/src/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c:4189 flags = {verbose = B_FALSE, isprefix = B_FALSE, istail = B_FALSE, dryrun = B_FALSE, force = B_FALSE, canmountoff = B_FALSE, resumable = B_FALSE, byteswap = B_FALSE, nomount = B_FALSE} err = 0 props = 0x0 nvp = <optimized out> abort_resumable = B_FALSE c = <optimized out> #12 0x000000000020d505 in main (argc=<optimized out>, argv=0x7fffffffec30) at /usr/src/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c:7561 ret = 0 progname = <optimized out> cmdname = 0x7fffffffee6c "receive" i = <optimized out> (gdb)
Found a workaround: receive -du works (which is weird and redundant, since the source dataset conatins only volumes - nothing to mount).
Follow-up: -d is enough (which is even weirder).
^Triage: mark OBE. I'm sorry that this PR did not get looked at in a timely fashion. However, in the interim, 12.X have gone out of support.