Sending and receiving ZFS snapshot on the same machine with latest main (3baefc8e7bd5), using: $ sudo zfs send -vLeR pool/dataset.old@replica | dd obs=256m | dd obs=256m | sudo zfs receive -Fv pool/dataset which basically tries to recompress an existing tree of datasets. ==== Unread portion of the kernel message buffer: panic: Assertion lock == sq->sq_lock failed at /usr/src/sys/kern/subr_sleepqueue.c:371 cpuid = 5 time = 1614415361 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe02bd63ff00 vpanic() at vpanic+0x181/frame 0xfffffe02bd63ff50 panic() at panic+0x43/frame 0xfffffe02bd63ffb0 sleepq_add() at sleepq_add+0x3e3/frame 0xfffffe02bd640000 _sleep() at _sleep+0x20e/frame 0xfffffe02bd6400b0 taskqueue_drain() at taskqueue_drain+0xfb/frame 0xfffffe02bd6400f0 taskq_wait_id() at taskq_wait_id+0x2a/frame 0xfffffe02bd640110 spa_taskq_dispatch_sync() at spa_taskq_dispatch_sync+0x89/frame 0xfffffe02bd640160 dump_bytes() at dump_bytes+0x35/frame 0xfffffe02bd640190 dump_record() at dump_record+0x121/frame 0xfffffe02bd6401d0 dmu_dump_write() at dmu_dump_write+0x2fc/frame 0xfffffe02bd640220 do_dump() at do_dump+0x9e0/frame 0xfffffe02bd6403b0 dmu_send_impl() at dmu_send_impl+0x115b/frame 0xfffffe02bd640550 dmu_send_obj() at dmu_send_obj+0x29d/frame 0xfffffe02bd640790 zfs_ioc_send() at zfs_ioc_send+0x1f9/frame 0xfffffe02bd640830 zfsdev_ioctl_common() at zfsdev_ioctl_common+0x4df/frame 0xfffffe02bd6408f0 zfsdev_ioctl() at zfsdev_ioctl+0x146/frame 0xfffffe02bd640920 devfs_ioctl() at devfs_ioctl+0xcc/frame 0xfffffe02bd640970 VOP_IOCTL_APV() at VOP_IOCTL_APV+0x4e/frame 0xfffffe02bd640990 vn_ioctl() at vn_ioctl+0x131/frame 0xfffffe02bd640aa0 devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe02bd640ac0 kern_ioctl() at kern_ioctl+0x289/frame 0xfffffe02bd640b30 sys_ioctl() at sys_ioctl+0x12a/frame 0xfffffe02bd640c00 amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe02bd640d30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe02bd640d30 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8015da7ea, rsp = 0x7fffffffafa8, rbp = 0x7fffffffb010 --- Uptime: 6h33m18s (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=textdump@entry=1) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xffffffff8068ef0f in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486 #3 0xffffffff8068f370 in vpanic (fmt=<optimized out>, ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:919 #4 0xffffffff8068f0c3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:843 #5 0xffffffff806eaa03 in sleepq_add (wchan=wchan@entry=0xfffff80167cfc3c0, lock=lock@entry=0xfffff800b3f2f240, wmesg=wmesg@entry=0xffffffff80b0e1df "tq_drain", flags=flags@entry=0, queue=queue@entry=0) at /usr/src/sys/kern/subr_sleepqueue.c:371 #6 0xffffffff8069b18e in _sleep (ident=ident@entry=0xfffff80167cfc3c0, lock=<optimized out>, lock@entry=0xfffff800b3f2f240, priority=priority@entry=0, wmesg=<unavailable>, sbt=sbt@entry=0, pr=pr@entry=0, flags=256) at /usr/src/sys/kern/kern_synch.c:205 #7 0xffffffff806f06fb in TQ_SLEEP (tq=0xfffff800b3f2f200, p=0xfffff80167cfc3c0, wm=<optimized out>) at /usr/src/sys/kern/subr_taskqueue.c:125 #8 taskqueue_drain (queue=0xfffff800b3f2f200, task=0xfffff80167cfc3c0) at /usr/src/sys/kern/subr_taskqueue.c:583 #9 0xffffffff814723fa in taskq_wait_id (tq=0xfffff80005898780, tid=<optimized out>) at /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_taskq.c:430 #10 0xffffffff81532ff9 in spa_taskq_dispatch_sync (spa=<optimized out>, t=<optimized out>, t@entry=ZIO_TYPE_FREE, q=<optimized out>, q@entry=ZIO_TASKQ_ISSUE, func=0xffffffff815f9990 <dump_bytes_cb>, arg=arg@entry=0xfffffe02bd640178, flags=flags@entry=0) at /usr/src/sys/contrib/openzfs/module/zfs/spa.c:1105 #11 0xffffffff815f9985 in dump_bytes (os=<optimized out>, buf=<optimized out>, len=<optimized out>, arg=<optimized out>) at /usr/src/sys/contrib/openzfs/module/zfs/zfs_ioctl.c:5354 #12 0xffffffff814de5c1 in dump_record (dscp=0xfffffe02bd6403c0, payload=0xfffffe02bb771000, payload_len=payload_len@entry=131072) at /usr/src/sys/contrib/openzfs/module/zfs/dmu_send.c:308 #13 0xffffffff814e0f1c in dmu_dump_write (dscp=<optimized out>, type=<optimized out>, object=<optimized out>, offset=<optimized out>, offset@entry=320733184, lsize=<optimized out>, psize=131072, bp=0xfffff801854e5650, data=0xfffffe02bb771000) at /usr/src/sys/contrib/openzfs/module/zfs/dmu_send.c:548
I'm not able to repro this. Is it still occurring?
I just experienced the same panic, though with a different stack trace. In my case I was using a build based on stable/13 from September-2021 panic: Assertion lock == sq->sq_lock failed at /usr/home/asomers/src/github/Axcient/freebsd-src/sys/kern/subr_sleepqueue.c:371 cpuid = 21 time = 1651466741 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x47/frame 0xfffffe0a7468ad20 vpanic() at vpanic+0x1cc/frame 0xfffffe0a7468ad80 panic() at panic+0x43/frame 0xfffffe0a7468ade0 sleepq_add() at sleepq_add+0x4ae/frame 0xfffffe0a7468ae40 _sleep() at _sleep+0x2b0/frame 0xfffffe0a7468aee0 taskqueue_quiesce() at taskqueue_quiesce+0x1f3/frame 0xfffffe0a7468af50 dmu_objset_sync() at dmu_objset_sync+0x57a/frame 0xfffffe0a7468b030 dsl_dataset_sync() at dsl_dataset_sync+0x218/frame 0xfffffe0a7468b080 dsl_pool_sync() at dsl_pool_sync+0x17f/frame 0xfffffe0a7468b110 spa_sync() at spa_sync+0x15ee/frame 0xfffffe0a7468b350 txg_sync_thread() at txg_sync_thread+0x346/frame 0xfffffe0a7468b430 fork_exit() at fork_exit+0xb3/frame 0xfffffe0a7468b470 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0a7468b470 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic
(In reply to Alan Somers from comment #2) Were you able to get a kernel dump? It would useful to see the contents of *sq->sq_lock from the sleepq_add() frame.
(In reply to Mark Johnston from comment #3) I only got a text dump, unfortunately. I'm going to try again, this time with full core dumps enabled.