Bug 254530 - ZFS (base) receive fails to receive zvol dataset if dataset already exist
Summary: ZFS (base) receive fails to receive zvol dataset if dataset already exist
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.2-RELEASE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-fs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-03-24 21:38 UTC by Patrik Hildingsson
Modified: 2021-06-16 05:37 UTC (History)
3 users (show)

See Also:


Attachments
patch to libzfs_sendrecv.c (717 bytes, patch)
2021-06-16 05:33 UTC, titus m
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Patrik Hildingsson 2021-03-24 21:38:56 UTC
This error was found while using zxfer(8) to send a zvol dataset from one dataset to another dataset on the same pool. The dataset did not exist on the destination prior to running zxfer.

Running the below chunk replicates the issue, at least on 12.2-RELEASE-p4
zfs create zroot/test
zfs create -o volmode=dev -V 50M zroot/test/disk0
dd if=/dev/random of=/dev/zvol/zroot/test/disk0
zfs snapshot zroot/test/disk0@test-now_ok
dd if=/dev/random of=/dev/zvol/zroot/test/disk0
zfs snapshot zroot/test/disk0@test-now_ok-2
dd if=/dev/random of=/dev/zvol/zroot/test/disk0
zfs create -V 52428800 -o 'reservation=none' -o 'volblocksize=8192' -o 'checksum=on' -o 'compression=lz4' -o 'readonly=off' -o 'copies=1' -o 'logbias=throughput' -o 'dedup=off' -o 'sync=standard' -o 'volmode=dev' -o 'snapshot_limit=none' -o 'redundant_metadata=most' zroot/test/disk1
zfs send zroot/test/disk0@test-now_ok | zfs receive -F zroot/test/disk1


At your convenience I have copy/pasted the above commands and the resulting error:
# zfs create zroot/test
# zfs create -o volmode=dev -V 50M zroot/test/disk0
# dd if=/dev/random of=/dev/zvol/zroot/test/disk0
dd: /dev/zvol/zroot/test/disk0: end of device
102401+0 records in
102400+0 records out
52428800 bytes transferred in 0.552508 secs (94892325 bytes/sec)
# zfs snapshot zroot/test/disk0@test-now_ok
# dd if=/dev/random of=/dev/zvol/zroot/test/disk0
dd: /dev/zvol/zroot/test/disk0: end of device
102401+0 records in
102400+0 records out
52428800 bytes transferred in 0.507651 secs (103277323 bytes/sec)
# zfs snapshot zroot/test/disk0@test-now_ok-2
# dd if=/dev/random of=/dev/zvol/zroot/test/disk0
dd: /dev/zvol/zroot/test/disk0: end of device
102401+0 records in
102400+0 records out
52428800 bytes transferred in 0.515789 secs (101647735 bytes/sec)
# zfs create -V 52428800 -o 'reservation=none' -o 'volblocksize=8192' -o 'checksum=on' -o 'compression=lz4' -o 'readonly=off' -o 'copies=1' -o 'logbias=throughput' -o 'dedup=off' -o 'sync=standard' -o 'volmode=dev' -o 'snapshot_limit=none' -o 'redundant_metadata=most' zroot/test/disk1
root@lrrr:~ # zfs send zroot/test/disk0@test-now_ok | zfs receive -F zroot/test/disk1
cannot open 'zroot/test/disk1/': trailing slash in name
warning: cannot send 'zroot/test/disk0@test-now_ok': signal received
 
 
# zfs send zroot/test/disk0@test-now_ok | zstreamdump
BEGIN record
        hdrtype = 1
        features = 0
        magic = 2f5bacbac
        creation_time = 605ba746
        type = 3
        flags = 0x4
        toguid = f1d16456c4cc9f82
        fromguid = 0
        toname = zroot/test/disk0@test-now_ok
END checksum = 64c114cf9127fc/5e00770266493cb3/7fb09ad1733d83d9/9fc28d91b9886732
SUMMARY:
        Total DRR_BEGIN records = 1
        Total DRR_END records = 1
        Total DRR_OBJECT records = 2
        Total DRR_FREEOBJECTS records = 4
        Total DRR_WRITE records = 6401
        Total DRR_WRITE_BYREF records = 0
        Total DRR_WRITE_EMBEDDED records = 0
        Total DRR_FREE records = 4
        Total DRR_SPILL records = 0
        Total records = 6413
        Total write size = 52429312 (0x3200200)
        Total stream length = 54430168 (0x33e89d8)

However the error never appears if zfs receive is run without in advance creating the dataset on the destination, as can be seen below:

# zfs create zroot/test
# zfs create -o volmode=dev -V 50M zroot/test/disk0
# dd if=/dev/random of=/dev/zvol/zroot/test/disk0
dd: /dev/zvol/zroot/test/disk0: end of device
102401+0 records in
102400+0 records out
52428800 bytes transferred in 1.084470 secs (48345071 bytes/sec)
# zfs snapshot zroot/test/disk0@test-now_ok
# dd if=/dev/random of=/dev/zvol/zroot/test/disk0
dd: /dev/zvol/zroot/test/disk0: end of device
102401+0 records in
102400+0 records out
52428800 bytes transferred in 0.627445 secs (83559234 bytes/sec)
# zfs snapshot zroot/test/disk0@test-now_ok-2
# dd if=/dev/random of=/dev/zvol/zroot/test/disk0
dd: /dev/zvol/zroot/test/disk0: end of device
102401+0 records in
102400+0 records out
52428800 bytes transferred in 0.512001 secs (102399789 bytes/sec)
# zfs send zroot/test/disk0@test-now_ok | zfs receive -F zroot/test/disk1
# echo $?
0
Comment 1 Alan Somers freebsd_committer 2021-03-25 00:36:47 UTC
Three questions:

1) Have you tested this on 14.0-CURRENT or on one of the 13.0 pre releases?  The ZFS codebase is substantially different in FreBSD 13+ .

2) Have you tested this on stable/12 after r368491?  That revision fixed a related bug for file systems.  I never tested it for zvols.

3) The error you pasted looks suspicious.  Are you sure you copy/pasted it correctly?  There is no trailing slash in the name as you pasted it.

root@lrrr:~ # zfs send zroot/test/disk0@test-now_ok | zfs receive -F zroot/test/disk1
cannot open 'zroot/test/disk1/': trailing slash in name
Comment 2 Patrik Hildingsson 2021-03-25 10:08:07 UTC
(In reply to Alan Somers from comment #1)
1) Please see the below copy/paste from a 13.0-RC3 system where it seems to work just fine.

root@testbsd:~ # sysctl -w vfs.zfs.vol.mode=2
root@testbsd:~ # zfs create zroot/test
root@testbsd:~ # zfs create -o volmode=dev -V 50M zroot/test/disk0
root@testbsd:~ # dd if=/dev/random of=/dev/zvol/zroot/test/disk0
dd: /dev/zvol/zroot/test/disk0: end of device
102401+0 records in
102400+0 records out
52428800 bytes transferred in 1.016171 secs (51594487 bytes/sec)
root@testbsd:~ # zfs snapshot zroot/test/disk0@test-now_ok
root@testbsd:~ # dd if=/dev/random of=/dev/zvol/zroot/test/disk0
dd: /dev/zvol/zroot/test/disk0: end of device
102401+0 records in
102400+0 records out
52428800 bytes transferred in 0.833823 secs (62877584 bytes/sec)
root@testbsd:~ # zfs snapshot zroot/test/disk0@test-now_ok-2
root@testbsd:~ # dd if=/dev/random of=/dev/zvol/zroot/test/disk0
dd: /dev/zvol/zroot/test/disk0: end of device
102401+0 records in
102400+0 records out
52428800 bytes transferred in 0.444901 secs (117843859 bytes/sec)
root@testbsd:~ # zfs create -V 52428800 -o 'reservation=none' -o 'volblocksize=8192' -o 'checksum=on' -o 'compression=lz4' -o 'readonly=off' -o 'copies=1' -o 'logbias=throughput' -o 'dedup=off' -o 'sync=standard' -o 'volmode=dev' -o 'snapshot_limit=none' -o 'redundant_metadata=most' zroot/test/disk1
root@testbsd:~ # zfs send zroot/test/disk0@test-now_ok | zfs receive -F zroot/test/disk1
root@testbsd:~ # uname -K
1300139
root@testbsd:~ # uname -U
1300139

2) No, I did not try 12-STABLE. I have only tried 12.2-RELEASE-p4. I have reproduced the error on two different installations of 12.2-RELEASE-p4.

3) The paste indeed looks suspicious. I forgot to trim away the prepending "root@lrrr:~ ", other than that it is correct. I did not leave out a trailing forward slash. It isn't supposed to be there. ZFS incorrectly prompts that there is a trailing forward slash.
Comment 3 Alan Somers freebsd_committer 2021-03-25 13:31:56 UTC
Please do try stable/12.  I think r368491 most likely fixed this problem.
Comment 4 Patrik Hildingsson 2021-03-25 14:43:43 UTC
(In reply to Alan Somers from comment #3)
The error occurs even in FreeBSD 12.2-STABLE r369477

root@testbsd:~ # zfs create zroot/test
root@testbsd:~ # zfs create -o volmode=dev -V 50M zroot/test/disk0
root@testbsd:~ # dd if=/dev/random of=/dev/zvol/zroot/test/disk0
dd: /dev/zvol/zroot/test/disk0: end of device
102401+0 records in
102400+0 records out
52428800 bytes transferred in 1.582797 secs (33124142 bytes/sec)
root@testbsd:~ # zfs snapshot zroot/test/disk0@test-now_ok
root@testbsd:~ # dd if=/dev/random of=/dev/zvol/zroot/test/disk0
dd: /dev/zvol/zroot/test/disk0: end of device
102401+0 records in
102400+0 records out
52428800 bytes transferred in 0.926790 secs (56570297 bytes/sec)
root@testbsd:~ # zfs snapshot zroot/test/disk0@test-now_ok-2
root@testbsd:~ # dd if=/dev/random of=/dev/zvol/zroot/test/disk0
dd: /dev/zvol/zroot/test/disk0: end of device
102401+0 records in
102400+0 records out
52428800 bytes transferred in 1.235393 secs (42438953 bytes/sec)
root@testbsd:~ # zfs create -V 52428800 -o 'reservation=none' -o 'volblocksize=8192' -o 'checksum=on' -o 'compression=lz4' -o 'readonly=off' -o 'copies=1' -o 'logbias=throughput' -o 'dedup=off' -o 'sync=standard' -o 'volmode=dev' -o 'snap
shot_limit=none' -o 'redundant_metadata=most' zroot/test/disk1
root@testbsd:~ # zfs send zroot/test/disk0@test-now_ok | zfs receive -F zroot/test/disk1
cannot open 'zroot/test/disk1/': trailing slash in name
warning: cannot send 'zroot/test/disk0@test-now_ok': signal received
root@testbsd:~ # uname -K
1202505
root@testbsd:~ # uname -U
1202505
root@testbsd:~ # freebsd-version
12.2-STABLE
Comment 5 Alan Somers freebsd_committer 2021-03-25 16:42:14 UTC
Well darn.  Are you interested to try debugging this yourself, Patrick?  I'm guessing that the problem lies in libzfs, not in the kernel.  I would focus on that "trailing slash in name" message.  Try to figure out where that comes from.
Comment 6 Patrik Hildingsson 2021-03-25 16:54:29 UTC
(In reply to Alan Somers from comment #5)
To be honest, not really. I plan to move to 13.0-RELEASE as soon as you release it.
I was just trying to put the issue in the spotlight, in case someone else runs into the same issue in the future.

Do you think it's worth the effort to debug and find the issue?
Comment 7 Alan Somers freebsd_committer 2021-03-25 17:46:52 UTC
It isn't worth the effort for me, but then I'm not the one affected.  I or another developer would certainly review a patch if you produce one.
Comment 8 Christopher Pallone 2021-03-30 20:15:40 UTC
I just ran into this same bug on my 12.2 system...
Not that it is really any different, but I tried to receive the data-stream from a local file with the same error. 

# freebsd-version
12.2-RELEASE-p1
# zfs snap zdata/vol_fbsd12-02-root@csp
# zfs send zdata/vol_fbsd12-02-root@csp > /tmp/csp.out
# zfs receive -F 'zdata/vol_smtp02-root' < /tmp/csp.out
cannot open 'zdata/vol_smtp02-root/': trailing slash in name

I have some automation scripts build around this. Depending on the timing of 13, I might try to hunt this down myself. Otherwise, I will have to devote my time to testing my stuff on the 13.0-RELEASE.
Comment 9 titus m 2021-06-16 05:33:43 UTC
Created attachment 225842 [details]
patch to libzfs_sendrecv.c
Comment 10 titus m 2021-06-16 05:37:38 UTC
the trailing slash is added in 
/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c in 
zfs_ioc_dataset_list_next while searching for child datasets

then in /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c
in zfs_receive_one test for destination children will alter dataset source name

if (is_volume &&
                            ioctl(hdl->libzfs_fd, ZFS_IOC_DATASET_LIST_NEXT,
                            &zc) == 0) {