PORT: net/rsync version 3.1.2 protocol version 31 O/S: FreeBSD wjdb1 10.1-STABLE FreeBSD 10.1-STABLE #0 r278180: Wed Feb 4 01:20:35 EST 2015 root@wjdb1:/usr/obj/usr/src/sys/WJDB1 amd64 COMMAND: rsync -aHXSE --partial --timeout=60 --fileflags --force-change --numeric-ids --force --super -8 /usr/bin/ /root/flags RESULT: rsync: link "/root/flags/ypchpass" => ypchsh failed: Operation not permitted (1) rsync: link "/root/flags/ypchfn" => ypchsh failed: Operation not permitted (1) rsync: link "/root/flags/chsh" => ypchsh failed: Operation not permitted (1) rsync: link "/root/flags/chpass" => ypchsh failed: Operation not permitted (1) rsync: link "/root/flags/chfn" => ypchsh failed: Operation not permitted (1) rsync: link "/root/flags/passwd" => yppasswd failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1200) [sender=3.1.2] This happens both for attempted creation as well as for updates - it also happens with rsync versions 3.1.1 & 3.0.8 The affected hard links are not created from these source files: -r-sr-xr-x 6 root wheel schg 22616 Feb 4 2015 /usr/bin/chfn* -r-sr-xr-x 6 root wheel schg 22616 Feb 4 2015 /usr/bin/chpass* -r-sr-xr-x 6 root wheel schg 22616 Feb 4 2015 /usr/bin/chsh* -r-sr-xr-x 2 root wheel schg 8384 Feb 4 2015 /usr/bin/passwd* -r-sr-xr-x 6 root wheel schg 22616 Feb 4 2015 /usr/bin/ypchfn* -r-sr-xr-x 6 root wheel schg 22616 Feb 4 2015 /usr/bin/ypchpass* ... although these source files are created: -r-sr-xr-x 6 root wheel schg 22616 Feb 4 2015 /usr/bin/ypchsh* -r-sr-xr-x 2 root wheel schg 8384 Feb 4 2015 /usr/bin/yppasswd* Apparently rsync is unable to correctly deal with a combination of hard links and immutable file flags. After fixing these manually, further updates no longer produce the errors.
I was able to reproduce it on 10.3-RELEASE-p11 but not on 11.0-STABLE. I tried a combination of 10.x/11.x/ufs/zfs and on 10.x it always produced the error you're describing but never on 11.x.
I'm not so sure what you're trying to say - it fails to work in that way on Freebsd 6.x through 10.x as far as I have seen (plus it will be a while before we'll be able to vet 11.x for production ... although it could be useful to know why it works differently there) Hopefully it can be fixed fairly soon so we don't have to fix such affected files manually any more :( One thing to add a slight bit of clarity: the bug only seems to happen at link creation time - once the links are created, rsync is ok with them ... however I have not yet tested the scenario where the file and its links have changed at the source, thus probably requiring the destination file to be recreated (which should most likely re-manifest the bug)
I'm saying that it's not causing these errors on FreeBSD 11.x. Unless a patch is submitted or reported upstream (https://bugzilla.samba.org/) it's not likely to get fixed for 10.x.
Is the rsync version relative to 11.x different from the one for 10.x (and/or has different patches)? I ask because I'm trying to understand why the two o/s's work differently in this regard -there's not much that is immediately obvious about why that should occur (in fact it's a bit worrisome, altho I might need to dig deeper in order to understand this possibility better) Also I imagine that the upstream people will probably not accept FreeBSD-based patches/fixes from anyone except "official" FreeBSD sources. Some google work suggests that this problem has cropped up before, perhaps more than once, and supposedly it was fixed - however the actual results as reported show that there has either been a regression, or that the "fix" was only partial. I'll try to find the time to come up with a workable patch myself, sigh ... stay tuned ...
I whipped up a 11.0-RELEASE virtualbox instance to see what the differences are - it appears that all those hard links have been turned into soft links, thus eliding the example problem :( I then made a directory "flags2" with these files: 19813 -rw-r--r-- 5 root wheel schg,uarch 0 Jan 3 03:58 x0 19813 -rw-r--r-- 5 root wheel schg,uarch 0 Jan 3 03:58 x1 19813 -rw-r--r-- 5 root wheel schg,uarch 0 Jan 3 03:58 x2 19813 -rw-r--r-- 5 root wheel schg,uarch 0 Jan 3 03:58 x3 19813 -rw-r--r-- 5 root wheel schg,uarch 0 Jan 3 03:58 x4 ... and ran this command: rsync -aHXSE --partial --timeout=60 --fileflags --force-change --numeric-ids --f orce --super -8 /root/flags2/ /root/flags3 Result: rsync: link "/root/flags3/x3" => x4 failed: Operation not permitted (1) rsync: link "/root/flags3/x2" => x4 failed: Operation not permitted (1) rsync: link "/root/flags3/x1" => x4 failed: Operation not permitted (1) rsync: link "/root/flags3/x0" => x4 failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1200) [sender=3.1.2] Contents of "flag3": 19815 -rw-r--r-- 1 root wheel schg,uarch 0 Jan 3 03:58 x4 ... in other words, identical behaviour to that of FreeBSD 6.x to 10.x
On 11.0-RELEASE after a portsnap & portupgrade to get the most current version of all installed ports, I updated the linked source files in "flags2" by removing the "schg" flag, echoing "!" to flags2/x0, and re-added the "schg" flag - then I removed the "schg" flag from flags3/x4, hard-linked x0, x1, x2, x3 to x4, and re-added the "schg" flag. "rsync -aHXSE --partial --timeout=60 --fileflags --force-change --numeric-ids --force --super -8 /root/flags2/ /root/flags3" produced these results: flags2: 19813 -rw-r--r-- 5 root wheel schg,uarch 2 Jan 3 23:58 x0 19813 -rw-r--r-- 5 root wheel schg,uarch 2 Jan 3 23:58 x1 19813 -rw-r--r-- 5 root wheel schg,uarch 2 Jan 3 23:58 x2 19813 -rw-r--r-- 5 root wheel schg,uarch 2 Jan 3 23:58 x3 19813 -rw-r--r-- 5 root wheel schg,uarch 2 Jan 3 23:58 x4 flags3: 36189 -rw-r--r-- 1 root wheel schg,uarch 2 Jan 3 23:58 x0 19815 -rw-r--r-- 4 root wheel uarch 0 Jan 3 03:58 x1 19815 -rw-r--r-- 4 root wheel uarch 0 Jan 3 03:58 x2 19815 -rw-r--r-- 4 root wheel uarch 0 Jan 3 03:58 x3 19815 -rw-r--r-- 4 root wheel uarch 0 Jan 3 03:58 x4 ... which confirmed my belief that when a file and its links have changed at the source, it would re-manifest the bug as mentioned above This shows that all versions of FreeBSD from 6.x (or less) to at least 11.x have this problem in that they all produce the same flawed behaviour with respect to copy of immutable files with hard links ...
rsync error output for above was: rsync: link "/root/flags3/.x1.22723" => x0 failed: Operation not permitted (1) rsync: link "/root/flags3/.x2.22723" => x0 failed: Operation not permitted (1) rsync: link "/root/flags3/.x3.22723" => x0 failed: Operation not permitted (1) rsync: link "/root/flags3/.x4.22723" => x0 failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1200) [sender=3.1.2]
I upgraded to FreeBSD 11.0-STABLE r311209 ... same failed results as FreeBSD 6.x to 10.x - what can be done to fix this severe bug? I tried to understand how to make a patch - however the code looks far too gnarly for a non-rsync-developer to mess with - I'm too afraid of breaking things :(
After thinking this through some more I think I should close this bug report and open a new one based on FreeBSD 11.0 using the information in comments 5-8
As I've mentioned earlier I really think this should be reported and tracked upstream (see link in previous post). That way future compatibility is guaranteed.
I don't have any official standing as a FreeBSD ports maintainer - it's been my general experience in the past that such O/S-specific fixes are usually referred back to the O/S maintainers, or outright refused Would you be able to forward this bug report to the rsync people so they know to take it seriously? Thanks - BB
I've created: https://bugzilla.samba.org/show_bug.cgi?id=12508
Excellent - many thanks! ... also a nice job of reducing it to its essentials :)
Any news here?
Back to the pool, I no longer maintain this port.
Hi, I'm the new rsync maintainer, the ticket open by ehaupt@ is under processing by the rsync team, maybe this need further investigations to provide them a patch...
The port work as expected for 11 and the 10 EOL is next month
the problem is independent of FreeBSD O/S version & is still not fixed in FreeBSD 11.x some hard links were changed to soft links which is not in any way a fix for the problem of copying hard linked files which are immutable via chflags
it's also still true for current version of rsync 3.1.3 tested on FreeBSD 11.2-STABLE - FreeBSD droplet02 11.2-STABLE FreeBSD 11.2-STABLE #1 r336657: Tue Jul 24 06:09:19 EDT 2018 root@droplet02:/var/misc/usr/obj/usr/src/sys/DROPLET02 amd64 https://bugzilla.samba.org/show_bug.cgi?id=12508