Bug 215688 - net/rsync fileflags & forcechange don't work for hardlinks
Summary: net/rsync fileflags & forcechange don't work for hardlinks
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Rodrigo Osorio
URL:
Keywords: needs-patch
Depends on:
Blocks:
 
Reported: 2016-12-31 22:44 UTC by hostmaster
Modified: 2018-09-22 23:03 UTC (History)
3 users (show)

See Also:
hostmaster: maintainer-feedback+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description hostmaster 2016-12-31 22:44:28 UTC
PORT:    net/rsync version 3.1.2  protocol version 31

O/S:     FreeBSD wjdb1 10.1-STABLE FreeBSD 10.1-STABLE #0 r278180: Wed Feb  4 01:20:35 EST 2015     root@wjdb1:/usr/obj/usr/src/sys/WJDB1  amd64

COMMAND: rsync -aHXSE --partial --timeout=60 --fileflags --force-change --numeric-ids --force --super -8 /usr/bin/ /root/flags

RESULT:
rsync: link "/root/flags/ypchpass" => ypchsh failed: Operation not permitted (1)
rsync: link "/root/flags/ypchfn" => ypchsh failed: Operation not permitted (1)
rsync: link "/root/flags/chsh" => ypchsh failed: Operation not permitted (1)
rsync: link "/root/flags/chpass" => ypchsh failed: Operation not permitted (1)
rsync: link "/root/flags/chfn" => ypchsh failed: Operation not permitted (1)
rsync: link "/root/flags/passwd" => yppasswd failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1200) [sender=3.1.2]

This happens both for attempted creation as well as for updates -
it also happens with rsync versions 3.1.1 & 3.0.8

The affected hard links are not created from these source files:
-r-sr-xr-x   6 root  wheel   schg    22616 Feb  4  2015 /usr/bin/chfn*
-r-sr-xr-x   6 root  wheel   schg    22616 Feb  4  2015 /usr/bin/chpass*
-r-sr-xr-x   6 root  wheel   schg    22616 Feb  4  2015 /usr/bin/chsh*
-r-sr-xr-x   2 root  wheel   schg     8384 Feb  4  2015 /usr/bin/passwd*
-r-sr-xr-x   6 root  wheel   schg    22616 Feb  4  2015 /usr/bin/ypchfn*
-r-sr-xr-x   6 root  wheel   schg    22616 Feb  4  2015 /usr/bin/ypchpass*

... although these source files are created:
-r-sr-xr-x   6 root  wheel   schg    22616 Feb  4  2015 /usr/bin/ypchsh*
-r-sr-xr-x   2 root  wheel   schg     8384 Feb  4  2015 /usr/bin/yppasswd*

Apparently rsync is unable to correctly deal with a combination of hard links and immutable file flags.

After fixing these manually, further updates no longer produce the errors.
Comment 1 Emanuel Haupt freebsd_committer 2017-01-01 12:05:39 UTC
I was able to reproduce it on 10.3-RELEASE-p11 but not on 11.0-STABLE. I tried a combination of 10.x/11.x/ufs/zfs and on 10.x it always produced the error you're describing but never on 11.x.
Comment 2 hostmaster 2017-01-03 03:51:42 UTC
I'm not so sure what you're trying to say - it fails to
work in that way on Freebsd 6.x through 10.x as far as I
have seen (plus it will be a while before we'll be able
to vet 11.x for production ... although it could be
useful to know why it works differently there)

Hopefully it can be fixed fairly soon so we don't have
to fix such affected files manually any more :(

One thing to add a slight bit of clarity:
the bug only seems to happen at link creation time -
once the links are created, rsync is ok with them ...
however I have not yet tested the scenario where the file
and its links have changed at the source, thus probably
requiring the destination file to be recreated (which 
should most likely re-manifest the bug)
Comment 3 Emanuel Haupt freebsd_committer 2017-01-03 10:30:29 UTC
I'm saying that it's not causing these errors on FreeBSD 11.x.

Unless a patch is submitted or reported upstream (https://bugzilla.samba.org/) it's not likely to get fixed for 10.x.
Comment 4 hostmaster 2017-01-03 11:31:16 UTC

Is the rsync version relative to 11.x different from the one for 10.x (and/or has different patches)? 
I ask because I'm trying to understand why the two o/s's work differently in this regard -there's not much that is immediately obvious about why that should occur (in fact it's a bit worrisome, altho I might need to dig deeper in order to understand this possibility better)

Also I imagine that the upstream people will probably not accept FreeBSD-based patches/fixes from anyone except "official" FreeBSD sources.

Some google work suggests that this problem has cropped up before, perhaps more than once, and supposedly it was fixed - however the actual results as reported show that there has either been a regression, or that the "fix" was only partial.

I'll try to find the time to come up with a workable patch myself, sigh
... stay tuned ...
Comment 5 hostmaster 2017-01-03 14:14:08 UTC
I whipped up a 11.0-RELEASE virtualbox instance to see what the differences are -
it appears that all those hard links have been turned into soft links, thus eliding the example problem :(

I then made a directory "flags2" with these files:
19813 -rw-r--r--  5 root  wheel  schg,uarch  0 Jan  3 03:58 x0
19813 -rw-r--r--  5 root  wheel  schg,uarch  0 Jan  3 03:58 x1
19813 -rw-r--r--  5 root  wheel  schg,uarch  0 Jan  3 03:58 x2
19813 -rw-r--r--  5 root  wheel  schg,uarch  0 Jan  3 03:58 x3
19813 -rw-r--r--  5 root  wheel  schg,uarch  0 Jan  3 03:58 x4

... and ran this command:
rsync -aHXSE --partial --timeout=60 --fileflags --force-change --numeric-ids --f
orce --super -8 /root/flags2/ /root/flags3

Result:
rsync: link "/root/flags3/x3" => x4 failed: Operation not permitted (1)
rsync: link "/root/flags3/x2" => x4 failed: Operation not permitted (1)
rsync: link "/root/flags3/x1" => x4 failed: Operation not permitted (1)
rsync: link "/root/flags3/x0" => x4 failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1200) [sender=3.1.2]

Contents of "flag3":
19815 -rw-r--r--  1 root  wheel  schg,uarch  0 Jan  3 03:58 x4

... in other words, identical behaviour to that of FreeBSD 6.x to 10.x
Comment 6 hostmaster 2017-01-04 05:19:39 UTC
On 11.0-RELEASE after a portsnap & portupgrade to get the most current version of all installed ports, I updated the linked source files in "flags2" by removing the "schg" flag, echoing "!" to flags2/x0, and re-added the "schg" flag -
then I removed the "schg" flag from flags3/x4, hard-linked x0, x1, x2, x3 to x4, and re-added the "schg" flag.

"rsync -aHXSE --partial --timeout=60 --fileflags --force-change --numeric-ids --force --super -8 /root/flags2/ /root/flags3"
produced these results:

flags2:
19813 -rw-r--r--  5 root  wheel  schg,uarch  2 Jan  3 23:58 x0
19813 -rw-r--r--  5 root  wheel  schg,uarch  2 Jan  3 23:58 x1
19813 -rw-r--r--  5 root  wheel  schg,uarch  2 Jan  3 23:58 x2
19813 -rw-r--r--  5 root  wheel  schg,uarch  2 Jan  3 23:58 x3
19813 -rw-r--r--  5 root  wheel  schg,uarch  2 Jan  3 23:58 x4

flags3:
36189 -rw-r--r--  1 root  wheel  schg,uarch  2 Jan  3 23:58 x0
19815 -rw-r--r--  4 root  wheel  uarch       0 Jan  3 03:58 x1
19815 -rw-r--r--  4 root  wheel  uarch       0 Jan  3 03:58 x2
19815 -rw-r--r--  4 root  wheel  uarch       0 Jan  3 03:58 x3
19815 -rw-r--r--  4 root  wheel  uarch       0 Jan  3 03:58 x4

... which confirmed my belief that when a file and its links have changed at the source, it would re-manifest the bug as mentioned above

This shows that all versions of FreeBSD from 6.x (or less) to at least 11.x have this problem in that they all produce the same flawed behaviour with respect to copy of immutable files with hard links ...
Comment 7 hostmaster 2017-01-04 05:29:56 UTC
rsync error output for above was:
rsync: link "/root/flags3/.x1.22723" => x0 failed: Operation not permitted (1)
rsync: link "/root/flags3/.x2.22723" => x0 failed: Operation not permitted (1)
rsync: link "/root/flags3/.x3.22723" => x0 failed: Operation not permitted (1)
rsync: link "/root/flags3/.x4.22723" => x0 failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1200) [sender=3.1.2]
Comment 8 hostmaster 2017-01-06 12:24:24 UTC
I upgraded to FreeBSD 11.0-STABLE r311209 ... same failed results
as FreeBSD 6.x to 10.x - what can be done to fix this severe bug?
I tried to understand how to make a patch - however the code looks
far too gnarly for a non-rsync-developer to mess with - I'm too
afraid of breaking things :(
Comment 9 hostmaster 2017-01-06 12:59:33 UTC
After thinking this through some more I think I should close this bug report and open a new one based on FreeBSD 11.0 using the information in comments 5-8
Comment 10 Emanuel Haupt freebsd_committer 2017-01-06 13:55:06 UTC
As I've mentioned earlier I really think this should be reported and tracked upstream (see link in previous post). That way future compatibility is guaranteed.
Comment 11 hostmaster 2017-01-06 14:02:03 UTC
I don't have any official standing as a FreeBSD ports maintainer -
it's been my general experience in the past that such O/S-specific fixes are usually referred back to the O/S maintainers, or outright refused

Would you be able to forward this bug report to the rsync people so they know to take it seriously?

Thanks - BB
Comment 12 Emanuel Haupt freebsd_committer 2017-01-10 11:51:57 UTC
I've created:
https://bugzilla.samba.org/show_bug.cgi?id=12508
Comment 13 hostmaster 2017-01-10 18:57:11 UTC
Excellent - many thanks!
... also a nice job of reducing it to its essentials :)
Comment 14 Walter Schwarzenfeld freebsd_triage 2018-01-17 07:37:18 UTC
Any news here?
Comment 15 Emanuel Haupt freebsd_committer 2018-02-11 08:44:22 UTC
Back to the pool, I no longer maintain this port.
Comment 16 Rodrigo Osorio freebsd_committer 2018-02-25 14:15:18 UTC
Hi, I'm the new rsync maintainer, the ticket open by ehaupt@ is under processing by the rsync team, maybe this need further investigations to provide them a patch...
Comment 17 Rodrigo Osorio freebsd_committer 2018-09-22 09:44:37 UTC
The port work as expected for 11 and the 10 EOL is next month
Comment 18 hostmaster 2018-09-22 14:25:20 UTC
the problem  is independent of FreeBSD O/S version & is still not fixed

in FreeBSD 11.x some hard links were changed to soft links which is not in any way a fix for the problem of copying hard linked files which are immutable via chflags
Comment 19 hostmaster 2018-09-22 23:03:19 UTC
it's also still true for current version of rsync 3.1.3

tested on FreeBSD 11.2-STABLE -

FreeBSD droplet02 11.2-STABLE FreeBSD 11.2-STABLE #1 r336657: Tue Jul 24 06:09:19 EDT 2018     root@droplet02:/var/misc/usr/obj/usr/src/sys/DROPLET02  amd64

https://bugzilla.samba.org/show_bug.cgi?id=12508