Bug 223943 - ports-mgmt/portupgrade and the ICU-libraries
Summary: ports-mgmt/portupgrade and the ICU-libraries
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Bryan Drewery
URL:
Keywords:
: 237069 (view as bug list)
Depends on:
Blocks: 227599 231694 237069
  Show dependency treegraph
 
Reported: 2017-11-28 16:26 UTC by Mikhail T.
Modified: 2021-10-29 22:28 UTC (History)
5 users (show)

See Also:
bugzilla: maintainer-feedback? (bdrewery)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mikhail T. 2017-11-28 16:26:37 UTC
Every time portupgrade replaces the ICU port, the binaries trying to use libicuuc.so.* and other bits installed by that port break.

This is probably because the ICU port still uses not only major, but also the minor library versions -- something, portupgrade is not handling well.

For example, although I have the libicuio.so.59.1 in ${LOCALBASE}/lib/compat/pkg/ now, it is not being found by anything, because other binaries look for it with just .59 (without the .1). For example:

Shared object "libicuuc.so.59" not found, required by "libQt5WebKit.so.5"

Once I create the necessary symlinks, things begin to work again.

Also, and probably for the same reason, portupgrade misses these other packages -- and fails to rebuild them when upgrading ICU -- even when run with `-r -R`.

This problem may be happening with other ports as well, but ICU is used by so many, lots of people are affected...
Comment 1 Marco Beishuizen 2017-11-28 16:39:44 UTC
Using portupgrade myself, I do not have issues with icu lately. So I think it's not a portupgrade issue. In general, all ports that depend on icu need to be rebuild after an update of icu.

What I usually do though after running portupgrade is a "rm -rf /usr/local/lib/compat/pkg/*" for removing old libraries, a "pkg autoremove" for removing build dependencies and a "portsclean -C -D -DD" to clean the ports.
Comment 2 Mikhail T. 2017-11-28 16:43:51 UTC
> all ports that depend on icu need to be rebuild after an update of icu.

Yes, they need to be rebuilt. But they aren't -- unless one _manually_ searches for all such packages and explicitly lists them for forced rebuild.

And I suspect, that's because portupgrade is confused by ICU's libraries using both MAJOR and MINOR numbers.
Comment 3 Marco Beishuizen 2017-11-28 18:39:11 UTC
Last update of icu (on 9-11-2017) went fine here using portupgrade, all dependent ports were rebuilt automagically. But iirc you do need to clean old libraries in /usr/local/lib/compat/pkg/ to prevent ports being built against the old ones.
Comment 4 Mikhail T. 2017-11-28 18:45:27 UTC
> you do need to clean old libraries in /usr/local/lib/compat/pkg/

Well, I've never done this -- nor heard of it being mandatory -- but all other ports upgrade fine, whereas ICU is always a minefield.

Maybe, the actual portupgrade AUTHORS ought to chime in, instead of us, mere USERS arguing...
Comment 5 Bryan Drewery freebsd_committer freebsd_triage 2017-11-28 19:05:44 UTC
(In reply to Marco Beishuizen from comment #3)
> Last update of icu (on 9-11-2017) went fine here using portupgrade, all
> dependent ports were rebuilt automagically. But iirc you do need to clean
> old libraries in /usr/local/lib/compat/pkg/ to prevent ports being built
> against the old ones.

This shouldn't be needed as that path comes last in the search list. Portupgrade
removes files from there when appropriate / with portsclean -L.
Comment 6 Bryan Drewery freebsd_committer freebsd_triage 2017-11-28 19:31:22 UTC
Try this patch for a week or two and let me know the results please. Also weave in some
portsclean -L into your workflow to clean out old libraries.  The patch may cause
other unexpected problems but I think it's fine. Like orphaned symlinks with portsclean -L,
or something linking against /usr/local/lib/compat/pkg.

https://people.freebsd.org/~bdrewery/patches/portupgrade-pr-223943.diff
Comment 7 Mikhail T. 2017-11-28 20:08:36 UTC
(In reply to Bryan Drewery from comment #6)
> Try this patch for a week or two

Thanks, I shall. But this patch seems to be limited to the symlink-creation, is it not? What about detecting the dependencies -- so that www/qt5-webkit is rebuilt, whenever ICU is rebuilt, for example?
Comment 8 Bryan Drewery freebsd_committer freebsd_triage 2017-11-28 20:14:12 UTC
(In reply to Mikhail T. from comment #7)
> (In reply to Bryan Drewery from comment #6)
> > Try this patch for a week or two
> 
> Thanks, I shall. But this patch seems to be limited to the symlink-creation,
> is it not? What about detecting the dependencies -- so that www/qt5-webkit
> is rebuilt, whenever ICU is rebuilt, for example?

That's already covered in Bug 217284.
Comment 9 Bryan Drewery freebsd_committer freebsd_triage 2018-01-24 01:04:10 UTC
Has this patch been working ok?
Comment 10 Mikhail T. 2018-01-24 03:04:33 UTC
(In reply to Bryan Drewery from comment #9)
Yes, I think it is working. Thank you.
Comment 11 commit-hook freebsd_committer freebsd_triage 2018-03-09 19:32:41 UTC
A commit references this bug:

Author: bdrewery
Date: Fri Mar  9 19:32:18 UTC 2018
New revision: 463994
URL: https://svnweb.freebsd.org/changeset/ports/463994

Log:
  Update to 2.4.16.

   - Preserve shlib symlink [1]

  PR:		223943 [1]

Changes:
  head/ports-mgmt/portupgrade/Makefile
  head/ports-mgmt/portupgrade/distinfo
  head/ports-mgmt/portupgrade-devel/Makefile
  head/ports-mgmt/portupgrade-devel/distinfo
Comment 12 Tomoaki AOKI 2018-03-11 11:43:31 UTC
(In reply to commit-hook from comment #11)

Unfortunately, this breaks forced reinstall (with -f option) of (at least) x11/nvidia-driver. Reverting this fixed the issue.
Maybe more ports would be affected.

Error messages with ports-mgmt/portupgrade-devel are as below.
(ports-mgmt/portupgrade is not tested.)


Output of `portupgrade -f -b x11/nvidia-driver` (last portion):

--->  Build of x11/nvidia-driver ended at: Sat, 10 Mar 2018 22:55:31 +0900 (consumed 00:00:16)
--->  Uninstallation of nvidia-driver-390.25 started at: Sat, 10 Mar 2018 22:55:31 +0900
--->  Fixing up dependencies before creating a package
--->  Backing up the old version
--->  Uninstalling the old version
[Reading data from pkg(8) ... - 1335 packages found - done]
--->  Deinstalling 'nvidia-driver-390.25'
--->  Preserving /compat/linux/usr/lib/libEGL_nvidia.so.0 as /usr/local/lib/compat/pkg/libEGL_nvidia.so.0
cp: symlink: libEGL_nvidia.so.390.25: File exists
Copy failed!
[Reading data from pkg(8) ... - 1335 packages found - done]
** Listing the failed packages (-:ignored / *:skipped / !:failed)
	! nvidia-driver-390.25	(copy failed)
--->  Uninstallation of nvidia-driver-390.25 ended at: Sat, 10 Mar 2018 22:57:04 +0900 (consumed 00:01:33)
--->  Reinstallation of x11/nvidia-driver ended at: Sat, 10 Mar 2018 22:57:04 +0900 (consumed 00:01:49)
--->  ** Upgrade tasks 1: 0 done, 0 ignored, 0 skipped and 1 failed
--->  Listing the results (+:done / -:ignored / *:skipped / !:failed)
	! x11/nvidia-driver (nvidia-driver-390.25)	(uninstall error)
--->  Packages processed: 0 done, 0 ignored, 0 skipped and 1 failed
--->  Session ended at: Sat, 10 Mar 2018 22:57:04 +0900 (consumed 00:02:00)
Comment 13 lampa 2018-04-24 12:06:08 UTC
Reason of the error is this change in /usr/local/sbin/pkg_glob:

+-	if system('/bin/cp', '-pf', file, dest)
++	if system('/bin/cp', '-Rpf', file, dest)

I don't understand what this has to do, -R with symbolic link doesn't make sense. Try to remove it and it should work better, at least it'll overwrite existing link without error. Preserving symbolic links however will never work as portupgrade calls portsclean -QL at the end:

--->  Cleaning out obsolete shared libraries

man portsclean 

     -L
     --libclean     Clean out old, duplicate and/or orphaned shared libraries.

                    portsclean first deletes duplicate shared libraries where
                    appropriate, then puts away old and orphaned shared
                    libaries to /usr/local/lib/compat/pkg updating symlinks
                    properly.  To keep binaries working, ldconfig(8) is run
                    after each library deletion or move.

Example:
Phase preserving in portupgrade
--->  Preserving /usr/local/lib/libdb-5.3.so.0 as /usr/local/lib/compat/pkg/libdb-5.3.so.0
   cp -pf /usr/local/lib/libdb-5.3.so.0 /usr/local/lib/compat/pkg/libdb-5.3.so.0
It's really there at this moment:
   ls -l  /usr/local/lib/compat/pkg/libdb-5.3.so.0
   -rwxr-xr-x  1 root  wheel  1652536 24 Apr 13:26 /usr/local/lib/compat/pkg/libdb-5.3.so.0
...
Phase Cleaning in portupgrade
   portsclean -L 
** /usr/local/lib/compat/pkg/libdb-5.3.so.0 is shadowed by /usr/local/lib/libdb-5.3.so.0
 --> The one in /usr/local/lib/compat/pkg is not used
Delete /usr/local/lib/compat/pkg/libdb-5.3.so.0

So this change in portupgrade is not very useful, all (or most) preserved symlinks are immediately deleted and "Preserving ..." message is now very confusing.
Comment 14 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-07 12:26:17 UTC
Re-opening due to fallout from ports r463994 (originally reported in comment 12) and then in:

Bug 237069 
Bug 231694 
Bug 227599
Comment 15 Bryan Drewery freebsd_committer freebsd_triage 2021-10-29 21:40:09 UTC
*** Bug 237069 has been marked as a duplicate of this bug. ***
Comment 16 Bryan Drewery freebsd_committer freebsd_triage 2021-10-29 21:47:56 UTC
> I don't understand what this has to do, -R with symbolic link doesn't make sense.

In the cp(1) manpage:

> -R ... This option also causes symbolic links to be copied, rather than indirected through

It seems like a bug in cp(1) to me. How does someone copy a symlink onto an existing symlink? -f is supposed to avoid the error but it does not.

GNU cp gets this right.

~/git/pkgtools % ls -al a
total 10
drwxr-xr-x   2 bryan  bryan   4 Oct 29 14:42 ./
drwxr-xr-x  13 bryan  bryan  21 Oct 29 14:42 ../
-rw-r--r--   1 bryan  bryan   0 Oct 29 14:42 1
lrwxr-xr-x   1 bryan  bryan   4 Oct 29 14:42 2@ -> blah
~/git/pkgtools % cp -R a b
~/git/pkgtools % ls -al b
total 10
drwxr-xr-x   2 bryan  bryan   4 Oct 29 14:46 ./
drwxr-xr-x  13 bryan  bryan  21 Oct 29 14:46 ../
-rw-r--r--   1 bryan  bryan   0 Oct 29 14:46 1
lrwxr-xr-x   1 bryan  bryan   4 Oct 29 14:46 2@ -> blah
~/git/pkgtools % find a \( -type f -o -type l \) -exec cp -vRfp {} b/ \;
cp: symlink: blah: File exists
a/1 -> b/1


And GNU cp:

~/git/pkgtools % find a \( -type f -o -type l \) -exec gcp -vRfp {} b/ \;
removed 'b/2'
'a/2' -> 'b/2'
'a/1' -> 'b/1'
~/git/pkgtools % ls -al b
total 10
drwxr-xr-x   2 bryan  bryan   4 Oct 29 14:46 ./
drwxr-xr-x  13 bryan  bryan  21 Oct 29 14:46 ../
-rw-r--r--   1 bryan  bryan   0 Oct 29 14:42 1
lrwxr-xr-x   1 bryan  bryan   4 Oct 29 14:42 2@ -> blah


Without -R it is wrong too as it tries to dereference a/2 to missing blah.

~/git/pkgtools % find a \( -type f -o -type l \) -exec gcp -vfp {} b/ \;
gcp: cannot stat 'a/2': No such file or directory
'a/1' -> 'b/1'
Comment 17 Bryan Drewery freebsd_committer freebsd_triage 2021-10-29 21:48:30 UTC
To be clear the first example is FreeBSD cp.
Comment 18 Dmitry Petrov 2021-10-29 22:28:48 UTC
(In reply to Bryan Drewery from comment #17)

Also see bug 174489.