Bug 197542 - Mk/Scripts/find-lib.sh: Doesn't find libiconv library
Summary: Mk/Scripts/find-lib.sh: Doesn't find libiconv library
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Port Management Team
URL:
Keywords: needs-qa, patch
Depends on:
Blocks:
 
Reported: 2015-02-11 17:37 UTC by cgreen
Modified: 2015-08-30 18:56 UTC (History)
2 users (show)

See Also:


Attachments
Patch to resolve multiple-level symlinks (1.22 KB, patch)
2015-02-11 18:26 UTC, cgreen
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description cgreen 2015-02-11 17:37:21 UTC
When installing gettext-tools, I get the following:

===>  Patching for gettext-tools-0.19.4
===>  Applying FreeBSD patches for gettext-tools-0.19.4
===>   gettext-tools-0.19.4 depends on shared library: libexpat.so - found (/usr/local/lib/libexpat.so.1.6.0)
===>   gettext-tools-0.19.4 depends on shared library: libintl.so - found (/usr/local/lib/libintl.so.8.1.3)
===>   gettext-tools-0.19.4 depends on shared library: libiconv.so.3 - not found

libiconv.so.3 is not found.

However, the package is installed:

cgreen:Smtp1:/usr/local/lib# pkg info libiconv
libiconv-1.14_6
Name           : libiconv
Version        : 1.14_6
Installed on   : Wed Feb 11 11:45:01 EST 2015
Origin         : converters/libiconv


And the files are present in /usr/local/lib:

cgreen:Smtp1:/usr/local/lib# ll /usr/local/lib/libicon*
-rw-r--r--  1 root  wheel  1117168 Feb 11 11:44 /usr/local/lib/libiconv.a
lrwxr-xr-x  1 root  wheel       17 Feb 11 11:44 /usr/local/lib/libiconv.so@ -> libiconv.so.2.5.1
lrwxr-xr-x  1 root  wheel       17 Feb 11 11:44 /usr/local/lib/libiconv.so.2@ -> libiconv.so.2.5.1
-rw-r--r--  1 root  wheel  1089102 Feb 11 11:44 /usr/local/lib/libiconv.so.2.5.1
lrwxr-xr-x  1 root  wheel       13 Feb 11 11:45 /usr/local/lib/libiconv.so.3@ -> libiconv.so.2


Therein lies the problem.

Gettext requires libiconv.so.3, which is a symlink.

/usr/ports/Mk/bsd.port.mk calls /usr/ports/Mk/Scripts/find-lib.sh to find the absolute path to the library.

find-lib.sh has a routine to resolve symlinks, but it fails (after setting the appropriate env vars):

cgreen:Smtp1:/usr/local/lib# sh /usr/ports/Mk/Scripts/find-lib.sh libiconv.so.3
cgreen:Smtp1:/usr/local/lib#

It's failing because while the libiconv.so.3 symlink is resolved to /usr/local/lib/libiconv.so.2, libiconv.so.2 is, as above, another symlink.

The resolved file /usr/local/lib/libiconv.so.2 is passed to "file -b -L --mime-type" which dutifully reports it as a symlink and not the x-sharedlib find-lib.sh is looking for, and find-lib exits unsuccessfully.

If you pass find-lib.sh libivonv.so.2 instead, it works as intended:

cgreen:Smtp1:/usr/local/lib# sh /usr/ports/Mk/Scripts/find-lib.sh libiconv.so.2
/usr/local/lib/libiconv.so.2.5.1

However, the double-symlink that libiconv installs (.so.3 -> .so.2 -> .so.2.5.1) trips up find-lib.sh.

While the issue can be fixed by having libiconv link .so.3 directly to .so.2.5.1 (my current workaround), I have no idea how many other ports may use a double-symlink, so I'm guessing fixing the find-lib.sh tool to watch for double-symlinks (or more levels or redirection?) would likely be easiest.
Comment 1 cgreen 2015-02-11 18:26:45 UTC
Created attachment 152872 [details]
Patch to resolve multiple-level symlinks

Very lightly tested.

I'm assuming infinite loop prevention is needed, but I have no idea if 10 loops is a reasonable cut-off.
Comment 2 Antoine Brodin freebsd_committer 2015-02-11 18:29:32 UTC
I can't reproduce your problem:

# env LIB_DIRS="/lib /usr/lib /usr/local/lib" LOCALBASE=/usr/local sh /usr/ports/Mk/Scripts/find-lib.sh libiconv.so
/usr/local/lib/libiconv.so.2.5.1
# env LIB_DIRS="/lib /usr/lib /usr/local/lib" LOCALBASE=/usr/local sh /usr/ports/Mk/Scripts/find-lib.sh libiconv.so.2
/usr/local/lib/libiconv.so.2.5.1
# env LIB_DIRS="/lib /usr/lib /usr/local/lib" LOCALBASE=/usr/local sh /usr/ports/Mk/Scripts/find-lib.sh libiconv.so.3
/usr/local/lib/libiconv.so.2
# pkg which /usr/local/lib/libiconv.so.2.5.1
/usr/local/lib/libiconv.so.2.5.1 was installed by package libiconv-1.14_6
# pkg which /usr/local/lib/libiconv.so.2
/usr/local/lib/libiconv.so.2 was installed by package libiconv-1.14_6
Comment 3 cgreen 2015-02-11 18:38:07 UTC
(In reply to Antoine Brodin from comment #2)

find-lib doesn't resolve multiple symlinks, at least as of:

# $FreeBSD: head/Mk/Scripts/find-lib.sh 378742 2015-02-09 17:16:43Z bapt $

So what does your libiconv.so.3 point to?

If it points to .so.2, then I'm confused, unless you get something different from readlink's output:

cgreen:Smtp1:/usr/ports/Mk/Scripts# readlink /usr/local/lib/libiconv.so.3
libiconv.so.2

And thence file's output:

cgreen:Smtp1:/usr/ports/Mk/Scripts# file -b -L --mime-type /usr/local/lib/libiconv.so.2
application/x-symlink

This is on a fairly recent FreeBSD 8.4-STABLE box, if it matters:

cgreen:Smtp1:/usr/ports/Mk/Scripts# uname -a
FreeBSD smtp1.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #6 r274586: Wed Dec 10 09:35:46 EST 2014     mdtancsa@smtp1.sentex.ca:/usr/obj/usr/src/sys/server  amd64
Comment 4 Antoine Brodin freebsd_committer 2015-02-11 18:43:24 UTC
On 8.4-RELEASE:

# file -b -L --mime-type /usr/local/lib/libiconv.so.2
application/x-sharedlib

# ll /usr/local/lib/libiconv.so*
lrwxr-xr-x  1 root  wheel       17 Dec  1 21:31 /usr/local/lib/libiconv.so@ -> libiconv.so.2.5.1
lrwxr-xr-x  1 root  wheel       17 Dec  1 21:31 /usr/local/lib/libiconv.so.2@ -> libiconv.so.2.5.1
-rw-r--r--  1 root  wheel  1089102 Dec  1 21:31 /usr/local/lib/libiconv.so.2.5.1
lrwxr-xr-x  1 root  wheel       13 Dec  1 21:31 /usr/local/lib/libiconv.so.3@ -> libiconv.so.2

file -L is supposed to follow symlinks
Comment 5 Antoine Brodin freebsd_committer 2015-02-11 18:44:18 UTC
# file -b -L --mime-type /usr/local/lib/libiconv.so.3
application/x-sharedlib
Comment 6 cgreen 2015-02-11 19:02:19 UTC
(In reply to Antoine Brodin from comment #5)

That's neat, except mine doesn't:

cgreen:Smtp1:/usr/ports/Mk/Scripts# file -b -L --mime-type /usr/local/lib/libiconv.so.2
application/x-symlink


Does on a FreeBSD 10 box, though:

cgreen:Smtp2:/usr/ports/devel/gettext-tools# uname -a
FreeBSD smtp2.sentex.ca 10.1-STABLE FreeBSD 10.1-STABLE #0 r277835: Wed Jan 28 13:37:52 EST 2015     cgreen@smtp2.sentex.ca:/usr/obj/usr/src/sys/server  amd64

cgreen:Smtp2:/usr/ports/devel/gettext-tools# file -b -L --mime-type /usr/local/lib/libiconv.so.2
application/x-sharedlib

cgreen:Smtp2:/usr/ports/devel/gettext-tools# file -b -L --mime-type /usr/local/lib/libiconv.so.3
application/x-sharedlib


It works on a more recent FreeBSD 8-STABLE:

FreeBSD smarthost1.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #10 r277831: Wed Jan 28 11:23:30 EST 2015     cgreen@smarthost1.sentex.ca:/usr/obj/usr/src/sys/smarthost  amd64

cgreen:Smarthost1:/usr/local/lib# file -b -L --mime-type /usr/local/lib/libiconv.so.3
application/x-sharedlib


Problem with file on "older" FreeBSD 8s? And by "older" I mean anything apparently over a month?


Also, if file dereferences symlinks, why is there a resolv_symlink() function in find-lib.sh?

To my thinking, if you're going to have a belt and suspenders, you might as well make certain the suspenders actually work. ;-)
Comment 7 cgreen 2015-02-11 19:05:03 UTC
Actually, while I quoted above that it worked on FreeBSD 8.4-STABLE #10 r277831:

cgreen:Smarthost1:/usr/local/lib# file -b -L --mime-type /usr/local/lib/libiconv.so.3
application/x-sharedlib

That was on an AMD64 box:

cgreen:Smarthost1:/usr/local/lib# uname -a
FreeBSD smarthost1.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #10 r277831: Wed Jan 28 11:23:30 EST 2015     cgreen@smarthost1.sentex.ca:/usr/obj/usr/src/sys/smarthost  amd64


It *doesn't* work on the same revision, but i386:

cgreen:Marble:/usr/ports/net-mgmt/nrpe# uname -a
FreeBSD marble.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #11 r277831: Wed Jan 28 10:20:51 EST 2015     cgreen@marble.sentex.ca:/usr/obj/usr/src/sys/Marble8  i386

cgreen:Marble:/usr/ports/net-mgmt/nrpe# file -b -L --mime-type /usr/local/lib/libiconv.so.3
application/x-symlink

This is a box that was, as above, compiled two weeks ago.
Comment 8 cgreen 2015-02-11 19:34:01 UTC
Okay, so going over my FreeBSD 8.4 boxes, I can't find a pattern, other than older releases don't work and some newer ones do.

Actually, I found a FreeBSD 9.3-STABLE box from a couple weeks ago that also doesn't work, so it's not just a FreeBSD 8 issue.


Don't work:

cgreen:Granite:/root# uname -a ; file --version ; file -b -L --mime-type /usr/local/lib/libiconv.so.3 ; ls -l /usr/local/lib/libiconv.so.3
FreeBSD granite.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #8 r274138: Wed Nov  5 12:21:05 EST 2014     cgreen@granite.sentex.ca:/usr/obj/usr/src/sys/granite  i386
file-5.03
magic file from /usr/share/misc/magic
application/x-symlink
lrwxr-xr-x  1 root  wheel  13 Dec 18 10:47 /usr/local/lib/libiconv.so.3@ -> libiconv.so.2


FreeBSD marble.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #11 r277831: Wed Jan 28 10:20:51 EST 2015     cgreen@marble.sentex.ca:/usr/obj/usr/src/sys/Marble8  i386
file-5.03
magic file from /usr/share/misc/magic
application/x-symlink
lrwxr-xr-x  1 root  wheel  13 Dec 10 09:29 /usr/local/lib/libiconv.so.3@ -> libiconv.so.2

FreeBSD pumice7.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #1: Thu Jun 12 14:14:47 EDT 2014     cgreen@pumice7.sentex.ca:/usr/obj/usr/src/sys/SPAM32_REL8  i386
file-5.03
magic file from /usr/share/misc/magic
application/x-symlink
lrwxr-xr-x  1 root  wheel  13 Jan 14 16:29 /usr/local/lib/libiconv.so.3@ -> libiconv.so.2


FreeBSD smtp1.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #6 r274586: Wed Dec 10 09:35:46 EST 2014     mdtancsa@smtp1.sentex.ca:/usr/obj/usr/src/sys/server  amd64
file-5.03
magic file from /usr/share/misc/magic
application/x-symlink
lrwxr-xr-x  1 root  wheel  13 Feb 11 11:45 /usr/local/lib/libiconv.so.3@ -> libiconv.so.2


FreeBSD jade.sentex.ca 9.3-STABLE FreeBSD 9.3-STABLE #1 r277835: Wed Jan 28 13:11:24 EST 2015     cgreen@jade.sentex.ca:/usr/obj/usr/src/sys/jade  i386
file-5.22
magic file from /usr/share/misc/magic
inode/symlink
lrwxr-xr-x  1 root  wheel  13 Dec 15 10:54 /usr/local/lib/libiconv.so.3@ -> libiconv.so.2




Do work:

FreeBSD auth.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #27 r277831: Wed Jan 28 10:13:37 EST 2015     cgreen@auth.sentex.ca:/usr/obj/usr/src/sys/server  i386
file-5.03
magic file from /usr/share/misc/magic
application/x-sharedlib
lrwxr-xr-x  1 root  wheel  13 Dec 10 10:59 /usr/local/lib/libiconv.so.3@ -> libiconv.so.2

FreeBSD auth2.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #17 r277831: Wed Jan 28 10:24:02 EST 2015     cgreen@auth2.sentex.ca:/usr/obj/usr/src/sys/server  i386
file-5.03
magic file from /usr/share/misc/magic
application/x-sharedlib
lrwxr-xr-x  1 root  wheel  13 Jan  6 17:16 /usr/local/lib/libiconv.so.3@ -> libiconv.so.2

FreeBSD smarthost1.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #10 r277831: Wed Jan 28 11:23:30 EST 2015     cgreen@smarthost1.sentex.ca:/usr/obj/usr/src/sys/smarthost  amd64
file-5.03
magic file from /usr/share/misc/magic
application/x-sharedlib
lrwxr-xr-x  1 root  wheel  13 Feb 11 12:13 /usr/local/lib/libiconv.so.3@ -> libiconv.so.2

FreeBSD smarthost2.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #13 r277831: Wed Jan 28 11:30:05 EST 2015     cgreen@smarthost2.sentex.ca:/usr/obj/usr/src/sys/server  amd64
file-5.03
magic file from /usr/share/misc/magic
application/x-sharedlib
lrwxr-xr-x  1 root  wheel  13 Dec 18 16:16 /usr/local/lib/libiconv.so.3@ -> libiconv.so.2

FreeBSD pyroxene.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #5 r274353: Mon Nov 10 21:25:43 EST 2014     mdtancsa@pyroxene.sentex.ca:/usr/obj/usr/src/sys/pyroxene  amd64
file-5.03
magic file from /usr/share/misc/magic
application/x-sharedlib
-rw-r--r--  1 root  wheel  1088983 Oct 20 13:28 /usr/local/lib/libiconv.so.3

FreeBSD vinyl3.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #8 r277831: Wed Jan 28 10:16:57 EST 2015     cgreen@vinyl3.sentex.ca:/usr/obj/usr/src/sys/vinyl  i386
file-5.03
magic file from /usr/share/misc/magic
application/x-sharedlib
-r--r--r--  1 root  wheel  1060625 Apr  9  2014 /usr/local/lib/libiconv.so.3

FreeBSD vinyl4.sentex.ca 8.4-STABLE FreeBSD 8.4-STABLE #6 r277831: Wed Jan 28 10:16:32 EST 2015     cgreen@vinyl4.sentex.ca:/usr/obj/usr/src/sys/vinyl  i386
file-5.03
magic file from /usr/share/misc/magic
application/x-sharedlib
-rw-r--r--  1 root  wheel  1060625 Nov 11 16:01 /usr/local/lib/libiconv.so.3

FreeBSD vinyl6.sentex.ca 9.3-STABLE FreeBSD 9.3-STABLE #11 r277831: Wed Jan 28 10:37:05 EST 2015     mdtancsa@vinyl6.sentex.ca:/usr/obj/usr/src/sys/server  amd64
file-5.22
magic file from /usr/share/misc/magic
application/x-sharedlib
lrwxr-xr-x  1 root  wheel  - 13 Dec 17 11:21 /usr/local/lib/libiconv.so.3 -> libiconv.so.2
Comment 9 cgreen 2015-02-11 19:43:36 UTC
And apparently I wasn't looking too closely at my own output, and a few of those "works" listings work because libiconv.so.3 is actually a file, and not a symlink, so ignore those. ;-)


I'll try to see why it's a file on some boxes and a link on others, but in the meantime it really seems to me that sometimes file -L resolves symlinks and sometimes it doesn't.

Since 8.4 and 9.3 remain production releases of FreeBSD, the ports tree needs to work for them -- and for at least some boxes, it won't.

find-lib.sh has a mechanism for resolving symlinks that at first glance seems extraneious.  Except possibly, I'm thinking, it's there because someone knows that file's -L behaviour is occasionally wonky and the resolve_symlinks subroutine was added to work around any issue file -L might have.

However, the subroutine can only resolve a single symlink redirection, and sometimes more is needed.

Patching find-lib.sh so it can resolve multiple layers of symlinks in those circumstances where file isn't working seems to me to be the right thing to do.
Comment 10 cgreen 2015-02-11 20:21:13 UTC
Okay, I've got it nailed down.

Examining the truss output of Marble (a box where file doesn't work) and Auth (a box where it does), revealed that file on Marble was reading in a file:

/usr/share/misc/magic.mime.mgc

This file doesn't exist on Auth.

When the file is removed on Marble, then Marble suddenly starts working.

cgreen:Marble:/usr/ports/net-mgmt/nrpe# ll /usr/share/misc/magic.mime.mgc
-r--r--r--  1 root  wheel  46208 Aug 17  2011 /usr/share/misc/magic.mime.mgc

The file date on the magic file is pretty old, which makes me think it's from a previous FreeBSD version.

With that clue, I could then see that every box which was having problems had been upgraded from (at least) FreeBSD 7, up to FreeBSD 8 (and thence up to FreeBSD 9, in the case of Jade, which also doesn't work).


So, it would seem that on virgin installations of FreeBSD8+ that there won't be any issues.

However, for any sysadmin who has upgraded their box from FreeBSD 7 (or earlier) up through FreeBSD8, the current ports tree will have issues when a port looks for a library through more a single level of symlink redirection -- like gettext requiring libiconv.so.3.
Comment 11 Antoine Brodin freebsd_committer 2015-02-11 20:33:47 UTC
Well, file(1) probably has more problems than this one when an old magic.mime.mgc is present.
It seems I committed  https://svnweb.freebsd.org/changeset/base/187694 6 years ago to help sysadmins to remove this file.
I will discuss with bapt if it's worth working around or not.
Comment 12 cgreen 2015-02-11 21:20:53 UTC
(In reply to Antoine Brodin from comment #11)

Fair cop -- apparently it's been far, far too long since I've bothered to read the handbook, and either the "make delete-old" and "make delete-old-libs" instructions weren't there, or my eyes glazed over after mergemaster, or it simply fell out of my head.

The situation is resolved to my satisfaction (by which I mean all my boxes work and if it crops up in the future then I know how to handle the issue) so what you do from here makes little difference to me.


However, I'd argue that either find-lib.sh should have the resolve_symlinks subroutine removed as it just replicates the functionality of "find -L", *or* resolve_symlinks should be fixed up so that it works in all cases.

As it stands, resolve_symlinks is either redundant or broken, and as there are circumstances where the redundancy is needed -- albeit on some apparently ill-kept machines -- I'd argue for fixing resolve_symlinks.

Please see the patch I attached earlier for one simple way of addressing the issue.



I very much appreciate your attention on this bug report. :-)

Regards, and cheers.
Comment 13 Antoine Brodin freebsd_committer 2015-02-11 23:11:46 UTC
Thanks for investigating,  now we know the root cause if another user reports the same issue.

In Mk/Scripts/actual-package-depends.sh ,  the goal is to find to which package an executable/file/package belongs to.
"pkg which" is ran first on the file/link, and if no package is found, "pkg which" is ran on a 1 step symlink resolution of the link.  (we do not use realpath because it could provide a wrong path if /usr/local is a symlink ;  we don't do a full resolution either because some metaports have things like python provided by lang/python -> python2 provided by lang/python2 -> python2.7 provided by lang/python27).
The 1 step symlink resolution is just to work around a few buggy ports (mainly gnupg*) that create symlinks that are not registered by the package manager (they are created by some install scripts).

In Mk/Scripts/find-lib.sh,  this code has been copied,  but I believe it's not necessary to resolve the link (except when file(1) is buggy where it can help a bit).

I switched Uses=iconv to depend on libiconv.so instead of the compat libiconv.so.3, it may help in your case as it changes from a double-symlink to a simple-symlink.

If other users report the same issue,  we may put a note in UPDATING to remind them about make check-old/make delete-old.
Comment 14 Friedrich Volkmann 2015-02-13 10:32:35 UTC
This is a serious bug, because lots of ports depend on e.g. libpng (which has a chain of symlinks) directly or indirectly. That's why installing most ports is currently broken, including Firefox.

I think that there's no need for a (possibly infinite) loop, because the -f flag does a recursive lookup. The following patch works for me:

Scripts# diff -u find-lib.sh.orig find-lib.sh
--- find-lib.sh.orig    2015-02-10 11:38:50.000000000 +0100
+++ find-lib.sh 2015-02-13 11:01:41.000000000 +0100
@@ -22,7 +22,7 @@
                return
        fi
 
-       tgt=`readlink ${file}`
+       tgt=`readlink -f ${file}`
        case $tgt in
        /*)
                echo $tgt(In reply to cgreen from comment #1)
Comment 15 Antoine Brodin freebsd_committer 2015-02-13 10:38:57 UTC
(In reply to bsd from comment #14)

This is wrong,  this breaks dependency registration when /usr/local is a symlink

Please remove stage magic.mime.mgc instead.
Comment 16 commit-hook freebsd_committer 2015-02-13 19:05:18 UTC
A commit references this bug:

Author: antoine
Date: Fri Feb 13 19:04:43 UTC 2015
New revision: 378939
URL: https://svnweb.freebsd.org/changeset/ports/378939

Log:
  Add a note in UPDATING about dependency registration rework

  PR:		197542

Changes:
  head/UPDATING