Bug 245191

Summary: devel/tig: Fails to run with Undefined symbol "rl_change_environment"
Product: Ports & Packages Reporter: Danny McGrath <danmcgrath.ca>
Component: Individual Port(s)Assignee: Tobias Kortkamp <tobik>
Status: Closed Works As Intended    
Severity: Affects Only Me CC: nefar
Priority: --- Flags: tobik: maintainer-feedback+
Version: Latest   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
Partial tig build log none

Description Danny McGrath 2020-03-30 17:18:21 UTC
Hi,

I noticed that our poudriere built devel/tig version 2.5.0 from 2020Q1 no longer runs, and only gives an error:

  Undefined symbol "rl_change_environment" referenced from COPY relocation in /usr/local/bin/tig

I tried uninstall/reinstall tig and git, but it didn't help. Any ideas what might be the problem here?


--

#> pkg info tig
tig-2.5.0
Name           : tig
Version        : 2.5.0
Installed on   : Wed Mar 25 16:43:17 2020 CET
Origin         : devel/tig
Architecture   : FreeBSD:11:amd64
Prefix         : /usr/local
Categories     : devel
Licenses       : GPLv2
Maintainer     : tobik@FreeBSD.org
WWW            : https://jonas.github.io/tig/
Comment        : Text-mode interface for git
Options        :
        GIT            : off
Annotations    :
        FreeBSD_version: 1103000
        repo_type      : binary
        repository     : poudriere
Flat size      : 680KiB
Comment 1 Tobias Kortkamp freebsd_committer freebsd_triage 2020-03-30 17:44:58 UTC
tig should depend on libreadline.so.8 (devel/readline from ports)
but for some reason it does not appear in your pkg info output.

Shared Libs required:
	libreadline.so.8

I wonder why that is.
Comment 2 Danny McGrath 2020-03-30 18:04:54 UTC
Created attachment 212868 [details]
Partial tig build log
Comment 3 Danny McGrath 2020-03-30 18:05:29 UTC
(In reply to Tobias Kortkamp from comment #1)

I was curious too after one of the developers in our community also mentioned readline. I took a look into readline stuff, but I wasn't able to understand exactly what the problem is. But not that you draw attention to the fact that it is indeed missing, I guess that makes sense as the problem.

I will take a look at the logs and the make.conf and see if anything stands out.

<minutes later>

Looking at a recent build log that included tig, I found only some entries that I attached. I also noticed `lang_php72_UNSET+= READLINE.`, but that should only affect php72, and the logs suggest it found readline fine, afaict.
Comment 4 Danny McGrath 2020-03-31 01:02:12 UTC
Okay, while I was going through my make.conf, it occurred to me that my make.conf may have had some typos for php.

Specifically, for www_mod_php72 and lang_php72, I was setting ZTS, then for lang_php72-extensions, I was setting the stuff that I want enabled, but then for the UNSET, I missed the "extensions" part of the declaration.

So instead of:

 "lang_php72-extensions_UNSET+= ... READLINE"

I had:

 "lang_php72_UNSET+= ... READLINE"

It never occurred to me that there was a typo, and was easy to overlook. I did a bulk build though, and it never rebuilt the packages, and ldd was showing file itself as linked against readline, so there may be more to this.

I will try clearing some caches on a build or two and see if it helps, but so far it didn't appear to solve the build. Either way, I'll let you know how it goes and maybe consider this lower priority in the meantime.
Comment 5 Danny McGrath 2020-03-31 01:32:27 UTC
An update:

I cleared the php72-extensions and tig ports from the cache and ran a quick bulk, then force upgraded tig, but it still didn't work. It does however, show the libreadline.so.8 lib as required now:

--
tig-2.5.0
Name           : tig
Version        : 2.5.0
Installed on   : Tue Mar 31 03:29:12 2020 CEST
Origin         : devel/tig
Architecture   : FreeBSD:11:amd64
Prefix         : /usr/local
Categories     : devel
Licenses       : GPLv2
Maintainer     : tobik@FreeBSD.org
WWW            : https://jonas.github.io/tig/
Comment        : Text-mode interface for git
Options        :
        GIT            : off
Shared Libs required:
        libreadline.so.8
Annotations    :
        FreeBSD_version: 1103000
        repo_type      : binary
        repository     : poudriere
Flat size      : 680KiB
--

My guess is that I may need to fully clear the cache, then reinstall everything in the jail, just to be safe.

Again, will keep you posted, but it's looking like it's just PEBKAC!
Comment 6 Danny McGrath 2020-04-01 01:37:04 UTC
Okay, I had a chance to get a maintenance window and reinstall all of the ports after they were flushed from the cache and recompiled, but it appears that tig still is giving the same error on 11.3.

I am kinda stuck trying to explain what it could be here, given that all logs suggest that the port is being linked against libreadline. Do you have any advice or suggestions on what I could maybe look at to try resolve this problem?
Comment 7 Tobias Kortkamp freebsd_committer freebsd_triage 2020-04-04 14:33:21 UTC
Right, so I can reproduce the problem by installing misc/compat10x (which
installs a libreadline.so.8 that does not have rl_change_environment) and
attempting to start tig with

$ env LD_LIBRARY_PATH=/usr/local/lib/compat tig
ld-elf.so.1: Undefined symbol "rl_change_environment" referenced from COPY relocation in /usr/local/bin/tig
Comment 8 Danny McGrath 2020-04-12 15:49:32 UTC
I am sorry, but why was this closed as works as expected? Do you have any suggestions on what might be causing my installation to compile improperly, or where I might look?
Comment 9 Tobias Kortkamp freebsd_committer freebsd_triage 2020-04-12 16:00:45 UTC
(In reply to Danny McGrath from comment #8)
See comment #7?  If you do not have misc/compat10x installed on the affected
system then you might have a stale libreadline.so.8 lying around. If you do
have compat10x installed make sure /usr/local/lib/compat is not before
/usr/local/lib in your ldconfig path (by default this is not the case).
Comment 10 Danny McGrath 2020-04-12 16:22:32 UTC
(In reply to Tobias Kortkamp from comment #9)

Aha, indeed I do appear to have a libreadline sitting around. Not sure where it was left from, however.

  #> locate libreadline
  /usr/local/lib/libreadline.a
  /usr/local/lib/libreadline.so
  /usr/local/lib/libreadline.so.8
  /usr/local/lib/libreadline.so.8.0


  #> pkg info | grep readline
  php72-readline-7.2.29          The readline shared extension for php
  readline-8.0.4                 Library for editing command lines as they are typed

  #> pkg info -r readline-8.0.4
  readline-8.0.4:
          sqlite3-3.30.1_1,1
          python37-3.7.7
          tig-2.5.0
          php72-readline-7.2.29
          gdbm-1.18.1_1

I can confirm that the file was compiled the other day, and installed from this compile run on poudriere:

  #> pkg info readline-8.0.4
  readline-8.0.4
  Name           : readline
  Version        : 8.0.4
  Installed on   : Mon Apr  6 23:06:42 2020 CEST
  [...]

The timestamps on the files also suggest they came from the pkg:

  #> ls -lh //usr/local/lib/libreadline.*
  -rw-r--r--  1 root  wheel   659K Apr  3 16:14 //usr/local/lib/libreadline.a
  lrwxr-xr-x  1 root  wheel    16B Apr  3 16:14 //usr/local/lib/libreadline.so -> libreadline.so.8
  lrwxr-xr-x  1 root  wheel    18B Apr  3 16:14 //usr/local/lib/libreadline.so.8 -> libreadline.so.8.0
  -rw-r--r--  1 root  wheel   338K Apr  3 16:14 //usr/local/lib/libreadline.so.8.0

Also, I do not have any of the compat* ports installed on the system, so there shouldn't be anything in that regard.

So it seems that my build environment is linking against readline 8.0.4. Any ideas about disabling this just for tig? The only make option that I see for it is for pulling in git, so I don't really have much of a choice in my environment but to pull in readline or not at all, since it appears that a bunch of stuff is using it on our hosts. Surely there must be a solution to this?
Comment 11 Tobias Kortkamp freebsd_committer freebsd_triage 2020-04-12 16:32:31 UTC
What does "ldconfig -r | grep libreadline" return?
Comment 12 Danny McGrath 2020-04-12 16:33:34 UTC
(In reply to Tobias Kortkamp from comment #11)

#> ldconfig -r | grep libreadline
        2:-lreadline.8 => /lib/libreadline.so.8
        233:-lreadline.8 => /usr/local/lib/libreadline.so.8
Comment 13 Tobias Kortkamp freebsd_committer freebsd_triage 2020-04-12 16:37:39 UTC
Where does /lib/libreadline.so.8 come from?  It should not exist on any
supported release.
Comment 14 Danny McGrath 2020-04-12 17:25:32 UTC
(In reply to Tobias Kortkamp from comment #13)

Good question. It is an ezjail environment, so:

  #> ls -ld /lib
  lrwxr-xr-x  1 root  wheel  13 May  9  2014 /lib -> /basejail/lib

And that entire jail gets rebuild during upgrades when the host is upgraded, so there shouldn't ideally be anything left over like this.

  #> ls -l /lib/libreadline.so.8
  -r--r--r--  1 root  wheel  273064 May  3  2017 /lib/libreadline.so.8

Indeed it has been an a very long time. I guess there is some cruft from ezjail, but a installworld should have cleaned that up. 

I did a quick `make DESTDIR=/ezjail/.... check-old` and it didn't find anything about that lib either. I wonder if needs a good cleanup.
Comment 15 Danny McGrath 2020-04-12 17:29:42 UTC
A quick check of the files in `/lib` shows that everything is Jan 17, minus:

#> ls -l /lib/ | grep -v "Jan 17"
total 8396
-r--r--r--  1 root  wheel   187224 May  3  2017 libcam.so.6
-r--r--r--  1 root  wheel  2043280 May  3  2017 libcrypto.so.7
-r--r--r--  1 root  wheel     7704 May  3  2017 libipx.so.5
-r--r--r--  1 root  wheel    35288 May  3  2017 libkvm.so.6
-r--r--r--  1 root  wheel   273064 May  3  2017 libreadline.so.8

So clearly it's not horrible, but there are some cruft files around. I can only guess that mtree isn't checking for them or whatever mechanism FreeBSD uses for that stuff.

Curious what the approach to clean this up is. I guess rm, then restart the jails so they aren't starting against those files. Sound about right?
Comment 16 Tobias Kortkamp freebsd_committer freebsd_triage 2020-04-12 17:36:24 UTC
> I did a quick `make DESTDIR=/ezjail/.... check-old` and it didn't find anything > about that lib either.

AFAIK check-old does not check for old libraries.  Try with check-old-libs and
then maybe delete-old-libs.
Comment 17 Danny McGrath 2020-04-12 17:38:47 UTC
(In reply to Tobias Kortkamp from comment #16)

I tried both, and it doesn't report anything unusual, it just returns to the cli immediately like it's supposed to when there is nothing to do.

I've been running that command for years just fine during upgrades, so I am not sure what would cause it to miss old files.
Comment 18 Danny McGrath 2020-04-12 20:27:52 UTC
Ahh, my mistake here! Seems that I was always doing a check-old-libs etc on the individual jails, and not on the basejail, which of course reports a bunch of old libs, one of which is readline!

I guess the problem was that I was assuming (incorrectly) that the basejail was rm'd in ezjail and reinstalled from scratch. Apologies for the noise, and excuse me while I go clean up a bunch of old libs!
Comment 19 Danny McGrath 2020-04-14 13:56:30 UTC
Just a bit of a follow up that after I cleaned the old libs from the basejail path, all the cruft libs are gone and tig runs perfect again.

Sorry for the noise, and thanks for the help! Cheers o/
Comment 20 nefar 2020-05-09 03:06:45 UTC
Hi, i'm running across the same error with the misc/nnn port:

[/u/p/m/nnn]$ nnn
ld-elf.so.1: Undefined symbol "rl_change_environment" referenced from COPY relocation in /usr/local/bin/nnn

any ideas how to resolve this ? I'm not doing anything fancy with jails. I realize this is a different tool, but help is appreciated
Comment 21 Tobias Kortkamp freebsd_committer freebsd_triage 2020-05-09 03:38:02 UTC
This is not a support forum.  Please provide the relevant information
like comment #11.  I bet you could figure it out for yourself after
that.
Comment 22 nefar 2020-05-09 03:51:30 UTC
(In reply to Tobias Kortkamp from comment #21)
/lib]$ ldconfig -r | grep libreadline
       22:-lreadline.8 => /lib/libreadline.so.8
       479:-lreadline.8 => /usr/local/lib/libreadline.so.8
       582:-lreadline.6 => /usr/local/lib/compat/libreadline.so.6
       695:-lreadline.7 => /usr/local/lib/compat/libreadline.so.7


maybe I should delete this /lib/readline.so.8 since you mention it shouldn't exist. looks like it's really old. no idea where it came from
Comment 23 nefar 2020-05-09 03:52:21 UTC
I moved it to another folder, and it seems to have fixed it. Should I even have a /lib folder?