Bug 230238 - sysutils/facter: encdb.so: Undefined symbol "rb_encdb_declare"
Summary: sysutils/facter: encdb.so: Undefined symbol "rb_encdb_declare"
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Some People
Assignee: Dima Panov
URL:
Keywords: regression
: 231652 (view as bug list)
Depends on:
Blocks: 231361
  Show dependency treegraph
 
Reported: 2018-07-31 12:54 UTC by Charles
Modified: 2018-10-12 12:49 UTC (History)
11 users (show)

See Also:
koobs: merge-quarterly?


Attachments
Full proposed change (2.04 KB, patch)
2018-09-20 09:04 UTC, Romain Tartière
no flags Details | Diff
Patch v2 - Fixes regressions (2.01 KB, patch)
2018-09-24 11:03 UTC, Kubilay Kocak
koobs: maintainer-approval? (ruby)
koobs: maintainer-approval? (ports-secteam)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Charles 2018-07-31 12:54:13 UTC
Fresh install of FreeBSD 11.2-RELEASE, install puppet5 from packages, run facter, get this error:

/usr/local/lib/ruby/2.4/amd64-freebsd11/enc/encdb.so: Undefined symbol "rb_encdb_declare"

trying to use puppet on AWS.  facter 2.5.x thinks the new c5 instance types are physical, so I thought I'd try puppet5 which comes with facter 3.11.x... 

I also tried a fresh install from the aws AMI, as well as the freebsd 11.2 vmdk.

This appears to be broken on FreeBSD 11.1 vmdk also.

From a fresh install, steps to recreate:

pkg install puppet5
facter
Comment 1 Romain Tartière freebsd_committer 2018-07-31 14:43:53 UTC
Hi Charles,

This seems to be a Ruby issue.  I had not noticed the issue so far, but I do have the same result as you when running facter without arguments.

As a workaround (depending on your usage) `facter --no-ruby` returns the expected facts.

Reverting the last commit to lang/ruby24 unbreaks facter:
https://svnweb.freebsd.org/ports?view=revision&revision=475483
Comment 2 albert.shih 2018-09-06 14:10:48 UTC
Hi every one

Any progress about this issue ?

If I ask it's because the depends on ticket

  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229898

are now close with FIXED tag.

So I'm wondering if anyone still working on this issue...

Regards
Comment 3 Matt LaChance 2018-09-18 13:07:59 UTC
(In reply to Romain Tartière from comment #1)

I tried doing what you suggested ("Reverting the last commit to lang/ruby24 unbreaks facter"), but I couldn't get it to work. What I tried exactly was:

    # cd /usr/ports/lang/ruby24
    # make deinstall
    # portdowngrade lang/ruby24 475483
    # make install clean

It's more than possible that I just don't know what I'm doing.

I also found a lot of posts in 2012 about this same error, except it was a make error in ruby 1.9, and it was fixable by running ./configure --disable-rdoc. I figured it was worth a try, but disabling rdoc in make config (and then rebuilding/reinstalling) didn't solve this for me either.

Has anybody managed to get this working yet? (Or does anybody know what I'm doing wrong in my attempt to revert ruby24?)

Thanks.
Comment 4 Romain Tartière freebsd_committer 2018-09-18 16:36:15 UTC
H Matt, I guess you reverted to the commit that introduced the breakage: 475483+ is broken.

You can either revert to a previous commit of the lang/ruby24 port or remove the 17 last lines of lang/ruby24/files/patch-configure.in before building.  I just gave it a try and it is enough to fix the problem.

Unfortunately, this ruby problem is really out of what I am used to in respect to Ruby, and I have no clue on how to fix this.  Adding fluffy@ in CC since he told he had some idea in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229898#c7
Comment 5 Matt LaChance 2018-09-18 17:45:21 UTC
Thank you Romain; I'll confirm that that does fix it for me as well. Now to see if I can figure out why this is a problem with facter but apparently not with anything else that runs ruby. I'll report back if I find anything.
Comment 6 Romain Tartière freebsd_committer 2018-09-18 17:48:20 UTC
In case it helps someone digging in this, sysutils/leatherman or sysutils/cpp-hocon (maybe both, can't recall) have a test suite that fail in a similar fashion when using the raw lang/ruby24 port.

cd sysutils/leatherman && make test
Comment 7 Dima Panov freebsd_committer 2018-09-19 15:08:01 UTC
(In reply to Romain Tartière from comment #4)

Looks like puppet and family (and only puppet for now) have led to overlinking via DLD_FLAGS, but no more ports were reported as failed yet.

Try this: https://people.freebsd.org/~fluffy/patches/ruby.diff

Patch partially undo the initial commit with "DLD_FLAGS" pollution part. It was initially got from ruby23, but recent ruby releases have some differences in runtime linking, which was not hit by exp-run :(

The first part of r475483 is unconditionally required to pass linker arguments to modern build systems such as ninja in the right way.
Comment 8 Romain Tartière freebsd_committer 2018-09-19 15:42:03 UTC
(In reply to Dima Panov from comment #7)
> Try this: https://people.freebsd.org/~fluffy/patches/ruby.diff

I have just applied this patch and build/installed lang/ruby24, and I confirm that I can't reproduce the problem: everything seems to be working as expected.

Because I really don't understand what this patch is supposed to do, the facts: building with an without the patch is always fine: only runtime happen to produce errors without your patch.  So maybe it's not enough to do an exp-run if the built program/libraries are never used in some way?  That might explain some undetected failures.  Running `make test` on these ports fails without the above patch:
  - devel/leatherman
  - sysutils/facter
Comment 9 albert.shih 2018-09-19 15:45:06 UTC
(In reply to Dima Panov from comment #7)

Hi

So I try the patch send by Dima. 

It's working (at leat with ruby24, I don't get time to try ruby25 because I need to rebuild everything depending on ruby). The facter run without any error

For what "comment 7" The devel/leatherman still don't pass the test, but devel/cpp-hocon pass the test.

Here the error with leatherman

[root@io leatherman]# make test
===>  Testing for leatherman-1.4.2_1
[0/1] cd /usr/ports/devel/leatherman/work/.build && /usr/local/bin/ctest --force-new-ctest-process
Test project /usr/ports/devel/leatherman/work/.build
    Start 1: leatherman tests
1/1 Test #1: leatherman tests .................***Failed    5.42 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   5.42 sec

The following tests FAILED:
          1 - leatherman tests (Failed)
Errors while running CTest
FAILED: CMakeFiles/test.util 
cd /usr/ports/devel/leatherman/work/.build && /usr/local/bin/ctest --force-new-ctest-process
ninja: build stopped: subcommand failed.
*** Error code 1

Stop.
make: stopped in /usr/ports/devel/leatherman
Comment 10 Romain Tartière freebsd_committer 2018-09-19 15:52:40 UTC
(In reply to albert.shih from comment #9)
> For what "comment 7" The devel/leatherman still don't pass the test, but                                                                                 
> devel/cpp-hocon pass the test.

Hehe, there is also some kind of bug in the testing framework of leatherman…  I have a local patch to workaround this (that I plan to commit with the update of leatherman):

Index: files/patch-json__container_tests_json__container__test.cc
===================================================================
--- files/patch-json__container_tests_json__container__test.cc  (nonexistent)
+++ files/patch-json__container_tests_json__container__test.cc  (copie de travail)
@@ -0,0 +1,16 @@
+--- json_container/tests/json_container_test.cc.orig
++++ json_container/tests/json_container_test.cc
+@@ -22,11 +22,13 @@
+     std::string json_value {};
+ 
+     SECTION("it should instantiate by passing any JSON value") {
++        json_value = "{}";
+         SECTION("object") {
+             json_value = JSON;
+         }
+ 
+         SECTION("array") {
++            json_value = "[]";
+             SECTION("of numbers") {
+                 json_value = "[1, 2, 3]";
+             }
Comment 11 Dima Panov freebsd_committer 2018-09-19 22:55:53 UTC
(In reply to Romain Tartière from comment #10)
Please ship both patches to the ports, if you wish.
I'm still busy IRL and will be mostly offline for a week or so, sorry.
Comment 12 Dima Panov freebsd_committer 2018-09-20 04:14:27 UTC
(In reply to Romain Tartière from comment #8)

>So maybe it's not enough to do an exp-run if the built program/libraries are never used in some way?  That might explain some undetected failures.

Sure, exp-run is showing only potentially safe results. In our case, no one gem was failed even at runtime since r475483, and only leatherman/facter got a dynamic linking error at real run.
Forcing 'make tests' (and ensure that test suite is present in every port) will dramatically increase build/verify time but allow to catch most runtime errors automagically.
Comment 13 Romain Tartière freebsd_committer 2018-09-20 06:36:02 UTC
(In reply to Dima Panov from comment #11)
> Please ship both patches to the ports, if you wish.

Great!  Can you please validate this commit message for lang/ruby{24,25}:

-------------------------------------- 8< --------------------------------------
Revert the "DLD_FLAGS" pollution part of r475483. It was initially got from ruby23, but recent ruby releases have some differences in runtime linking, which was not hit by exp-run, and caused trouble with some ports (e.g. devel/leatherman, sysutils/facter).

PR:           230238
Submitted by: fluffy
Reported by:  freebsd@ptty.com
Approved by:  fluffy
-------------------------------------- 8< --------------------------------------
Comment 14 Dima Panov freebsd_committer 2018-09-20 08:49:11 UTC
(In reply to Romain Tartière from comment #13)
Sure. Ship it!
Comment 15 Romain Tartière freebsd_committer 2018-09-20 09:04:48 UTC
Created attachment 197263 [details]
Full proposed change

Sorry to bother again, I am 99.9% sure I have to bump both PORTREVISIONS in bsd.ruby.mk but since it's an area I do not know and a lot of user depend on this, I prefer to be 100% sure that this looks good.  I attached the output of svn diff.

Thanks for your patience :-)
Comment 16 Romain Tartière freebsd_committer 2018-09-20 09:10:58 UTC
/me is stupid, I was looking at the wrong changeset and am now 100% confident with the patch.

Committing!
Comment 17 commit-hook freebsd_committer 2018-09-20 09:12:24 UTC
A commit references this bug:

Author: romain
Date: Thu Sep 20 09:12:06 UTC 2018
New revision: 480158
URL: https://svnweb.freebsd.org/changeset/ports/480158

Log:
  Revert the "DLD_FLAGS" pollution part of r475483.

  It was initially got from ruby23, but recent ruby releases have some
  differences in runtime linking, which was not hit by exp-run, and caused
  trouble with some ports (e.g. devel/leatherman, sysutils/facter).

  PR:		230238
  Submitted by:	fluffy
  Reported by:	freebsd@ptty.com
  Approved by:	fluffy

Changes:
  head/Mk/bsd.ruby.mk
  head/lang/ruby24/files/patch-configure.in
  head/lang/ruby25/files/patch-configure.ac
Comment 18 Romain Tartière freebsd_committer 2018-09-20 09:13:08 UTC
Thanks you all for fixing this!
Comment 19 Charles 2018-09-24 00:58:01 UTC
Awesome, thanks everyone!
Comment 20 Kubilay Kocak freebsd_committer freebsd_triage 2018-09-24 02:13:14 UTC
Remove spurious 'depends on'. It was the originating issue (for this regression). Move instead to See Also.

Assign to committer that resolved.

@Romain The original commit was also MFH'd in ports r475488 (in bug 229898). Does the reversion (ports r480158) also need to be MFH'd ?
Comment 21 Walter Schwarzenfeld freebsd_triage 2018-09-24 05:49:46 UTC
It is not solved, sie Bug #PR 231652..
Comment 22 Walter Schwarzenfeld freebsd_triage 2018-09-24 05:50:27 UTC
Bug #PR 231652..
Comment 23 Walter Schwarzenfeld freebsd_triage 2018-09-24 05:55:36 UTC
Sorry I was in the wrong PR.
Comment 24 Walter Schwarzenfeld freebsd_triage 2018-09-24 05:58:30 UTC
Sorry, seems I am a little "chaotic" today, it was right, see bug #231652.
Comment 25 Dima Panov freebsd_committer 2018-09-24 06:04:53 UTC
(In reply to w.schwarzenfeld from comment #24)

False positive caused by the regression in ruby dynamic linking. 
You should rebuild/reinstall all ruby-related packages instead of reverting latest ruby patch
Comment 26 Walter Schwarzenfeld freebsd_triage 2018-09-24 06:12:33 UTC
Reinstall does not help.
Comment 27 Kubilay Kocak freebsd_committer freebsd_triage 2018-09-24 07:19:27 UTC
*** Bug 231652 has been marked as a duplicate of this bug. ***
Comment 28 Dima Panov freebsd_committer 2018-09-24 10:58:28 UTC
Folks, stage 2 on the way.

Initial direction was fine but the first attempt failed at dynamic linking.
Now it links almost fine but drives 'pkg check -d' crazy alongside with dynamic shlib loader for libruby24.so.

Got a deeper investigation with linker manual and ruby internal makefiles, was able to track a right way AFAIT. The initial mistake was issued by linker environment variable 
names (LDFLAGS/DLDFLAGS) reuse across private and external build processes for libruby without any demarcation for different stages.

https://people.freebsd.org/~fluffy/patches/ruby-stage2.diff

With stage two we have original ninja-related issues resolved, correct private args to linker while building libruby itself (and no more external LDFLAGS pollution), full symbols table export, pet 'pkg check -d', no issues with weechat/vim/facter at least on my 3 different machines.

Sorry to all for the inconvenience while attempt to track hidden linker issues, this case is a good example for situations which can't be automatically caught by exp-run at all.

Romain@, MFh process can be done after we completely confirm all described issues closed for all pals in this conversation :)
Comment 29 Kubilay Kocak freebsd_committer freebsd_triage 2018-09-24 11:03:23 UTC
Created attachment 197427 [details]
Patch v2 - Fixes regressions

Attach Dima's patch from comment 28
Comment 30 Romain Tartière freebsd_committer 2018-09-24 11:07:45 UTC
Dima, I just re-built Ruby with your patch and confirm that facter is working as expected.

Thanks!
Comment 31 Dima Panov freebsd_committer 2018-09-24 11:55:53 UTC
(In reply to Romain Tartière from comment #30)

Good to have a confirmation :)

Just to be clear with real tests, I've recompiled all ruby-related locally, not only ruby24 itself. 
All ruby-required software now links correctly with libruby24.so.24 (without v2 patch all links to non-versioned libruby24.so which drives 'pkg check' mad) as it should be. 
Weechat switched to use ninja build on my box, and it runs fine.
Comment 32 Walter Schwarzenfeld freebsd_triage 2018-09-24 12:18:09 UTC
Confirm, this is working, thanks.
Comment 33 Kenji Rikitake 2018-09-24 13:35:41 UTC
Patch v2 by koobs fixed the building of

* www/rubygem-hpricot
* textproc/rubygem-rdiscount

Which had caused error of

rubygem-hpricot is missing a required shared library: libruby24.so
rubygem-rdiscount is missing a required shared library: libruby24.so

generated by "pkg check -v -a"
on FreeBSD amd64 11.2-STABLE r338678.

Thanks!
Comment 34 commit-hook freebsd_committer 2018-09-25 13:11:29 UTC
A commit references this bug:

Author: romain
Date: Tue Sep 25 13:11:15 UTC 2018
New revision: 480661
URL: https://svnweb.freebsd.org/changeset/ports/480661

Log:
  Address more linker issues

  This resolves a ninja-related issues, corrects passing private args to linker
  while building libruby itself (without external LDFLAGS), exports the full
  symbols table, and pet 'pkg check -d'. Issues with weechat, vim, facter,
  rubygem-hpricot and rubygem-rdiscount should be fixed.  See [1] for long story
  of hunting and fixing these issues.

  Bump PORTREVISIONS accordingly.

  PR:		230238 [1], 231361
  Submitted by:	fuffy
  Approved by:	fuffy

Changes:
  head/Mk/bsd.ruby.mk
  head/lang/ruby24/files/patch-configure.in
  head/lang/ruby25/files/patch-configure.ac
Comment 35 Romain Tartière freebsd_committer 2018-09-25 13:19:28 UTC
Thank you for your help in fixing this issue!
Comment 36 Kubilay Kocak freebsd_committer freebsd_triage 2018-09-26 02:36:30 UTC
@Romain Shouldn't this (ports r480661) be merged to quarterly?
Comment 37 Romain Tartière freebsd_committer 2018-09-26 06:42:01 UTC
(In reply to Kubilay Kocak from comment #36)
> @Romain Shouldn't this (ports r480661) be merged to quarterly?

Probably?  Dima (@fluffy) did all the work and I just tested and committed his patches, so I am assigning the issue to him because I feel clueless for too long time with this issue :-)
Comment 38 Dima Panov freebsd_committer 2018-10-12 12:49:40 UTC
2018Q4 was already hit the ports tree with latest commits, close PR finally.