Bug 270482

Comment 1 Romain Tartière 2023-03-27 21:42:18 UTC

Hi,

There where a bunch of updates and fixes.  As you saw, the rubygem-concurrent dependency was updated to use rubygem-concurrent-ruby11-1.1.10 (so the version that still has the RubyThreadLocalVar class) because the comment in https://github.com/puppetlabs/puppet/commit/9182bc3dd2576f409a6d01fb5c08d392670e90a2#diff-4587e027e140fb910d00cf675bb11cb7d99c4c22d78d5a702becc3c78ffc117c suggest that bad things can happen:

# We want to use the pure Ruby implementation even on JRuby. If we use the Java
# implementation of ThreadLocal, we end up leaking references to JRuby instances
# and preventing them from being garbage collected.

I see nothing obviously wrong in your setup.  Maybe you can `pkg autoremove` to remove rubygem-concurrent-ruby-1.2.1 and rubygem-fast_gettext1-1.8.0 which where previous dependencies of Puppet and where fixed, but I don't think this will change much.

Have you tuned the config from PuppetServer?  In /usr/local/etc/puppetserver/conf.d/, check that each config file and the associated .sample file are in sync: I have to update puppetserver.conf by hand when I change the version of Ruby because I have some custom settings there.

If it still does not help, can you confirm that the output of `puppetserver gem list` list the expected versions of the dependencies?

`pkg autoremove` was the solution for me, now puppetserver starts properly. Thanks for the hint!

I forgot to check the gem list before running autoremove, but since the error disappeared I assume it picked up the wrong version of concurrent. I'm leaving the bug open in case you want to investigate more, but my problem is solved so feel free to close it.

In case it's still useful information the only change in puppetserver conf.d I've done is to add an additional file path in auth.conf.

Still an issue with both puppet 6 and 7. While puppet itself correctly depends on concurrent-ruby11, rails61 installs concurrent-ruby 1.2.2 next to it. Having both versions of the gem installed causes puppet to fail again. I use puppet to maintain our ruby on rails servers, so I'm in a bit of pickle right now.

Odd. Puppet has the correct requirements:

# gem uninstall concurrent-ruby -v 1.1.10

You have requested to uninstall the gem:
        concurrent-ruby-1.1.10

puppet-7.23.0 depends on concurrent-ruby (~> 1.0, < 1.2.0)
If you remove this gem, these dependencies will not be met.
Continue with Uninstall? [yN]  n

Yet it appears to pick the 1.2.2 version of the gem causing the error to occur. 

# gem list concurrent-ruby

*** LOCAL GEMS ***

concurrent-ruby (1.2.2, 1.1.10)

# puppet agent -t
Traceback (most recent call last):
        10: from /usr/local/bin/puppet:4:in `<main>'
         9: from /usr/local/lib/ruby/site_ruby/2.7/rubygems/core_ext/kernel_require.rb:88:in `require'
         8: from /usr/local/lib/ruby/site_ruby/2.7/rubygems/core_ext/kernel_require.rb:88:in `require'
         7: from /usr/local/lib/ruby/site_ruby/2.7/puppet/util/command_line.rb:12:in `<top (required)>'
         6: from /usr/local/lib/ruby/site_ruby/2.7/puppet/util/command_line.rb:12:in `require_relative'
         5: from /usr/local/lib/ruby/site_ruby/2.7/puppet.rb:41:in `<top (required)>'
         4: from /usr/local/lib/ruby/site_ruby/2.7/puppet.rb:44:in `<module:Puppet>'
         3: from /usr/local/lib/ruby/site_ruby/2.7/puppet.rb:44:in `require_relative'
         2: from /usr/local/lib/ruby/site_ruby/2.7/puppet/context.rb:1:in `<top (required)>'
         1: from /usr/local/lib/ruby/site_ruby/2.7/puppet/context.rb:1:in `require_relative'
/usr/local/lib/ruby/site_ruby/2.7/puppet/thread_local.rb:6:in `<top (required)>': uninitialized constant Concurrent::RubyThreadLocalVar (NameError)
Did you mean?  Concurrent::RubyThreadPoolExecutor

(Ruby default version is still on 2.7 for us, can't upgrade to 3.0 or 3.1 yet)

# pkg info -r rubygem-concurrent-ruby
rubygem-concurrent-ruby-1.2.2:
        rubygem-activesupport61-6.1.7.2
        rubygem-tzinfo-2.0.6
        rubygem-i18n-1.12.0,2
        rubygem-sprockets3-3.7.2

# pkg info -r rubygem-concurrent-ruby11
rubygem-concurrent-ruby11-1.1.10:
        puppet7-7.23.0_2

I think I fixed the issue by modifying two files in puppet itself. Both files have a "require 'concurrent'" that pulls in the 'wrong' version.

/usr/local/lib/ruby/site_ruby/2.7/puppet/thread_local.rb:

gem 'concurrent-ruby', '< 1.2.0'
require 'concurrent'

And /usr/local/lib/ruby/site_ruby/2.7/puppet/settings.rb

gem 'concurrent-ruby', '< 1.2.0'
require 'concurrent'

In both those files I added "gem 'concurrent-ruby', '< 1.2.0'" before the require. That forces ruby to load the 1.1.x version.

Comment 6 Romain Tartière 2023-03-30 05:30:09 UTC

sirdice I can't reproduce the issue with Puppet itself, only with PuppetServer which use Puppet.  Is it also what you are reporting (I see no reference to puppetserver in your messages)?

PuppetServer bundle JRuby and I guess the way it loads code is different from Ruby when we run it.

The fact that other ports depend on rubygem-concurrent-ruby currently makes them unusable on the same host as PuppetServer.  Note that this previous patch can be a workaround:
https://cgit.freebsd.org/ports/commit/sysutils/puppet7?id=9b4522cb7997070282339d841af44ee8e57ae798

Beware however that this will probably lead to memory leaks (reason why it was reverted):
https://github.com/puppetlabs/puppet/blob/main/lib/puppet/thread_local.rb

As soon as a solution "approved" by upstream is found, I will integrate it, but for now I prefer to stick to what is known working rather than on a workaround that is likely to cause trouble for large sites.

Thanks!

>sirdice I can't reproduce the issue with Puppet itself, only with PuppetServer >which use Puppet.  Is it also what you are reporting (I see no reference to ?puppetserver in your messages)?

Can't reproduce? Clean system, only puppet7 installed (from FreeBSD latest):
root@fbsd-test:~ # puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for fbsd-test.dicelan.home
Info: Applying configuration version '1680270824'
Notice: Applied catalog in 1.55 seconds

Works. No issue. 

root@fbsd-test:~ # pkg install rubygem-concurrent-ruby
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        rubygem-concurrent-ruby: 1.2.2 [FreeBSD]

Number of packages to be installed: 1

236 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching rubygem-concurrent-ruby-1.2.2.pkg: 100%  236 KiB 241.4kB/s    00:01
Checking integrity... done (0 conflicting)
[1/1] Installing rubygem-concurrent-ruby-1.2.2...
[1/1] Extracting rubygem-concurrent-ruby-1.2.2: 100%
root@fbsd-test:~ # puppet agent -t
/usr/local/lib/ruby/site_ruby/3.1/puppet/thread_local.rb:6:in `<top (required)>': uninitialized constant Concurrent::RubyThreadLocalVar (NameError)

class Puppet::ThreadLocal < Concurrent::RubyThreadLocalVar
                                      ^^^^^^^^^^^^^^^^^^^^
Did you mean?  Concurrent::RubyThreadPoolExecutor
        from /usr/local/lib/ruby/site_ruby/3.1/puppet/context.rb:1:in `require_relative'
        from /usr/local/lib/ruby/site_ruby/3.1/puppet/context.rb:1:in `<top (required)>'
        from /usr/local/lib/ruby/site_ruby/3.1/puppet.rb:44:in `require_relative'
        from /usr/local/lib/ruby/site_ruby/3.1/puppet.rb:44:in `<module:Puppet>'
        from /usr/local/lib/ruby/site_ruby/3.1/puppet.rb:41:in `<top (required)>'
        from /usr/local/lib/ruby/site_ruby/3.1/puppet/util/command_line.rb:12:in `require_relative'
        from /usr/local/lib/ruby/site_ruby/3.1/puppet/util/command_line.rb:12:in `<top (required)>'
        from <internal:/usr/local/lib/ruby/site_ruby/3.1/rubygems/core_ext/kernel_require.rb>:88:in `require'
        from <internal:/usr/local/lib/ruby/site_ruby/3.1/rubygems/core_ext/kernel_require.rb>:88:in `require'
        from /usr/local/bin/puppet:4:in `<main>'
root@fbsd-test:~ #

So with the puppet agent it will cause issues if concurrent-ruby 1.2.x is installed. Which will happen if you install rails61 for example. This means our puppet agents are all broken on our Ruby on Rails servers.  

My fix by adding the 'gem "concurrent-ruby", "< 1.2.0"' line does work for puppet7 itself. But you're right, it doesn't seem to work for puppetserver. It would still fail if concurrent-ruby 1.2.x is installed. 

>The fact that other ports depend on rubygem-concurrent-ruby currently makes >them unusable on the same host as PuppetServer.

Yes, that seems to be the case. For our puppetserver that's not a big issue, I don't have anything else installed there that would pull in rubygem-concurrent-ruby. 

> As soon as a solution "approved" by upstream is found, I will integrate it

I agree this needs to be properly addressed upstream. But for the time being my 'quick and dirty' fix does fix the agents.

Dammit, mixed up two different solutions I tried. 

One solution was to add `gem 'concurrent-ruby', '< 1.2.0'` to /usr/local/bin/puppet. That fixes the agent but does nothing for puppetserver.

Modifying puppet/settings.rb and puppet/thread_local.rb fixes both puppetserver and puppet agent. 

> but for now I prefer to stick to what is known working rather than on a 
> workaround that is likely to cause trouble for large sites.

I don't see the problem here. All I'm doing is adding some restrictions to (J)Ruby to not load the latest version of a gem, thus forcing it to load 1.1.x instead of 1.2.x of concurrent-ruby.

Comment 9 Romain Tartière 2023-04-08 20:47:46 UTC

This will be part of the next update of the port (I am finishing work on this ATM).

Thanks!

Just for reference, the problem hits the "client side" (sysutils/puppet7) in the same way.

I did "pkg upgrade" yesterday, which gave me ruby 3.1, ruby31-gems and brought along "rubygem-concurrent-ruby11-1.1.10" as new package.

After the upgrade, the puppet-agent did not start anymore, with the same error message

/usr/local/lib/ruby/site_ruby/3.1/puppet/thread_local.rb:6:in `<top (required)>': uninitialized constant Concurrent::RubyThreadLocalVar (NameError)

class Puppet::ThreadLocal < Concurrent::RubyThreadLocalVar
                                      ^^^^^^^^^^^^^^^^^^^^

doing "pkg autoremove" removed "rubygem-concurrent-ruby-1.2.2", and after that the agent started normally again (so thanks for that tip).

Comment 11 commit-hook 2023-04-10 09:13:42 UTC

A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=91e06061821d106bf0195c6bd08856f8f92dbd39

commit 91e06061821d106bf0195c6bd08856f8f92dbd39
Author:     Romain Tartière <romain@FreeBSD.org>
AuthorDate: 2023-04-08 21:07:22 +0000
Commit:     Romain Tartière <romain@FreeBSD.org>
CommitDate: 2023-04-10 09:10:54 +0000

    sysutils/puppet7: Update to 7.24.0

    Release notes:
    https://www.puppet.com/docs/puppet/7/release_notes_puppet.html#release_notes_puppet_x-7-24-0

    While here, cope with incompatible concurrent-ruby version installed
    beside the needed one [1].

    With hat:       puppet
    PR:             270482 [1]
    Reported by:    freebsd@wb9.se [1]

 sysutils/puppet7/Makefile                          |  23 ++-
 sysutils/puppet7/distinfo                          |  46 +++---
 ...atch-lib_puppet_provider_service_daemontools.rb |   6 +-
 .../files/patch-lib_puppet_settings.rb (new)       |  10 ++
 .../files/patch-lib_puppet_thread__local.rb (new)  |   7 +
 sysutils/puppet7/pkg-plist                         | 161 ++++++++-------------
 6 files changed, 114 insertions(+), 139 deletions(-)

Comment 12 Romain Tartière 2023-04-10 09:18:23 UTC

Should be fixed now. Thanks!