Created attachment 241144 [details] startup log from puppetserver At ~2023-03-25T23:54:22+01:00 my puppetserver automatically upgraded puppet7 and also installed rubygem-concurrent-ruby11-1.1.10 (new dependency?) and rebooted. After this puppetserver does not start any more. See attached log for details, but I think the most relevant part is "uninitialized constant Concurrent::RubyThreadLocalVar". Looking around I found this issue in ruby-concurrent which is most likely relevant: https://github.com/ruby-concurrency/concurrent-ruby/issues/986 I assume this is why puppet7 was updated, but it seems to affect puppetserver as well? In case it's relevant it should be noted that the packages are built using poudriere. puppet and ruby packages installed: # pkg info | grep -E '(puppet|ruby)' puppet7-7.23.0_2 Configuration management framework written in Ruby puppetdb-terminus7-7.12.1 PuppetDB storeconfigs backend terminus module puppetdb7-7.12.1 PuppetDB storeconfigs backend puppetserver7-7.9.5 Puppet Server running in the JVM ruby-3.1.3_2,1 Object-oriented interpreted scripting language ruby31-gems-3.4.7 Package management framework for the Ruby language rubygem-concurrent-ruby-1.2.1 Modern concurrency tools for Ruby rubygem-concurrent-ruby11-1.1.10 Modern concurrency tools for Ruby rubygem-deep_merge-1.2.2 Recursive hash merger rubygem-facter-4.2.14 Cross-platform Ruby library for retrieving facts from OS rubygem-fast_gettext-2.3.0 Fast, memory-efficient and threadsafe GetText for Ruby rubygem-fast_gettext1-1.8.0 Fast, memory-efficient and threadsafe GetText for Ruby rubygem-ffi-1.15.5_1 Extension for dynamic libraries and binding functions rubygem-hiera-3.12.0 Pluggable data store for hierarchical data rubygem-hiera-eyaml-3.3.0_1 Backend for Hiera that provides encryption/decryption for properties rubygem-highline-2.1.0 High-level IO library for command-line interfaces rubygem-hocon-1.3.1 Ruby port of the Typesafe Config library rubygem-json_pure-2.6.3 Parse JSON texts and generate them from ruby data structures in Ruby rubygem-locale-2.1.3 Pure ruby library to support locales rubygem-multi_json-1.15.0 Ruby library provide swappable JSON backends rubygem-optimist-3.0.1 Command-line option parser for Ruby rubygem-puppet-resource_api-1.8.4 Simple way to write new native resources for puppet rubygem-puppetserver-ca-2.4.0 Ruby CLI tool to interact with the Puppet Server Certificate Authority rubygem-rexml-3.2.5 XML toolkit for Ruby rubygem-ruby-augeas-0.5.0_4 Ruby bindings for Augeas rubygem-scanf-1.0.0 Ruby implementation of the C function scanf(3) rubygem-semantic_puppet-1.0.4 Library for working with Semantic Versions and module dependencies rubygem-sys-filesystem-1.4.3 Ruby interface for getting filesystem information rubygem-thor-1.2.1 Scripting framework that replaces rake, sake, and rubigen
Hi, There where a bunch of updates and fixes. As you saw, the rubygem-concurrent dependency was updated to use rubygem-concurrent-ruby11-1.1.10 (so the version that still has the RubyThreadLocalVar class) because the comment in https://github.com/puppetlabs/puppet/commit/9182bc3dd2576f409a6d01fb5c08d392670e90a2#diff-4587e027e140fb910d00cf675bb11cb7d99c4c22d78d5a702becc3c78ffc117c suggest that bad things can happen: # We want to use the pure Ruby implementation even on JRuby. If we use the Java # implementation of ThreadLocal, we end up leaking references to JRuby instances # and preventing them from being garbage collected. I see nothing obviously wrong in your setup. Maybe you can `pkg autoremove` to remove rubygem-concurrent-ruby-1.2.1 and rubygem-fast_gettext1-1.8.0 which where previous dependencies of Puppet and where fixed, but I don't think this will change much. Have you tuned the config from PuppetServer? In /usr/local/etc/puppetserver/conf.d/, check that each config file and the associated .sample file are in sync: I have to update puppetserver.conf by hand when I change the version of Ruby because I have some custom settings there. If it still does not help, can you confirm that the output of `puppetserver gem list` list the expected versions of the dependencies?
`pkg autoremove` was the solution for me, now puppetserver starts properly. Thanks for the hint! I forgot to check the gem list before running autoremove, but since the error disappeared I assume it picked up the wrong version of concurrent. I'm leaving the bug open in case you want to investigate more, but my problem is solved so feel free to close it. In case it's still useful information the only change in puppetserver conf.d I've done is to add an additional file path in auth.conf.
Still an issue with both puppet 6 and 7. While puppet itself correctly depends on concurrent-ruby11, rails61 installs concurrent-ruby 1.2.2 next to it. Having both versions of the gem installed causes puppet to fail again. I use puppet to maintain our ruby on rails servers, so I'm in a bit of pickle right now.
Odd. Puppet has the correct requirements: # gem uninstall concurrent-ruby -v 1.1.10 You have requested to uninstall the gem: concurrent-ruby-1.1.10 puppet-7.23.0 depends on concurrent-ruby (~> 1.0, < 1.2.0) If you remove this gem, these dependencies will not be met. Continue with Uninstall? [yN] n Yet it appears to pick the 1.2.2 version of the gem causing the error to occur. # gem list concurrent-ruby *** LOCAL GEMS *** concurrent-ruby (1.2.2, 1.1.10) # puppet agent -t Traceback (most recent call last): 10: from /usr/local/bin/puppet:4:in `<main>' 9: from /usr/local/lib/ruby/site_ruby/2.7/rubygems/core_ext/kernel_require.rb:88:in `require' 8: from /usr/local/lib/ruby/site_ruby/2.7/rubygems/core_ext/kernel_require.rb:88:in `require' 7: from /usr/local/lib/ruby/site_ruby/2.7/puppet/util/command_line.rb:12:in `<top (required)>' 6: from /usr/local/lib/ruby/site_ruby/2.7/puppet/util/command_line.rb:12:in `require_relative' 5: from /usr/local/lib/ruby/site_ruby/2.7/puppet.rb:41:in `<top (required)>' 4: from /usr/local/lib/ruby/site_ruby/2.7/puppet.rb:44:in `<module:Puppet>' 3: from /usr/local/lib/ruby/site_ruby/2.7/puppet.rb:44:in `require_relative' 2: from /usr/local/lib/ruby/site_ruby/2.7/puppet/context.rb:1:in `<top (required)>' 1: from /usr/local/lib/ruby/site_ruby/2.7/puppet/context.rb:1:in `require_relative' /usr/local/lib/ruby/site_ruby/2.7/puppet/thread_local.rb:6:in `<top (required)>': uninitialized constant Concurrent::RubyThreadLocalVar (NameError) Did you mean? Concurrent::RubyThreadPoolExecutor (Ruby default version is still on 2.7 for us, can't upgrade to 3.0 or 3.1 yet) # pkg info -r rubygem-concurrent-ruby rubygem-concurrent-ruby-1.2.2: rubygem-activesupport61-6.1.7.2 rubygem-tzinfo-2.0.6 rubygem-i18n-1.12.0,2 rubygem-sprockets3-3.7.2 # pkg info -r rubygem-concurrent-ruby11 rubygem-concurrent-ruby11-1.1.10: puppet7-7.23.0_2
I think I fixed the issue by modifying two files in puppet itself. Both files have a "require 'concurrent'" that pulls in the 'wrong' version. /usr/local/lib/ruby/site_ruby/2.7/puppet/thread_local.rb: gem 'concurrent-ruby', '< 1.2.0' require 'concurrent' And /usr/local/lib/ruby/site_ruby/2.7/puppet/settings.rb gem 'concurrent-ruby', '< 1.2.0' require 'concurrent' In both those files I added "gem 'concurrent-ruby', '< 1.2.0'" before the require. That forces ruby to load the 1.1.x version.
sirdice I can't reproduce the issue with Puppet itself, only with PuppetServer which use Puppet. Is it also what you are reporting (I see no reference to puppetserver in your messages)? PuppetServer bundle JRuby and I guess the way it loads code is different from Ruby when we run it. The fact that other ports depend on rubygem-concurrent-ruby currently makes them unusable on the same host as PuppetServer. Note that this previous patch can be a workaround: https://cgit.freebsd.org/ports/commit/sysutils/puppet7?id=9b4522cb7997070282339d841af44ee8e57ae798 Beware however that this will probably lead to memory leaks (reason why it was reverted): https://github.com/puppetlabs/puppet/blob/main/lib/puppet/thread_local.rb As soon as a solution "approved" by upstream is found, I will integrate it, but for now I prefer to stick to what is known working rather than on a workaround that is likely to cause trouble for large sites. Thanks!
>sirdice I can't reproduce the issue with Puppet itself, only with PuppetServer >which use Puppet. Is it also what you are reporting (I see no reference to ?puppetserver in your messages)? Can't reproduce? Clean system, only puppet7 installed (from FreeBSD latest): root@fbsd-test:~ # puppet agent -t Info: Using environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for fbsd-test.dicelan.home Info: Applying configuration version '1680270824' Notice: Applied catalog in 1.55 seconds Works. No issue. root@fbsd-test:~ # pkg install rubygem-concurrent-ruby Updating FreeBSD repository catalogue... FreeBSD repository is up to date. All repositories are up to date. The following 1 package(s) will be affected (of 0 checked): New packages to be INSTALLED: rubygem-concurrent-ruby: 1.2.2 [FreeBSD] Number of packages to be installed: 1 236 KiB to be downloaded. Proceed with this action? [y/N]: y [1/1] Fetching rubygem-concurrent-ruby-1.2.2.pkg: 100% 236 KiB 241.4kB/s 00:01 Checking integrity... done (0 conflicting) [1/1] Installing rubygem-concurrent-ruby-1.2.2... [1/1] Extracting rubygem-concurrent-ruby-1.2.2: 100% root@fbsd-test:~ # puppet agent -t /usr/local/lib/ruby/site_ruby/3.1/puppet/thread_local.rb:6:in `<top (required)>': uninitialized constant Concurrent::RubyThreadLocalVar (NameError) class Puppet::ThreadLocal < Concurrent::RubyThreadLocalVar ^^^^^^^^^^^^^^^^^^^^ Did you mean? Concurrent::RubyThreadPoolExecutor from /usr/local/lib/ruby/site_ruby/3.1/puppet/context.rb:1:in `require_relative' from /usr/local/lib/ruby/site_ruby/3.1/puppet/context.rb:1:in `<top (required)>' from /usr/local/lib/ruby/site_ruby/3.1/puppet.rb:44:in `require_relative' from /usr/local/lib/ruby/site_ruby/3.1/puppet.rb:44:in `<module:Puppet>' from /usr/local/lib/ruby/site_ruby/3.1/puppet.rb:41:in `<top (required)>' from /usr/local/lib/ruby/site_ruby/3.1/puppet/util/command_line.rb:12:in `require_relative' from /usr/local/lib/ruby/site_ruby/3.1/puppet/util/command_line.rb:12:in `<top (required)>' from <internal:/usr/local/lib/ruby/site_ruby/3.1/rubygems/core_ext/kernel_require.rb>:88:in `require' from <internal:/usr/local/lib/ruby/site_ruby/3.1/rubygems/core_ext/kernel_require.rb>:88:in `require' from /usr/local/bin/puppet:4:in `<main>' root@fbsd-test:~ # So with the puppet agent it will cause issues if concurrent-ruby 1.2.x is installed. Which will happen if you install rails61 for example. This means our puppet agents are all broken on our Ruby on Rails servers. My fix by adding the 'gem "concurrent-ruby", "< 1.2.0"' line does work for puppet7 itself. But you're right, it doesn't seem to work for puppetserver. It would still fail if concurrent-ruby 1.2.x is installed. >The fact that other ports depend on rubygem-concurrent-ruby currently makes >them unusable on the same host as PuppetServer. Yes, that seems to be the case. For our puppetserver that's not a big issue, I don't have anything else installed there that would pull in rubygem-concurrent-ruby. > As soon as a solution "approved" by upstream is found, I will integrate it I agree this needs to be properly addressed upstream. But for the time being my 'quick and dirty' fix does fix the agents.
Dammit, mixed up two different solutions I tried. One solution was to add `gem 'concurrent-ruby', '< 1.2.0'` to /usr/local/bin/puppet. That fixes the agent but does nothing for puppetserver. Modifying puppet/settings.rb and puppet/thread_local.rb fixes both puppetserver and puppet agent. > but for now I prefer to stick to what is known working rather than on a > workaround that is likely to cause trouble for large sites. I don't see the problem here. All I'm doing is adding some restrictions to (J)Ruby to not load the latest version of a gem, thus forcing it to load 1.1.x instead of 1.2.x of concurrent-ruby.
This will be part of the next update of the port (I am finishing work on this ATM). Thanks!
Just for reference, the problem hits the "client side" (sysutils/puppet7) in the same way. I did "pkg upgrade" yesterday, which gave me ruby 3.1, ruby31-gems and brought along "rubygem-concurrent-ruby11-1.1.10" as new package. After the upgrade, the puppet-agent did not start anymore, with the same error message /usr/local/lib/ruby/site_ruby/3.1/puppet/thread_local.rb:6:in `<top (required)>': uninitialized constant Concurrent::RubyThreadLocalVar (NameError) class Puppet::ThreadLocal < Concurrent::RubyThreadLocalVar ^^^^^^^^^^^^^^^^^^^^ doing "pkg autoremove" removed "rubygem-concurrent-ruby-1.2.2", and after that the agent started normally again (so thanks for that tip).
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=91e06061821d106bf0195c6bd08856f8f92dbd39 commit 91e06061821d106bf0195c6bd08856f8f92dbd39 Author: Romain Tartière <romain@FreeBSD.org> AuthorDate: 2023-04-08 21:07:22 +0000 Commit: Romain Tartière <romain@FreeBSD.org> CommitDate: 2023-04-10 09:10:54 +0000 sysutils/puppet7: Update to 7.24.0 Release notes: https://www.puppet.com/docs/puppet/7/release_notes_puppet.html#release_notes_puppet_x-7-24-0 While here, cope with incompatible concurrent-ruby version installed beside the needed one [1]. With hat: puppet PR: 270482 [1] Reported by: freebsd@wb9.se [1] sysutils/puppet7/Makefile | 23 ++- sysutils/puppet7/distinfo | 46 +++--- ...atch-lib_puppet_provider_service_daemontools.rb | 6 +- .../files/patch-lib_puppet_settings.rb (new) | 10 ++ .../files/patch-lib_puppet_thread__local.rb (new) | 7 + sysutils/puppet7/pkg-plist | 161 ++++++++------------- 6 files changed, 114 insertions(+), 139 deletions(-)
Should be fixed now. Thanks!