Bug 224623 - sysutils/puppet4: Ruby 2.3.6 Update breaks puppetmaster rc.d script
Summary: sysutils/puppet4: Ruby 2.3.6 Update breaks puppetmaster rc.d script
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Some People
Assignee: FreeBSD Puppet Team
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-12-27 23:28 UTC by Rainbow
Modified: 2018-02-10 08:09 UTC (History)
4 users (show)

See Also:
bugzilla: maintainer-feedback? (puppet)


Attachments
sock.accept patch (409 bytes, patch)
2017-12-30 15:18 UTC, Rainbow
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Rainbow 2017-12-27 23:28:37 UTC
Due to this code ( https://bugs.ruby-lang.org/issues/14005 ) added to the Ruby 2.3.6 release and beyond, Webrick, the default http daemon for Ruby processes, can not create OpenSSL connections using 1.0.2k-freebsd in Base.

While a separate issue is being looked into with Ruby upstream and hopefully resolved, this is an opportunity for us to move towards our port being in compliance with Puppet best practices.

The sysutils/puppet4 port currently relies on the system installation of Ruby, which isn't a problem. However, we currently ship a "puppetmaster" rc.d script with puppet4 that launches a webrick process instead of pointing folks to the appropriate way to manage a puppetserver in production.

As is shown here ( 
https://puppet.com/docs/puppet/4.10/services_master_webrick.html#important-deprecation-warning
https://docs.puppet.com/puppet/4.1/deprecated_servers.html
), this method of running a puppet server is being deprecated upstream and soon will no longer be functional even on the versions of Ruby puppet explicitly tests with.

The way upstream suggests to work with puppet is via puppetserver: 
( https://docs.puppet.com/puppetserver/2.1/services_master_puppetserver.html )

While causing sysutils/puppet4 to rely on puppetserver is not ideal (though that should be the default with sysutils/puppet5), it would be beneficial to users of Puppet4 on FreeBSD to receive a message after installing the port or package that provided some of the links above, and informed them that Puppet Server was the way forward. In future releases, it may make sense to remove the puppetmaster rc.d script entirely, but due to the upcoming (in the next few years) deprecation of Puppet4 entirely, it may not make sense entirely to do so, even with the script being broken after the current Ruby update.

With that being said, as it stands currently, puppetmaster is broken, and I feel we should evaluate the benefits of including a message like the one suggested above to inform users, as all current guides for deploying puppet on freebsd suggest the use of the puppetmaster rc.d script instead of installing sysutils/puppetserver (which acts as a drop-in replacement for puppetmaster)


-----
Logs associated with the aforementioned failure in:/var/log/puppet/masterhttp.log
--
[2017-12-25 21:30:26] ERROR OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=SSLv3 read client hello B: unexpected record
        /usr/local/lib/ruby/site_ruby/2.3/puppet/network/http/webrick.rb:32:in `accept'
        /usr/local/lib/ruby/site_ruby/2.3/puppet/network/http/webrick.rb:32:in `block (2 levels) in listen'
        /usr/local/lib/ruby/2.3/webrick/server.rb:314:in `block in start_thread'

-----
System Information:
FreeBSD 11.1-RELEASE-p1 x64

puppet4-4.10.8
Name           : puppet4
Version        : 4.10.8
Installed on   : Mon Dec 25 21:06:47 2017 PST
Origin         : sysutils/puppet4

ruby-2.3.6,1
Name           : ruby
Version        : 2.3.6,1
Installed on   : Tue Dec 26 21:40:43 2017 PST
Origin         : lang/ruby23
Comment 1 Rainbow 2017-12-28 10:26:49 UTC
As an addendum:
I've opened a bug with Puppet after determining this is specifically an issue with a Puppet supplied ruby file, webrick.rb.

The issue occurs in 
/usr/local/lib/ruby/site_ruby/2.3/puppet/network/http/webrick.rb:32 {
sock.accept
}

If we patch this locally to: {
sock.accept_nonblock
}
it would resolve the issue on our end, until Puppet does the same upstream.

Puppet bug is open at:
https://tickets.puppetlabs.com/browse/PUP-8297
Comment 2 Zach Leslie freebsd_committer freebsd_triage 2017-12-29 23:12:10 UTC
I'd support both of the options you've presented.  Both fix the issue so as not to block, as well as include a message telling users to use sysutils/puppetserver.  Do you have patches for these already?  If so, we can get them tested and committed.  If not, I can work them up.  I'm about to leave on a trip and won't return until after the new year but can give some attention then.  Thanks for raising the issue.
Comment 3 Rainbow 2017-12-30 12:13:50 UTC
@Zach,
Thanks for responding!

>  Do you have patches for these already?
For removing rc.d/puppetmaster, and/or printing a message, I do not, but I can create one! I'll make a diff for adding an advisory message after installing the port, and attach it here.

As for the sock.accept issue, I can create a patch. I've never done that before, but it should be as simple as including ( sed -i -e 's/sock.accept/sock.accept_nonblock/g' /usr/local/lib/ruby/site_ruby/2.3/puppet/network/http/webrick.rb ) in postinstall in the port Makefile. Again, I can include that in the diff :)
Comment 4 Rainbow 2017-12-30 15:18:43 UTC
Created attachment 189222 [details]
sock.accept patch

Attached patch-lib_puppet_network_http_webrick.rb
Comment 5 Rainbow 2017-12-30 15:21:31 UTC
^ I figured out the right way to do it, without relying on sed.
Thank you porter's handbook <3

I debated writing out a post-install message for saying "hey puppetmaster is being deprecated, go install sysutils/puppetserver", but I'm afraid I don't have much of a way with words, and I feel like someone more experienced at writing those messages should take the whack at it.
Comment 6 Romain Tartière freebsd_committer freebsd_triage 2017-12-31 15:43:02 UTC
Hey!

Thanks for the information.  Just like Zach, I'll be back from holiday with time to test and commit things in a few days, and it looks a good idea to tell users of the puppetmaster rc script that this support will be dropped in the future.

AFAIK, when running the puppet master from Apache with passenger, I have a warning each time the service starts reminding me that I should switch to the Java puppet server.  I guess the same message is produced when using the webrick server.

Reproducing this message may be a good starting point.  But a post-install message is probably not the best place because on a typical site, you will run a single puppet-master (which could use the puppetmaster rc script and therefore concerned by this warning) and a bunch of agents that are not concerned by this.

So emitting a warning from the puppetmaster rc script is maybe a better option that a post-install warning?
Comment 7 Rainbow 2017-12-31 19:32:13 UTC
(In reply to Romain Tartière from comment #6)
I think a warning from the script is perfectly reasonable.
The webrick server, as it stands, doesn't ping you with a message like the Passenger version. Hence why I had no idea about the deprecation until I went looking to figure out why my puppetmaster didn't work. Adding one to the rc.d script would save a lot of people a lot of headache :)
Comment 8 Romain Tartière freebsd_committer freebsd_triage 2018-01-06 10:57:36 UTC
Hello!

I tried to reproduce the problem, but have a different one.  I build within poudriere in interactive mode, that is, after the build, I am in a jail with puppet installed.

I change the hostname, configure the puppet master with this catalog:
https://marvin.blogreen.org/~romain/configure-puppetmaster.pp

(so `puppet apply configure-puppetmaster.pp`) and start the puppet master.



However, the webrick server seems to be incapable of speaking with TLS, for example, if I try to talk to it through `openssl s_client -connect ip:port` everything starts well until  I send data in the socket:

34380867032:error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac:/usr/src/crypto/openssl/ssl/s3_pkt.c:535:

(of course, same result with a Puppet Agent)



I had the same result with ssl=libressl and have not tested with ssl=openssl but would not be surprised to see the same results.



@Rainbow, can you please confirm that this is not the issue you where initially reporting?
Comment 9 Rainbow 2018-01-06 22:36:06 UTC
(In reply to Romain Tartière from comment #8)
Looks very similar to the one I was reporting.
Can you try the patch I attached and see if that resolves the issue? From my understanding of puppet's use of webrick and the changes in Ruby, it should work :)
Comment 10 Romain Tartière freebsd_committer freebsd_triage 2018-01-07 12:26:04 UTC
(In reply to Rainbow from comment #9)
I have this behavior with and without the patch…

Since I am using a recent snapshot of the ports tree, maybe I should downgrade e.g. Ruby.  What version of ruby have you been testing with?

My poudriere jail has 2.4.3p205
Comment 11 Rainbow 2018-01-08 18:08:52 UTC
(In reply to Romain Tartière from comment #10)
Interesting. I'm a newbie at Ruby (it rhymes!), so my patch may not exactly work...

It's possible there are some other files that need changing.
I'll dig into it later today.
Comment 12 Rainbow 2018-01-18 20:54:11 UTC
My understanding of ruby is awful! Whee!
I started digging into the issue in depth and found some stuff that looked like it was a ruby issue itself.

I had opened a similar bug with puppet, and they've merged fixes upstream to resolve this in the latest push from the 4.10.x series.

This bug is resolved by updating.
Comment 13 Romain Tartière freebsd_committer freebsd_triage 2018-02-08 10:27:35 UTC
Err, I don't understand why this PR is closed.  Should not it still be open?
Comment 14 Walter Schwarzenfeld 2018-02-08 10:30:43 UTC
Yes, was my error - sorry, reopen.
Comment 15 commit-hook freebsd_committer freebsd_triage 2018-02-08 10:34:14 UTC
A commit references this bug:

Author: romain
Date: Thu Feb  8 10:33:39 UTC 2018
New revision: 461209
URL: https://svnweb.freebsd.org/changeset/ports/461209

Log:
  Update to 4.10.10

  Release notes:
  https://puppet.com/docs/puppet/4.10/release_notes.html#puppet-41010

  While here, warn users about running a rack/webrick puppet master [1].

  With hat:	puppet
  PR:		224623
  Submitted by:	w.schwarzenfeld@utanet.at

Changes:
  head/sysutils/puppet4/Makefile
  head/sysutils/puppet4/distinfo
  head/sysutils/puppet4/files/puppetmaster.in
Comment 16 commit-hook freebsd_committer freebsd_triage 2018-02-08 10:35:17 UTC
A commit references this bug:

Author: romain
Date: Thu Feb  8 10:34:12 UTC 2018
New revision: 461210
URL: https://svnweb.freebsd.org/changeset/ports/461210

Log:
  Update to 5.3.4

  Release notes:
  https://puppet.com/docs/puppet/5.3/release_notes.html#puppet-534

  While here, warn users about running a rack/webrick puppet master [1].

  With hat:       puppet
  PR:             224623
  Submitted by:   w.schwarzenfeld@utanet.at

Changes:
  head/sysutils/puppet5/Makefile
  head/sysutils/puppet5/distinfo
  head/sysutils/puppet5/files/puppetmaster.in
Comment 17 Romain Tartière freebsd_committer freebsd_triage 2018-02-08 10:43:26 UTC
@w.schwarzenfeld ah okay, thanks!

@all, the puppetmaster issue remains, but as suggested, I added a warning when installing the port.
Comment 18 Walter Schwarzenfeld 2018-02-08 10:46:48 UTC
Ok, but I have not submitted this, this was "rainbow" ;-))
Comment 19 Romain Tartière freebsd_committer freebsd_triage 2018-02-08 10:50:11 UTC
RAhhhh…  I hate myself!

Copying an address in Firefox does not update the X selection…  And when the X selection already has an e-mail address, it definitively looks like an e-mail address.  Sorry about that Rainbow!
Comment 20 Rainbow 2018-02-09 20:42:18 UTC
heh, no worries.
Bug has been resolved in:
https://tickets.puppetlabs.com/browse/PUP-8297
Not sure if it's made it into a release yet, but soon :)
Comment 21 Romain Tartière freebsd_committer freebsd_triage 2018-02-10 08:09:05 UTC
Oh!  It seems to be fixed in 5.3.4!

--------------------------------8<--------------------------------
root@11_1_RELEASE_amd64-default:/tmp # pkg info | egrep '(puppet|ruby|libressl)-'
libressl-2.6.4                 Free version of the SSL/TLS protocol forked from OpenSSL
puppet5-5.3.4                  Configuration management framework written in Ruby
ruby-2.4.3,1                   Object-oriented interpreted scripting language


root@11_1_RELEASE_amd64-default:/tmp # fetch https://marvin.blogreen.org/~romain/configure-puppetmaster.pp
configure-puppetmaster.pp                     100% of  607  B 8847 kBps 00m00s


root@11_1_RELEASE_amd64-default:/tmp # puppet apply configure-puppetmaster.pp
Notice: Compiled catalog for 11_1_release_amd64-default in environment production in 0.03 seconds
Notice: /Stage[main]/Main/File[/usr/local/etc/puppet/puppet.conf]/ensure: defined content as '{md5}c75c4c13656e6168385a7c5d9818bb15'
Notice: /Stage[main]/Main/File[/usr/local/etc/puppet/environments]/ensure: created
Notice: /Stage[main]/Main/File[/usr/local/etc/puppet/environments/production]/ensure: created
Notice: /Stage[main]/Main/File[/usr/local/etc/puppet/environments/production/manifests]/ensure: created
Notice: /Stage[main]/Main/File[/usr/local/etc/puppet/environments/production/manifests/site.pp]/ensure: defined content as '{md5}0b25476b38c8d65aff666b32196a0ff3'
Notice: Applied catalog in 0.02 seconds


root@11_1_RELEASE_amd64-default:/tmp # service puppetmaster onestart
Performing sanity check of puppetmaster configuration: OK
Starting puppetmaster.
--------------------------------8<--------------------------------

I can fetch the CA certificate (using curl --insecure 'https://127.0.0.1:8149/puppet-ca/v1/certificate/ca?environment=production'), and unauthorized endpoints report a proper error message.

\o/  Thanks for the follow-up!

I close this issue because I am confident that the problem has been fixed upstream.
Please comment and spam-me to re-open if I am wrong!