Bug 233766 - sysutils/puppetserver6 doesn't start on FreeBSD 12.0-RC3
Summary: sysutils/puppetserver6 doesn't start on FreeBSD 12.0-RC3
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Many People
Assignee: FreeBSD Puppet Team
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-12-04 07:32 UTC by bas
Modified: 2019-05-05 02:10 UTC (History)
14 users (show)

See Also:
bugzilla: maintainer-feedback? (puppet)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description bas 2018-12-04 07:32:48 UTC
Install puppetserver6 using pkg on a clean 12.0-RC3 system, enable the puppetserver service and start it. Monitor /var/log/puppetserver/puppetserver.log. You'll see a stream of errors relating to Puppet's inability to modify certain files and/or directories on the system and the service won't start up.

Perform the exact same actions on 11.2-RELEASE and the Puppet server starts without any issues.
Comment 1 Zach Leslie freebsd_committer 2018-12-10 20:34:45 UTC
I'll spend some time in the coming weeks getting a 12 box up.  The motherboard for my build box is currently on my dining table, so some things need addressing first, but I'll try to reproduce this here shortly.
Comment 2 blahblah.1981 2018-12-13 09:26:53 UTC
Just to add to the initial report, I've installed a clean 12.0-RELEASE and am also having the same problems with puppet6-6.0.4, puppetserver5-5.3.6, and even puppetserver-2.8.0_1.

They all seem to have similar errors in /var/log/puppetserver/puppetserver.log like the following.

If you need help testing, I will try to help if I can.

2018-12-13 18:19:08,045 ERROR [clojure-agent-send-pool-0] [puppetserver] /File[/var/run/puppetserver]/ensure change from socket to directory failed: Could not back up files of type socket
2018-12-13 18:19:08,089 INFO  [clojure-agent-send-pool-0] [puppetserver] /File[/var/log/puppetserver/masterhttp.log] Dependency File[/var/log/puppetserver] has failures: true
2018-12-13 18:19:08,090 WARN  [clojure-agent-send-pool-0] [puppetserver] /File[/var/log/puppetserver/masterhttp.log] Skipping because of failed dependencies
RuntimeError: Got 3 failure(s) while initializing: File[/usr/local/etc/puppet]: change from socket to directory failed: Could not back up files of type socket; File[/var/log/puppetserver]: change from unknown to directory failed: Could not back up files of type unknown; File[/var/run/puppetserver]: change from socket to directory failed: Could not back up files of type socke
Comment 3 Zach Leslie freebsd_committer 2018-12-17 01:41:10 UTC
I've filed https://tickets.puppetlabs.com/browse/SERVER-2402 though, if you're seeing the issue with all versions of puppetserver, this could be something else to do with FreeBSD 12.  I'll test puppetserve5 on 12 shortly, which has been working without issue on 11 for quite some time.
Comment 4 Andre Rikkert de Koe - ARK-ICT 2018-12-20 06:14:41 UTC
same with puppetserver6-6.1.0 :

root@infra2:~ # grep ERROR /var/log/puppetserver/puppetserver.log 
2018-12-20T07:10:45.829+01:00 ERROR [clojure-agent-send-pool-0] [puppetserver] Puppet Could not remove files of type unknown
2018-12-20T07:10:45.845+01:00 ERROR [clojure-agent-send-pool-0] [puppetserver] /File[/usr/local/etc/puppet]/ensure change from 'unknown' to 'directory' failed: Could not remove files of type unknown
2018-12-20T07:10:45.854+01:00 ERROR [clojure-agent-send-pool-0] [puppetserver] Puppet Could not remove files of type unknown
2018-12-20T07:10:45.866+01:00 ERROR [clojure-agent-send-pool-0] [puppetserver] /File[/var/puppet/server/data/puppetserver]/ensure change from 'unknown' to 'directory' failed: Could not remove files of type unknown
2018-12-20T07:10:45.880+01:00 ERROR [clojure-agent-send-pool-0] [puppetserver] Puppet Could not remove files of type unknown
2018-12-20T07:10:45.891+01:00 ERROR [clojure-agent-send-pool-0] [puppetserver] /File[/var/log/puppetserver]/ensure change from 'unknown' to 'directory' failed: Could not remove files of type unknown
2018-12-20T07:10:45.899+01:00 ERROR [clojure-agent-send-pool-0] [puppetserver] Puppet Could not remove files of type unknown
2018-12-20T07:10:45.913+01:00 ERROR [clojure-agent-send-pool-0] [puppetserver] /File[/var/run/puppetserver]/ensure change from 'unknown' to 'directory' failed: Could not remove files of type unknown
2018-12-20T07:10:46.048+01:00 ERROR [clojure-agent-send-pool-0] [p.t.internal] shutdown-on-error triggered because of exception!
2018-12-20T07:10:46.049+01:00 ERROR [async-dispatch-2] [p.t.internal] Error during service init!!!
2018-12-20T07:10:46.068+01:00 ERROR [async-dispatch-3] [p.t.internal] Encountered error during shutdown sequence
root@infra2:~ # freebsd-version
12.0-RELEASE-p1
root@infra2:~ #
Comment 5 Joseph Mulloy 2018-12-30 03:24:13 UTC
This is actually a jruby/jnr-posix bug.

https://github.com/jruby/jruby/issues/5521
https://github.com/jnr/jnr-posix/issues/126

---

root@puppet:~ # jirb
irb(main):001:0> File.stat("/usr/local/etc/puppet").ftype()
=> "blockSpecial"
irb(main):002:0> File.stat("/usr/local/etc/puppet/puppet.conf").ftype()
=> "unknown"

---

root@puppet:~ # irb
irb(main):001:0> File.stat("/usr/local/etc/puppet").ftype()
=> "directory"
irb(main):002:0> File.stat("/usr/local/etc/puppet/puppet.conf").ftype()
=> "file"
Comment 6 Muhammad al-iman 2019-01-28 04:01:25 UTC
is there any way for quick fix for this.. I saw there is quick fix for FreeBSD 12 but I dont know how to implement it.
Comment 7 Romain Tartière freebsd_committer 2019-01-28 05:45:51 UTC
(In reply to Muhammad al-iman from comment #6)
> is there any way for quick fix for this.. I saw there is quick fix for FreeBSD 12 but I dont know how to implement it.


For now, the "reliable" way of running puppetserver6 on FreeBSD 12+ seems to be to build a 11.2 jail and run puppetserver from this jail.

If you have some Java skills, jumping head-first in the issues referenced by  Joseph Mulloy and submitting Pull-Requests upstream is likely the most efficient way to see some motion :-)
Comment 8 mikael.urankar 2019-02-18 10:24:33 UTC
Hi puppet@
When can we expect to have a fix for this issue?
Comment 9 Romain Tartière freebsd_committer 2019-03-15 19:20:06 UTC
I started to work on adding support for dlvsym(3) to jnr/jffi;

The fix in jnr/jffi will allow us to update jnr/jnr-ffi.
The fix update jnr/jnr-ffi will allow us to update jnr/jnr-posix.
The fix update jnr/jnr-posix would fix puppetserver6.

However, I am not used to Java *at all*, and fail to find a way to actually test the Java part of the changes:
https://github.com/jnr/jffi/pull/66

If someone reading this has some Java insights, you will find two FIXME in the src/test/java/com/kenai/jffi/VersionTest.java file in my changeset.  Feel free to comment with a fix or submit a Pull Request to my fork that replaces the FIXME with actual code so that I can merge your changes which would update the upstream Pull Request.

Also, if you know who to ping to see motion on this issue, feel free to do so (GitHub seems to be somewhat broken while I am writing this).
Comment 10 commit-hook freebsd_committer 2019-03-15 19:34:43 UTC
A commit references this bug:

Author: romain
Date: Fri Mar 15 19:34:29 UTC 2019
New revision: 495804
URL: https://svnweb.freebsd.org/changeset/ports/495804

Log:
  Mark broken on FreeBSD 12+

  On FreeBSD 12 and FreeBSD 13, puppetserver6 fails to start.  The issue will
  likely not be fixed in a timely fashion, so make it clear to end-users that
  the port is broken.

  As a workaround, users of FreeBSD 12.0 can install puppetserver6 in a
  FreeBSD 11.2 jail.

  More details, progress and ways to help can be found in the PR 233766.

  With hat:	puppet
  PR:		233766
  Reported by:	bas@area536.com

Changes:
  head/sysutils/puppetserver6/Makefile
Comment 11 mikael.urankar 2019-03-16 15:24:07 UTC
(In reply to commit-hook from comment #10)
Is puppetserver5 also affected by this issue?
Comment 12 Romain Tartière freebsd_committer 2019-03-16 15:50:59 UTC
(In reply to mikael.urankar from comment #11)
According to tests I have done in a FreeBSD 12.0 jail, puppetserver5 is not affected.

root@puppet:~ # freebsd-version 
11.2-RELEASE-p9
root@puppet:~ # puppetserver --version
puppetserver version: 6.2.1
root@puppet:~ # nc -v 127.0.0.1 8140
Connection to 127.0.0.1 8140 port [tcp/*] succeeded!
^C
root@puppet:~ # puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for puppet
Info: Applying configuration version '1552751384'
Notice: Applied catalog in 0.03 seconds
Comment 13 mikael.urankar 2019-03-16 15:58:42 UTC
(In reply to Romain Tartière from comment #12)
root@puppet:~ # freebsd-version 
11.2-RELEASE-p9

the issue is with 12.0+ ?
Comment 14 Romain Tartière freebsd_committer 2019-03-16 16:07:30 UTC
(In reply to Romain Tartière from comment #12)
D'oh!  Ignore my previous snippet, it was running Puppet 6 on a FreeBSD 11.2 jail, exactly what you are *not* asking about.

I tweaked my scripts to build a Puppet 5 infra on a bunch of FreeBSD 12 nodes and the same problem as Puppet 6 on FreeBSD 12 seems to appear.

This all makes sense since we believe the change is tied to 64 bits inodes.  Let's update PuppetServer 5 too!

Thanks for the heads-up!
Comment 15 commit-hook freebsd_committer 2019-03-16 16:13:42 UTC
A commit references this bug:

Author: romain
Date: Sat Mar 16 16:12:55 UTC 2019
New revision: 495904
URL: https://svnweb.freebsd.org/changeset/ports/495904

Log:
  Mark broken on FreeBSD 12+

  On FreeBSD 12 and FreeBSD 13, puppetserver5 fails to start.  The issue will
  likely not be fixed in a timely fashion, so make it clear to end-users that
  the port is broken.

  As a workaround, users of FreeBSD 12.0 can install puppetserver5 in a
  FreeBSD 11.2 jail.

  More details, progress and ways to help can be found in the PR 233766.

  These changes are linked to the PuppetServer 6 ones committed previously in
  r495804.

  With hat:	puppet
  PR:		233766
  Reported by:	mikael.urankar@gmail.com

Changes:
  head/sysutils/puppetserver5/Makefile
Comment 16 Tommy P 2019-05-05 02:10:34 UTC
I'm unable to build within 11.2 jail.

root@d-build-fbsd:~ # uname -a
FreeBSD d-build-fbsd.dev.corp 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 r346942 GENERIC  amd64

root@d-build-fbsd:~ # poudriere bulk -j 11amd64 -p default -z server -f amd64-default-server-pkglist
[00:00:00] Creating the reference jail... done
[00:00:20] Mounting system devices for 11amd64-default-server
[00:00:20] Mounting ports/packages/distfiles
[00:00:20] Converting package repository to new format
[00:00:20] Stashing existing package repository
[00:00:20] Mounting packages from: /poudriere/data/packages/11amd64-default-server
[00:00:20] Copying /var/db/ports from: /usr/local/etc/poudriere.d/12amd64-default-server-options
[00:00:20] Appending to make.conf: /root/make.conf
/etc/resolv.conf -> /poudriere/data/.m/11amd64-default-server/ref/etc/resolv.conf
[00:00:20] Starting jail 11amd64-default-server
make: "/usr/ports/Mk/bsd.port.mk" line 1203: UNAME_r (12.0-RELEASE-p3) and OSVERSION (1102000) do not agree on major version number.
make: "/usr/ports/Mk/bsd.port.mk" line 1203: UNAME_r (12.0-RELEASE-p3) and OSVERSION (1102000) do not agree on major version number.
[00:00:21] Logs: /poudriere/data/logs/bulk/11amd64-default-server/2019-05-04_19h04m15s
[00:00:21] Loading MOVED for /poudriere/data/.m/11amd64-default-server/ref/usr/ports
make: "/usr/ports/Mk/bsd.port.mk" line 1203: UNAME_r (12.0-RELEASE-p3) and OSVERSION (1102000) do not agree on major version number.
[00:00:24] Error: Error looking up pre-build ports vars
[00:00:24] Cleaning up
[00:00:24] Unmounting file systems