Bug 228438 - security/acme-client: dies with "bad exit: netproc(29011): 1\0"
Summary: security/acme-client: dies with "bad exit: netproc(29011): 1\0"
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Bernard Spil
URL: https://github.com/kristapsdz/acme-cl...
Keywords:
Depends on:
Blocks:
 
Reported: 2018-05-23 10:45 UTC by Dave Cottlehuber
Modified: 2018-08-09 17:32 UTC (History)
7 users (show)

See Also:
bugzilla: maintainer-feedback? (brnrd)


Attachments
dtruss log FTW (38.15 KB, text/plain)
2018-05-23 10:45 UTC, Dave Cottlehuber
no flags Details
Covert to the net/openntpd method of statically linking against libressl libraries (1.36 KB, patch)
2018-06-25 06:03 UTC, Craig Leres
no flags Details | Diff
acme-client.diff (1.66 KB, patch)
2018-07-19 20:43 UTC, Tobias Kortkamp
no flags Details | Diff
(revised) Covert to the net/openntpd method of statically linking against libressl libraries (2.39 KB, patch)
2018-07-23 16:10 UTC, Craig Leres
leres: maintainer-approval?
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Dave Cottlehuber freebsd_committer 2018-05-23 10:45:17 UTC
Created attachment 193632 [details]
dtruss log FTW

dtruss log attached for a failed renewal:

- 11.1Rp10 amd64
- security/acme-client 0.1.16_3 from latest pkg.freebsd.org 

# /usr/local/share/dtrace-toolkit/dtruss -f -l /usr/local/bin/acme-client -b -mnN -vvv -s -F \                                                                                                                                                                                              -e \                                                                                                                                                                                                                                                                                      -C /var/www/acme  \                                                                                                                                                                                                                                                                     i09.example.com www.example.com > /tmp/acme.dtruss.log 2>&1

This seems to break on both existing production systems and on a fresh test install; this log is from a fresh install. Reverting to 

AFAICT the port now statically includes libressl and therefore should work fine in a "normal openssl" deployed environment.

previous working release (when custom built with default SSL=libressl ports tree) is 0.1.16_2.
Comment 1 Craig Leres freebsd_committer 2018-05-31 01:45:11 UTC
It was the 2.7.2 update to security/libressl that broke acme-client. If I update my ports tree to r471185 and then downgrade security/libressl to r468414 I get a working acme-client binary. Building with security/libressl at r468572 (the 2.7.2 update) results in a broken binary.
Comment 2 Bernard Spil freebsd_committer 2018-05-31 18:47:55 UTC
Hi Dave, Greg,

I have no experience reading dtruss logs, opened an issue upstream (see URL).

I was recently contemplating removing the port as I have not seen any updates to the portable acme-client recently whilst upstream in OpenBSD there are still updates to acme-client. Good to learn there's actually people using the port.

I've not seen this behavior on renewals on my systems, these are using the statically linked acme-client.

Cheers, Bernard.
Comment 3 Bernard Spil freebsd_committer 2018-05-31 18:49:38 UTC
(In reply to Bernard Spil from comment #2)
FWIW, just ran acme-client that is dynamically linked to libssl/libcrypto and it succesfully renewed a cert.
Comment 4 Guido Falsi freebsd_committer 2018-06-09 09:24:36 UTC
(In reply to Bernard Spil from comment #3)
> (In reply to Bernard Spil from comment #2)
> FWIW, just ran acme-client that is dynamically linked to libssl/libcrypto
> and it succesfully renewed a cert.

I was having some problems with acme-client and I found out this bug report (and the one on github upstream repo).

I'm compiling acme-client in poudriere, statically linked to LibreSSL. I'm not installing LibreSSL globally.

I'm seeing this:

# /usr/local/etc/acme/acme-client.sh -v
acme-client: /usr/local/etc/acme/www.madpilot.net/privkey.pem: account key exists (not creating)
acme-client: /usr/local/etc/ssl/acme/private/www.madpilot.net/privkey.pem: domain key exists (not creating)
acme-client: tls_init: No error: 0
[...]

and if I force a renewal it fails like this:

# acme-client -bmnNe -vvv -F -C '/jails/webserver/usr/local/www/acme-challenge' www.madpilot.net pkg.madpilot.net                                                                                                                 
acme-client: /usr/local/etc/acme/www.madpilot.net/privkey.pem: account key exists (not creating)
acme-client: /usr/local/etc/ssl/acme/private/www.madpilot.net/privkey.pem: domain key exists (not creating)
acme-client: tls_init: No error: 0
acme-client: /usr/local/etc/acme/www.madpilot.net/privkey.pem: loading RSA account key
acme-client: /usr/local/etc/ssl/acme/private/www.madpilot.net/privkey.pem: loading domain key
acme-client: /usr/local/etc/ssl/acme/private/www.madpilot.net/privkey.pem: loaded RSA domain key
acme-client: adding SAN: pkg.madpilot.net
acme-client: /usr/local/etc/ssl/acme/www.madpilot.net/cert.pem: certificate valid: 40 days left
acme-client: /usr/local/etc/ssl/acme/www.madpilot.net/cert.pem: forcing renewal
acme-client: bad exit: netproc(92685): 1


Downgrading LibreSSL to 2.6.4 and forcing a recompilation and reinstalling does fix the issue, the  tls_init errors disappear, and it does work correctly.
Comment 5 Guido Falsi freebsd_committer 2018-06-10 16:26:24 UTC
I tried to investigate this further and I think this upstream commit is related:

https://github.com/libressl-portable/openbsd/commit/275ef90f1110d91ccbcb0a78ee5c6fdd5e021938

(this commit was backed out and then added again with minor changes)

I don't know much about threading libraries to really go any further, but i suspect this code acts up when statically linked in acme-client.

Unluckily simply reverting that is not working, other problems come out, most probably due to other related changes in the library made at a later time.

I think we need someone with solid pthreads knowledge to look at this.
Comment 6 Jan Bramkamp 2018-06-18 13:09:57 UTC
The latest acme-client version from ports still fails to refresh certificates:

# acme-client -F example.org
acme-client: tls_init: No error: 0
acme-client: /usr/local/etc/ssl/acme/cert.pem: forcing renewal
Comment 7 Craig Leres freebsd_committer 2018-06-25 06:03:14 UTC
Created attachment 194620 [details]
Covert to the net/openntpd method of statically linking against libressl libraries

FYI, here's the test I use from inside a "poudriere testport -i ..." jail:

    acme-client -bmnNe -vvv -F -C /tmp example.net example.net 

If this line is present:

    acme-client: tls_init: No error: 0

Then acme-client is broken.

I did some testing today and found you can create a working acme-client by adding:

    DEFAULT_VERSIONS+= ssl=libressl

to make.conf. So there must be some significant difference between the static and dynamic versions of the libressl libraries.

acme-client broke when security/libressl was updated to 2.7.2. I notice that a bunch of other ports needed changes for libressl 2.7 as well and looked at a number of them. Most were compile time issues but I noticed that net/openntpd uses a different method of building libressl when the default is not libressl. The main difference is that acme-client tries to remove the -l libraries from the makefile:

    post-configure:
            ${REINPLACE_CMD} 's/-l[a-z]*//g' ${WRKSRC}/GNUmakefile

while openntpd copies the libressl stage tree leaving behind the .so files. This seems like a better strategy especially when you look at what acme-client changes:

    39c39
    < LIBADD        += $(shell pkg-config --libs libbsd)
    ---
    > LIBADD        += $(shell pkg-config - libbsd)
    64c64
    < #LIBADD       += $(shell pkg-config --libs libseccomp)
    ---
    > #LIBADD       += $(shell pkg-config - libseccomp)
    115c115
    < LDFLAGS       += $(shell pkg-config --libs libtls 2>/dev/null || echo '-L/usr/pkg/libressl/lib')
    ---
    > LDFLAGS       += $(shell pkg-config - libtls 2>/dev/null || echo '-L/usr/pkg/libressl/lib')
    127c127
    <       $(CC) -o $@ $(OBJS) $(LDFLAGS) -ltls -lssl -lcrypto $(LIBADD)
    ---
    >       $(CC) -o $@ $(OBJS) $(LDFLAGS)    $(LIBADD)

Which is clearly broken. And I just noticed that there's also files/extra-patch-GNUmakefile which I don't think is ever used? Here is a link to the net/openntpd PR:

    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213691

I tried the openntpd way with acme-client but unfortunately this doesn't solve the PR. But I think it's a better method so here's a patch.

I also opened an issue on github:

    https://github.com/kristapsdz/acme-client-portable/issues/52

I believe now that this is probably a libressl bug but the acme-client guy is probably best equipped to diagnose it.
Comment 8 Tobias Kortkamp freebsd_committer 2018-07-19 20:43:13 UTC
Created attachment 195282 [details]
acme-client.diff

(In reply to Guido Falsi from comment #5)
Thanks for the hint.  Adding -lpthread to LDFLAGS seems to solve
the problem (or at least I don't get the "tls_init: No error: 0"
message anymore, still need to do a real world test).
Comment 9 Guido Falsi freebsd_committer 2018-07-20 14:21:55 UTC
(In reply to Tobias Kortkamp from comment #8)
> Created attachment 195282 [details]
> acme-client.diff
> 
> (In reply to Guido Falsi from comment #5)
> Thanks for the hint.  Adding -lpthread to LDFLAGS seems to solve
> the problem (or at least I don't get the "tls_init: No error: 0"
> message anymore, still need to do a real world test).

Good, I'll try to test you patch asap and report back!
Comment 10 Guido Falsi freebsd_committer 2018-07-21 10:11:00 UTC
I made a test and using tobik patch I have forced the renewal of some certificates successfully.

I'd say the patch works fine!
Comment 11 Jan Bramkamp 2018-07-23 07:37:37 UTC
I can confirm that the patch restores acme-client to a working state with ssl=openssl in make.conf.
Comment 12 Craig Leres freebsd_committer 2018-07-23 16:10:58 UTC
Created attachment 195392 [details]
(revised) Covert to the net/openntpd method of statically linking against libressl libraries

The acme-client.diff patch tobik@ attached produces a working acme-client with ssl=openssl for me as well.

I would still lobby for getting rid of the post-configure REINPLACE_CMD that mangles GNUmakefile (see comment 7). I've attached an updated version of my "net/openntpd method" patch that adds -lpthread LDFLAGS case (and also removes files/extra-patch-GNUmakefile).
Comment 13 Tobias Kortkamp freebsd_committer 2018-07-26 11:25:45 UTC
(In reply to Craig Leres from comment #12)
I'm proposing a new USES=libtls so that this can be handled in a uniform way.

https://reviews.freebsd.org/D16454
Comment 14 commit-hook freebsd_committer 2018-08-02 05:33:49 UTC
A commit references this bug:

Author: tobik
Date: Thu Aug  2 05:33:17 UTC 2018
New revision: 476218
URL: https://svnweb.freebsd.org/changeset/ports/476218

Log:
  security/acme-client: Fix runtime when SSL_DEFAULT != libressl

  - Make sure we link with libthr as well to prevent runtime errors
    like "tls_init: No error: 0" [1].

  - Adopt the nicer approach for linking with libtls taken by
    net/openntpd; it doesn't require patching of build files [2].

  - Remove unused patch

  PR:		228438
  Submitted by:	madpilot, tobik [1], leres [2]
  Reported by:	dch
  Approved by:	brnrd (maintainer timeout, 2 weeks)

Changes:
  head/security/acme-client/Makefile
  head/security/acme-client/files/extra-patch-GNUmakefile
Comment 15 Dave Cottlehuber freebsd_committer 2018-08-02 07:43:45 UTC
thanks Tobi & Craig!
Comment 16 Bernard Spil freebsd_committer 2018-08-09 17:32:44 UTC
Thanks for tracking this down!