Created attachment 193632 [details] dtruss log FTW dtruss log attached for a failed renewal: - 11.1Rp10 amd64 - security/acme-client 0.1.16_3 from latest pkg.freebsd.org # /usr/local/share/dtrace-toolkit/dtruss -f -l /usr/local/bin/acme-client -b -mnN -vvv -s -F \ -e \ -C /var/www/acme \ i09.example.com www.example.com > /tmp/acme.dtruss.log 2>&1 This seems to break on both existing production systems and on a fresh test install; this log is from a fresh install. Reverting to AFAICT the port now statically includes libressl and therefore should work fine in a "normal openssl" deployed environment. previous working release (when custom built with default SSL=libressl ports tree) is 0.1.16_2.
It was the 2.7.2 update to security/libressl that broke acme-client. If I update my ports tree to r471185 and then downgrade security/libressl to r468414 I get a working acme-client binary. Building with security/libressl at r468572 (the 2.7.2 update) results in a broken binary.
Hi Dave, Greg, I have no experience reading dtruss logs, opened an issue upstream (see URL). I was recently contemplating removing the port as I have not seen any updates to the portable acme-client recently whilst upstream in OpenBSD there are still updates to acme-client. Good to learn there's actually people using the port. I've not seen this behavior on renewals on my systems, these are using the statically linked acme-client. Cheers, Bernard.
(In reply to Bernard Spil from comment #2) FWIW, just ran acme-client that is dynamically linked to libssl/libcrypto and it succesfully renewed a cert.
(In reply to Bernard Spil from comment #3) > (In reply to Bernard Spil from comment #2) > FWIW, just ran acme-client that is dynamically linked to libssl/libcrypto > and it succesfully renewed a cert. I was having some problems with acme-client and I found out this bug report (and the one on github upstream repo). I'm compiling acme-client in poudriere, statically linked to LibreSSL. I'm not installing LibreSSL globally. I'm seeing this: # /usr/local/etc/acme/acme-client.sh -v acme-client: /usr/local/etc/acme/www.madpilot.net/privkey.pem: account key exists (not creating) acme-client: /usr/local/etc/ssl/acme/private/www.madpilot.net/privkey.pem: domain key exists (not creating) acme-client: tls_init: No error: 0 [...] and if I force a renewal it fails like this: # acme-client -bmnNe -vvv -F -C '/jails/webserver/usr/local/www/acme-challenge' www.madpilot.net pkg.madpilot.net acme-client: /usr/local/etc/acme/www.madpilot.net/privkey.pem: account key exists (not creating) acme-client: /usr/local/etc/ssl/acme/private/www.madpilot.net/privkey.pem: domain key exists (not creating) acme-client: tls_init: No error: 0 acme-client: /usr/local/etc/acme/www.madpilot.net/privkey.pem: loading RSA account key acme-client: /usr/local/etc/ssl/acme/private/www.madpilot.net/privkey.pem: loading domain key acme-client: /usr/local/etc/ssl/acme/private/www.madpilot.net/privkey.pem: loaded RSA domain key acme-client: adding SAN: pkg.madpilot.net acme-client: /usr/local/etc/ssl/acme/www.madpilot.net/cert.pem: certificate valid: 40 days left acme-client: /usr/local/etc/ssl/acme/www.madpilot.net/cert.pem: forcing renewal acme-client: bad exit: netproc(92685): 1 Downgrading LibreSSL to 2.6.4 and forcing a recompilation and reinstalling does fix the issue, the tls_init errors disappear, and it does work correctly.
I tried to investigate this further and I think this upstream commit is related: https://github.com/libressl-portable/openbsd/commit/275ef90f1110d91ccbcb0a78ee5c6fdd5e021938 (this commit was backed out and then added again with minor changes) I don't know much about threading libraries to really go any further, but i suspect this code acts up when statically linked in acme-client. Unluckily simply reverting that is not working, other problems come out, most probably due to other related changes in the library made at a later time. I think we need someone with solid pthreads knowledge to look at this.
The latest acme-client version from ports still fails to refresh certificates: # acme-client -F example.org acme-client: tls_init: No error: 0 acme-client: /usr/local/etc/ssl/acme/cert.pem: forcing renewal
Created attachment 194620 [details] Covert to the net/openntpd method of statically linking against libressl libraries FYI, here's the test I use from inside a "poudriere testport -i ..." jail: acme-client -bmnNe -vvv -F -C /tmp example.net example.net If this line is present: acme-client: tls_init: No error: 0 Then acme-client is broken. I did some testing today and found you can create a working acme-client by adding: DEFAULT_VERSIONS+= ssl=libressl to make.conf. So there must be some significant difference between the static and dynamic versions of the libressl libraries. acme-client broke when security/libressl was updated to 2.7.2. I notice that a bunch of other ports needed changes for libressl 2.7 as well and looked at a number of them. Most were compile time issues but I noticed that net/openntpd uses a different method of building libressl when the default is not libressl. The main difference is that acme-client tries to remove the -l libraries from the makefile: post-configure: ${REINPLACE_CMD} 's/-l[a-z]*//g' ${WRKSRC}/GNUmakefile while openntpd copies the libressl stage tree leaving behind the .so files. This seems like a better strategy especially when you look at what acme-client changes: 39c39 < LIBADD += $(shell pkg-config --libs libbsd) --- > LIBADD += $(shell pkg-config - libbsd) 64c64 < #LIBADD += $(shell pkg-config --libs libseccomp) --- > #LIBADD += $(shell pkg-config - libseccomp) 115c115 < LDFLAGS += $(shell pkg-config --libs libtls 2>/dev/null || echo '-L/usr/pkg/libressl/lib') --- > LDFLAGS += $(shell pkg-config - libtls 2>/dev/null || echo '-L/usr/pkg/libressl/lib') 127c127 < $(CC) -o $@ $(OBJS) $(LDFLAGS) -ltls -lssl -lcrypto $(LIBADD) --- > $(CC) -o $@ $(OBJS) $(LDFLAGS) $(LIBADD) Which is clearly broken. And I just noticed that there's also files/extra-patch-GNUmakefile which I don't think is ever used? Here is a link to the net/openntpd PR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213691 I tried the openntpd way with acme-client but unfortunately this doesn't solve the PR. But I think it's a better method so here's a patch. I also opened an issue on github: https://github.com/kristapsdz/acme-client-portable/issues/52 I believe now that this is probably a libressl bug but the acme-client guy is probably best equipped to diagnose it.
Created attachment 195282 [details] acme-client.diff (In reply to Guido Falsi from comment #5) Thanks for the hint. Adding -lpthread to LDFLAGS seems to solve the problem (or at least I don't get the "tls_init: No error: 0" message anymore, still need to do a real world test).
(In reply to Tobias Kortkamp from comment #8) > Created attachment 195282 [details] > acme-client.diff > > (In reply to Guido Falsi from comment #5) > Thanks for the hint. Adding -lpthread to LDFLAGS seems to solve > the problem (or at least I don't get the "tls_init: No error: 0" > message anymore, still need to do a real world test). Good, I'll try to test you patch asap and report back!
I made a test and using tobik patch I have forced the renewal of some certificates successfully. I'd say the patch works fine!
I can confirm that the patch restores acme-client to a working state with ssl=openssl in make.conf.
Created attachment 195392 [details] (revised) Covert to the net/openntpd method of statically linking against libressl libraries The acme-client.diff patch tobik@ attached produces a working acme-client with ssl=openssl for me as well. I would still lobby for getting rid of the post-configure REINPLACE_CMD that mangles GNUmakefile (see comment 7). I've attached an updated version of my "net/openntpd method" patch that adds -lpthread LDFLAGS case (and also removes files/extra-patch-GNUmakefile).
(In reply to Craig Leres from comment #12) I'm proposing a new USES=libtls so that this can be handled in a uniform way. https://reviews.freebsd.org/D16454
A commit references this bug: Author: tobik Date: Thu Aug 2 05:33:17 UTC 2018 New revision: 476218 URL: https://svnweb.freebsd.org/changeset/ports/476218 Log: security/acme-client: Fix runtime when SSL_DEFAULT != libressl - Make sure we link with libthr as well to prevent runtime errors like "tls_init: No error: 0" [1]. - Adopt the nicer approach for linking with libtls taken by net/openntpd; it doesn't require patching of build files [2]. - Remove unused patch PR: 228438 Submitted by: madpilot, tobik [1], leres [2] Reported by: dch Approved by: brnrd (maintainer timeout, 2 weeks) Changes: head/security/acme-client/Makefile head/security/acme-client/files/extra-patch-GNUmakefile
thanks Tobi & Craig!
Thanks for tracking this down!