Bug 237757 - www/nginx-devel: OCSP stapling broken with security/libressl 2.9.1
Summary: www/nginx-devel: OCSP stapling broken with security/libressl 2.9.1
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Sergey A. Osokin
Depends on:
Reported: 2019-05-05 20:43 UTC by Peter Putzer
Modified: 2020-05-14 21:51 UTC (History)
4 users (show)

See Also:

fix/unify access to SSL_CTX certificate chains (924 bytes, patch)
2019-05-12 23:40 UTC, Elias Ohm
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Putzer 2019-05-05 20:43:32 UTC
LibreSSL has been updated to version 2.9.1, after the rebuild, nginx fails to properly do OCSP stapling. OCSP stapling was working fine before, now I get an error for each server section:

2019/05/05 22:38:07 [error] 199#101438: OCSP_basic_verify() failed (SSL: error:27FFF076:OCSP routines:CRYPTO_internal:signer certificate not found) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer:, certificate: "/www/vhosts/<domain>/.ssl/cert.pem"

Even though the configuration was working fine before, I've experimented with adding all intermediate and root certificates to ssl_trusted_certificate, without avail. 

A similar question has been asked on ServerFault: https://serverfault.com/questions/965324/why-am-i-unable-to-make-ocsp-stapling-work-with-my-nginx-libressl-on-freebsd-12?newreg=02a125f6203747b5bb564b489b16ec44
Comment 1 Peter Putzer 2019-05-08 20:43:01 UTC
I can confirm that after I changed make.conf to use security/openssl and recompiling all depending ports OCSP works again. I'd like to use LibreSSL, but having working websites takes precedence.
Comment 2 Bernard Spil freebsd_committer 2019-05-11 12:10:33 UTC
No clue where this comes from. Reached out to OpenBSD to see if they have encountered this too. The OpenBSD port is also 1.16.0 and I don't see any relevant patches for this issue there.
Comment 3 Peter Putzer 2019-05-11 12:14:11 UTC
(In reply to Bernard Spil from comment #2)

Maybe it is not an Nginx issue but a LibreSSL bug? The OCSP responses verify using openssl tools from the command line, but those are compiles with base OpenSSL.
Comment 4 Elias Ohm 2019-05-12 23:40:54 UTC
Created attachment 204342 [details]
fix/unify access to SSL_CTX certificate chains

for newer OpenSSL versions (1.0.2+) the chain is stored in the dedicated chain field (SSL_CTX_set0_chain_certs) belonging to a certificate while in older versions the extra_chain had to be used (SSL_CTX_add_extra_chain_cert) which is always global to the context.

reading the chain is still implemented with SSL_CTX_get_extra_chain_certs for newer versions (if not directly from staple->ssl_ctx->extra_certs in older versions).
however, this works for OpenSSL where the SSL_CTX_get_extra_chain_certs falls back to read chain_certs when no extra_certs are available but breaks for some other implementations where SSL_CTX_get_extra_chain_certs is implemented as SSL_CTX_get_extra_chain_certs_only in OpenSSL is implemented. in addition this is inconsistent use of the functions and the functionality of trying etxra certs and falling back to certifiactes chain is not needed here.
Comment 5 Elias Ohm 2019-05-13 00:21:13 UTC
I do not exactly now why to use LibreSSL instead of OpenSSL, but anyway had a look on hat issue yust for interest.

I din't check commit history for LireSSL 2.9.1, but eighter

- the cause for bringing that up now ist just this one `   - Implemented further missing OpenSSL 1.1 API.` (implemented API function SSL_CTX_set0_chain_certs)
if this function is defined new then nginx just switches to use this function which referes to a chain attached to a certificate in the context instead of an extra chain available at the context.. Whereas the parts in nginx that reads the chain did not implement the get-counter part but sticked at the OpenSSL 1.0.1 function SSL_CTX_get_extra_chain_certs). the function is implemented differently in OpenSSL and LibreSSL, OpenSSL falls back to the certificate chain if no extra chain is present so never fall about the inconsitency in nginx Code where LibreSSL implemented that function with the semantics of the OpenSSL SSL_CTX_get_extra_chain_certs_only (so no fallback to the certificate chain)

- or LibreSSL now decided to to Change sematics of SSL_CTX_get_extra_chain_certs to that of the OpenSSLs SSL_CTX_get_extra_chain_certs_only…

In effect the certificate chain has been stored on the chain field belonging to the certificate but for the stapling check it tried to read that from the extra_chain (so getting no certificate to verify and passing empty value for chain to OCSP_basic_verify, resulting in "signer certificate not found"...

So if You want to have it work You can try with attached patch.
I also submitted it to nginx directly as this is a thing that should be generally fixed (even when it's working with OpenSSL it's not fine to not use the correct counter parts of newly implemeted interfaces).
Comment 6 Sergey A. Osokin freebsd_committer 2019-05-13 19:41:20 UTC

Maxim Dounin has replied that's probably a bug in LibreSSL code, please see https://forum.nginx.org/read.php?29,284141,284147#msg-284147 for details.

So I see no reason to update nginx source code with the patch instead of fixing an issue in LibreSSL.
Comment 7 Elias Ohm 2019-05-13 20:41:52 UTC
Sure it would be fine if LibreSSL would provide same Interface as OpenSSL.
Not sure why this differs, so not I cannot say whether it's a bug or just the design of the library.
It could be a matter of improving security or applying some best practices, or just "laziness" (or beeing short on time or such) that they did it differently.
Of course it would be fine to be 100% interchangable from interface and functionality, but don't know whether they want that.

For me personally it sounds reasonable to make a function called "SSL_CTX_get_extra_chain_certs" to only get the extra chain and not falling back to some other chain (which could be not what is wanted from plain wording of the function).
Would it be the other way round, yes. (Having a SSL_CTX_get_extra_chain_certs and a SSL_CTX_get_extra_chain_certs_or_chain.)

For OpenSSL this has historical reasons, first implementing a convenience function or (more or less undocumented) interface for the chain certificates at a time where the per certificate chain where not available.
Afterwards, yes, for compatibility of 3rd party library code that expects the function to just return the chain from the usual location it's nice to provide such a fallback to not break them all or force the applications to itself stick on the extra-chain (and only working with libs that do so).
Otherwise I assume they would have called the differently to have the function Name doing what it says in both cases on not just in the new one added for proper Access to the extra vars (and exactly them and not the one or the other).
Where on the other Hand thats not really fine - yes and leads to such behaviour the nginx devs Show (it's still working, so why adjust code).
Comment 8 Elias Ohm 2019-05-13 21:15:52 UTC
But for the nginx part they really should fix that.
It's not that they need such fallback within their own code and not that there is no more approriate way to handle that is not available using official documented interfaces.
If they account for the interface enhancment and evolution in the one place they should do it in the other places also.
And even if they expect they could get an pre-populated context from Plugins which may still use the old extra vars instead of the more appropriate new certificate chain (which I did not checked to be honest), it's much better then to expect the plugins to deliver the contexed with the newer structure, implementing an expolicit fallback to the extra chain (recognising the issue and can report warnings about legacy plugins that should better be updated and such).

For builders that want to build nginx against a non default lib they eighther could provide some configuration - or the builders would be in the Duty to deal with that (so define SSL_CTX_get_extra_chain_certs_only to point to SSL_CTX_get_extra_chain_certs).

For the LibreSSL it would at least be a possibly breaking Change for applications that expect exactly the LibreSSL interface and rely on getting extra chain from the context never returns the chain of the current certificate.
Thought there are probably not too many applications affected by that.

Anyway that is the exact explanation for the issue.
And for me a patch to nginx is more approriate. Best at upstream nginx or if they deny for any (for me not realy understandable reason) as a local fix to be able to build with libressl, if someone insits to do so.
Of course if they would change LibreSSL that would solve the specific issue reported here eighter. (While it's unlilkely that there are too many similiar cases like this around that would Profit from that, I assume.)
Comment 9 Peter Putzer 2019-05-23 13:18:31 UTC
(In reply to Elias Ohm from comment #8)
I agree with you, Elias. Maybe you could argue the case on the nginx-devel list (in response to Maxim)?
Comment 10 Peter Putzer 2019-10-20 17:09:27 UTC
Is anyone intending to fix this for the ports?
Comment 11 Toni Ballesta 2019-10-22 12:18:09 UTC
Same issue for the libressl-devel-3.01 and libressl-2.9.2. With nginx package and openssl, no problem.
Comment 12 Peter Putzer 2020-01-02 13:32:44 UTC
The issue has been fixed upstream, not sure when a corresponding version will land in ports.

Comment 13 Peter Putzer 2020-05-11 14:26:58 UTC
This is issue would seem to be fixed with security/libressl 3.1.1.
Comment 14 Sergey A. Osokin freebsd_committer 2020-05-14 21:51:07 UTC
The issue has been fixed with the security/libressl update to the version 3.1.1.