Bug 274322 - security/ca_root_nss and net/openntpd failed to load constraint
Summary: security/ca_root_nss and net/openntpd failed to load constraint
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Dag-Erling Smørgrav
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-10-07 07:36 UTC by jakub_lach
Modified: 2023-10-27 09:17 UTC (History)
14 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description jakub_lach 2023-10-07 07:36:24 UTC
After update of security/ca_root_nss I see -

Oct  7 09:31:27 Thinkpad ntpd[826]: constraint: failed to load constraint ca
Oct  7 09:31:27 Thinkpad ntpd[827]: constraint: failed to load constraint ca
Oct  7 09:31:27 Thinkpad ntpd[823]: constraints configured but none available
Oct  7 09:31:43 Thinkpad ntpd[822]: no reply received in time, skipping initial tting

Not sure if its related.
Comment 1 Herbert J. Skuhra 2023-10-07 08:23:42 UTC
openntpd's configure sets CONSTRAINT_CA="/etc/ssl/cert.pem".

With ca_root_nss-3.93:

98966: openat(AT_FDCWD,"/etc/ssl/cert.pem",O_RDONLY,00) = 5 (0x5)

With ca_root_nss-3.93_1:

25196: openat(AT_FDCWD,"/etc/ssl/cert.pem",O_RDONLY,00) ERR#2 'No such file or directory'
Comment 2 Christos Chatzaras 2023-10-07 08:56:07 UTC
I had the same issue. TLS connections from Roundcube to Dovecot stopped working. I had to install ca_root_nss-3.93 to make it work.
Comment 3 Christian Weisgerber freebsd_committer freebsd_triage 2023-10-07 09:45:42 UTC
(In reply to Herbert J. Skuhra from comment #1)

Actually, CONSTRAINT_CA is not used anywhere.

net/openntpd does this:
  tls_load_file(tls_default_ca_cert_file(), ...)

tls_default_ca_cert_file() is from security/libretls, where it is a wrapper around X509_get_default_cert_file() from OpenSSL. X509_get_default_cert_file() returns X509_CERT_FILE, which is defined to "/etc/ssl/cert.pem".
Comment 4 Herbert J. Skuhra 2023-10-07 11:12:12 UTC
(In reply to Christian Weisgerber from comment #3)
Yes, sorry! I only figured that out after adding my comment. Thanks.
Comment 5 Christos Chatzaras 2023-10-07 11:46:05 UTC
With ca_root_nss-3.93 inside /etc/ssl I see this symlink:

cert.pem -> ../../usr/local/share/certs/ca-root-nss.crt

With ca_root_nss-3.93_1 this symlink doesn't exist.

I create it manually and run `certctl rehash` but still Roundcube can't connect to Dovecot and the logs show:

imap-login: Disconnected: Connection closed: SSL_accept() failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca: SSL alert number 48 (no auth attempts in 0 secs): user=<>, rip=144.76.xxx.xxx, lip=144.76.xxx.xxx, TLS handshaking: SSL_accept() failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca: SSL alert number 48, session=<Yy+16xsHxaSQTGjm>

SSL alert number 48 means "unknown_ca".

Any idea what else changed between these two versions?
Comment 6 Herbert J. Skuhra 2023-10-07 12:07:05 UTC
(In reply to Christos Chatzaras from comment #5)

Have you tried to reinstall ca_root_nss-3.93?

Or you can try to copy/symlink /usr/local/share/certs/ca-root-nss.crt to /usr/local/etc/ssl/cert.pem and /usr/local/openssl/cert.pem.

-rw-r--r--  1 root wheel 746519 Oct  7 14:02 /usr/local/etc/ssl/cert.pem
-rw-r--r--  1 root wheel 746519 Oct  7 14:02 /usr/local/openssl/cert.pem
-rw-r--r--  1 root wheel 746519 Oct  7 14:02 /usr/local/share/certs/ca-root-nss.crt

# file /usr/local/share/certs/ca-root-nss.crt /usr/local/etc/ssl/cert.pem /usr/local/openssl/cert.pem
/usr/local/share/certs/ca-root-nss.crt: ASCII text
/usr/local/etc/ssl/cert.pem:            ASCII text
/usr/local/openssl/cert.pem:            ASCII text
Comment 7 Michael Osipov 2023-10-07 12:08:05 UTC
(In reply to Christos Chatzaras from comment #5)

certctl(8) has zero relation to ca_root_nss, it completely replaces it. I bet that one needs to check the source code and remove the dependency to ca_root_nss longterm to make it right.
Comment 8 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2023-10-07 12:43:30 UTC
libretls uses an old API to get the path to the system trust store.  This API incorrectly returns "/etc/ssl/cert.pem".  I'm not sure if libretls can easily be made to use a hashed directory instead of a bundle, so for now, rather than rushing to fix libretls, I propose partly restoring the ETC_SYMLINK option:

https://reviews.freebsd.org/D42120
Comment 9 Christos Chatzaras 2023-10-07 12:53:12 UTC
With ca_root_nss-3.93:

ls -la /etc/ssl | grep cert.pem
lrwxr-xr-x   1 root  wheel     43 Oct  7 15:39 cert.pem -> ../../usr/local/share/certs/ca-root-nss.crt

ls -la /usr/local/etc/ssl | grep cert.pem
-rw-r--r--   1 root  wheel  746820 Oct  7 15:39 cert.pem

ls -la /usr/local/openssl | grep cert.pem
-rw-r--r--   1 root  wheel  746820 Oct  7 15:39 cert.pem

------


With ca_root_nss-3.93_1:

ls -la /etc/ssl | grep cert.pem

ls -la /usr/local/etc/ssl | grep cert.pem

ls -la /usr/local/openssl | grep cert.pem


------

So after upgrading from ca_root_nss-3.93 to ca_root_nss-3.93_1 all these files are missing. The Roundcube (PHP) needs /usr/local/openssl/cert.pem (the other 2 files are not needed).

Any idea why upgrading to the latest version deletes these files? Here are the changes: https://cgit.freebsd.org/ports/commit/?id=483e74f44b82f20bddd5608beef74b2a5ab38a88
Comment 10 Christos Chatzaras 2023-10-07 13:00:05 UTC
If I upgrade to ca_root_nss-3.93_1 , then copy cert.pem to /usr/local/openssl and then reinstall ca_root_nss-3.93_1 then cert.pem is not deleted. So I believe that during the upgrade and during ca_root_nss-3.93 deinstall this file is removed and then during  ca_root_nss-3.93_1 install the file is not installed.
Comment 11 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2023-10-07 13:05:18 UTC
(In reply to Christos Chatzaras from comment #9)
> Any idea why upgrading to the latest version deletes these files?

Because they're not needed and ports that use them are wrong.  Doubly so when they use them and _don't declare a dependency_ as is the case for libretls and roundcube.

FreeBSD has shipped a fully populated trust store in base since 12.2 and ports should be using that instead of the cert.pem; ca_root_nss should exist solely as a way of getting a newer version of the trust store when FreeBSD lags behind Mozilla.
Comment 12 Christos Chatzaras 2023-10-07 13:17:36 UTC
(In reply to Dag-Erling Smørgrav from comment #11)

Thank you for the reply.

The removal of these 3 files is done during ca_root_nss-3.93 deinstallation and before ca_root_nss-3.93_1 installation, correct?

Also on the ca_root_nss-3.93 Makefile I see this:

do-install:
        ${MKDIR} ${STAGEDIR}${PREFIX}/${CERTDIR}
        ${INSTALL_DATA} ${WRKDIR}/ca-root-nss.crt ${STAGEDIR}${PREFIX}/${CERTDIR}
        ${MKDIR} ${STAGEDIR}${PREFIX}/etc/ssl
        ${LN} -sf ../../${CERTDIR}/ca-root-nss.crt ${STAGEDIR}${PREFIX}/etc/ssl/cert.pem.sample
        ${MKDIR} ${STAGEDIR}${PREFIX}/openssl
        ${LN} -sf ../${CERTDIR}/ca-root-nss.crt ${STAGEDIR}${PREFIX}/openssl/cert.pem.sample

I see no reference for /usr/local/openssl/cert.pem , so how is this installed?
Comment 13 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2023-10-07 13:23:22 UTC
(In reply to Christos Chatzaras from comment #12)
> I see no reference for /usr/local/openssl/cert.pem , so how is this installed?

It  is automatically created by pkg, as a copy of cert.pem.sample, if and only if it does not already exist.
Comment 14 Christos Chatzaras 2023-10-07 13:38:14 UTC
(In reply to Dag-Erling Smørgrav from comment #13)

Thank you. I found it at security/ca_root_nss/pkg-plist:

"@sample openssl/cert.pem.sample"

and I just find out that @sample is a special keyword that does what you explained.

I see that you add cert.pem in /usr/local/openssl too:

https://reviews.freebsd.org/D42120

BTW I don't believe the issue is specific to Roundcube port, but maybe to PHP port or OpenSSL port, because I had same issue with another web application (Laravel) that connects to an external API using PHP curl and my PHP is linked with OpenSSL port and not base system OpenSSL.

I add @bofh and @brnrd in case they need to check something related to their ports.
Comment 15 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2023-10-07 13:50:33 UTC
(In reply to Christos Chatzaras from comment #14)
> BTW I don't believe the issue is specific to Roundcube port, but maybe to PHP port or OpenSSL port,

OpenSSL does not need these files.  It is Roundcube (through the guzzlehttp package that it includes and uses) which forces OpenSSL to use it instead of the system trust store.  Although now that I look at the code it appears that guzzlehttp should work without the symlink, so perhaps there's more going on behind the scenes.
Comment 16 Bernard Spil freebsd_committer freebsd_triage 2023-10-07 15:28:03 UTC
Probably the wrong PR, but when I remove the ca_root_nss port, I get a MySQL SSL first. Checking to see what the fix should be.
Comment 17 Christos Chatzaras 2023-10-07 16:05:46 UTC
(In reply to Dag-Erling Smørgrav from comment #15)

My other application (Laravel) uses guzzlehttp too and in /vendor/guzzlehttp/guzzle/src/Utils.php I see:

    public static function defaultCaBundle(): string
    {
        static $cached = null;
        static $cafiles = [
            // Red Hat, CentOS, Fedora (provided by the ca-certificates package)
            '/etc/pki/tls/certs/ca-bundle.crt',
            // Ubuntu, Debian (provided by the ca-certificates package)
            '/etc/ssl/certs/ca-certificates.crt',
            // FreeBSD (provided by the ca_root_nss package)
            '/usr/local/share/certs/ca-root-nss.crt',
            // SLES 12 (provided by the ca-certificates package)
            '/var/lib/ca-certificates/ca-bundle.pem',
            // OS X provided by homebrew (using the default path)
            '/usr/local/etc/openssl/cert.pem',
            // Google app engine
            '/etc/ca-certificates.crt',
            // Windows?
            'C:\\windows\\system32\\curl-ca-bundle.crt',
            'C:\\windows\\curl-ca-bundle.crt',
        ];
Comment 18 Bernard Spil freebsd_committer freebsd_triage 2023-10-07 16:43:15 UTC
(In reply to Bernard Spil from comment #16)
Issues I refered to was in Roundcube.

Issue was in my config (pre CA store in base), the DSN referred to ca=/etc/ssl/cert.pem. Removing that restored the MariaDB connection.

My connection from Roundcube to Dovecot/IMAP over imaps is fine, uses a LetsEncrypt cert. ($config['imap_host'] = 'ssl://imap.example.org:993')

Connection to Postfix is also OK ($config['smtp_host'] = 'tls://smtp.example.org'; $config['smtp_port'] = '587';) 

Basically the fix is reverting settings to default.

Have some issues with phpMyAdmin, it works but keeps complaining "SSL is used without certification authority" even though verify is set to "true".

Not sure where GuzzleHttp is used in Roundcube... From what I can see it uses bundles exclusively.

For Nextcloud, it required a change of dbdriveropts array

> \PDO::MYSQL_ATTR_SSL_CA => '/etc/ssl/cert.pem',
to
> \PDO::MYSQL_ATTR_SSL_CAPATH => '/etc/ssl/certs',

Guess the removal of the dependency warrants an UPDATING entry.
Comment 19 Andrey Korobkov 2023-10-07 16:58:53 UTC
mail/opensmtpd is another port with such a hidden dependency.
After ports #bde578cbfcf9:

Oct  7 01:42:28 smtpd[39182]: info: OpenSMTPD 7.3.0-portable starting
Oct  7 01:42:29 smtpd[40143]: dispatcher: tls_config_set_ca_file: failed to open CA file '/etc/ssl/cert.pem': No such file or directory
Oct  7 01:42:29 smtpd[39400]: warn: lost child: dispatcher exited abnormally
Oct  7 01:42:29 smtpd[39400]: smtpd: process control socket closed
Comment 20 Michael Osipov 2023-10-07 22:37:53 UTC
(In reply to Christos Chatzaras from comment #17)

This is just soo bad and terribly brittle. Everyone should rather use SSL_CTX_set_default_verify_paths() and done. These "discovery" mechanisms are like a plague.
Comment 21 Krzysztof 2023-10-08 00:16:49 UTC
I've just tested lynx. lynx is searching CA certificates at /usr/local/openssl.

Just after upgrade there is problem with lynx -dump https://some.site.

So I've made:
CERTDESTDIR=/usr/local/openssl/certs certctl  rehash


After that lynx -dump https://some.site is connecting without any problems.

So I think (that's my feeling) that additional 'certctl rehash' shoud be added...
Comment 22 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2023-10-08 04:09:49 UTC
For opensmtpd, see https://reviews.freebsd.org/D42123.  A similar code change to libretls should take care of openntpd.

For lynx, see https://reviews.freebsd.org/D42120.
Comment 23 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2023-10-08 04:21:43 UTC
Correction, libretls appears to already know how to do the right thing, but openntpd insists on preloading a bundle because sandboxing won't let it read the trust store.  But that should not matter on FreeBSD.
Comment 24 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2023-10-08 04:30:49 UTC
Correction to the correction.  It still matters on FreeBSD because although pledge() / unveil() is a no-op, openntpd still chroots into /var/empty and does not have access to the trust store.  So for now, OpenNTPD needs a bundle.
Comment 25 commit-hook freebsd_committer freebsd_triage 2023-10-08 04:37:54 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=52e0c40367d3ebd09ab7169e025c37fbf70b8dee

commit 52e0c40367d3ebd09ab7169e025c37fbf70b8dee
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2023-10-08 04:36:54 +0000
Commit:     Dag-Erling Smørgrav <des@FreeBSD.org>
CommitDate: 2023-10-08 04:36:54 +0000

    security/ca_root_nss: Restore the ETC_SYMLINK.

    It turns out that some ports have an undisclosed dependency on the
    symlink and cannot be trivially changed to use the system trust
    store instead.

    Amend the package message to make it clear that software which relies
    on this symlink is not following recommended practice.

    I will look into getting certctl(8) to provide cert.pem instead, but
    it may take a while until we can rely on this being in place on all
    supported releases.

    This partly reverts commit 483e74f44b82.

    PR:             274322
    MFH:            2023Q4
    Reviewed by:    fluffy
    Differential Revision:  https://reviews.freebsd.org/D42120

 security/ca_root_nss/Makefile             | 12 +++++++++++-
 security/ca_root_nss/files/pkg-message.in | 15 +++++++++++++--
 security/ca_root_nss/pkg-plist            |  3 +++
 3 files changed, 27 insertions(+), 3 deletions(-)
Comment 26 Oleh Hushchenkov 2023-10-08 05:38:44 UTC
Here an error from www/aria2 port:

10/08 08:26:25 [ERROR] Failed to load trusted CA certificates from /etc/ssl/cert.pem. Cause: error:02001002:system library:fopen:No such file or directory

After restoring the ETC_SYMLINK in ca_root_nss it works again.

But now it has untracked dependency to security/ca_root_nss.
Comment 27 Po-Chuan Hsieh freebsd_committer freebsd_triage 2023-10-08 15:54:30 UTC
(In reply to Oleh Hushchenkov from comment #26)

It should be fixed by des@ in ports 06b2f3340c1f6dc764575a8e836c0e2fb226e0b7.
Comment 28 Michael Osipov 2023-10-09 07:16:32 UTC
See also Bug 269473.
Comment 29 Michael Osipov 2023-10-09 07:46:16 UTC
(In reply to Dag-Erling Smørgrav from comment #22)

D42123 looks reasonable to me.
Comment 30 commit-hook freebsd_committer freebsd_triage 2023-10-27 09:17:22 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=e11bc472633868a658ecbb8176b2b3ede4ae6e0d

commit e11bc472633868a658ecbb8176b2b3ede4ae6e0d
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2023-10-27 09:16:29 +0000
Commit:     Dag-Erling Smørgrav <des@FreeBSD.org>
CommitDate: 2023-10-27 09:16:43 +0000

    mail/opensmtpd: Use the correct OpenSSL idiom to load the trust store.

    Fixes:          bde578cbfcf9
    PR:             274322
    MFH:            2023Q4
    Approved by:    fluffy
    Differential Revision:  https://reviews.freebsd.org/D42123

 mail/opensmtpd/Makefile                                  |  2 +-
 mail/opensmtpd/files/patch-mk_smtpd_Makefile.in          |  6 +++---
 .../files/patch-openbsd-compat_libtls_tls.c (new)        | 16 ++++++++++++++++
 3 files changed, 20 insertions(+), 4 deletions(-)