I have already discussed this some time ago with kevans@ who wanted to talk with bapt@ about this, but we never have got around to report an actual issue. We heavily rely on certctl(8) developed by allanjude@ and kevans@ to add all root and intermediate CAs for Siemens in /usr/local/share/certs: > # certctl list | grep -i -e "QuoVadis Enterprise" -e Siemens > 0e436b80.0 Siemens Issuing CA Class OneAD 04 > 0e91b229.0 Siemens Issuing CA Class OneAD 14 > 17c10339.0 Siemens Issuing CA Class OneAD 05 > 1c683ff3.0 Siemens Issuing CA Class OneAD 13 > 1c7620aa.0 Siemens Issuing CA Internet Code Signing 2016 > 1d069c00.0 Siemens Issuing CA Class OneAD 10 > 2b9d74f0.0 Siemens Issuing CA EE Network Smartcard Auth 2016 > 31e09dd6.0 Siemens Issuing CA Class OneAD 11 > 3210d285.0 Siemens Issuing CA Class OneAD 03 > 4ca5f54b.0 QuoVadis Enterprise Trust CA 2 G3 > 55683be0.0 Siemens Issuing CA EE Auth 2020 > 681ad634.0 Siemens Issuing CA Internet Server 2016 > 780389f9.0 QuoVadis Enterprise Trust CA 1 G3 > 7b1c186e.0 Siemens Issuing CA Class OneAD 06 > 81841955.0 Siemens Issuing CA EE Network Smartcard Auth 2020 > 8aac0ad6.0 Siemens Issuing CA EE Enc 2020 > 8af2d467.0 Siemens OneAD Root CA > 8dc03e53.0 Siemens Internet CA V1.0 > 9245e478.0 Siemens Issuing CA Class OneAD 02 > 930b5fed.0 Siemens Issuing CA Multi Purpose 2016 > 970040fc.0 Siemens Issuing CA Class OneAD 09 > 9ec27e42.0 Siemens Issuing CA Intranet Server 2016 > a331fcb4.0 Siemens Issuing CA EE Auth 2016 > a382b08f.0 Siemens Trust Center Root-CA V2.0 > aec1ff25.0 Siemens Issuing CA Class OneAD 08 > b428065f.0 Siemens Issuing CA EE Enc 2016 > b5d79467.0 QuoVadis Enterprise Trust CA 3 G3 > bc2924b7.0 Siemens Issuing CA Class OneAD 01 > bd411d1f.0 Siemens Issuing CA Class OneAD 00 > be133774.0 Siemens Issuing CA Medium Strength Authentication 2020 > c2cf79c6.0 Siemens Issuing CA Class OneAD 12 > d4555404.0 Siemens Issuing CA Intranet Server 2017 > d7532a42.0 Siemens Issuing CA Internet Server 2017 > d9d79a66.0 Siemens Root CA V3.0 2016 > ddc2d14f.0 Siemens Issuing CA MSA Impersonalized Entities 2016 > e6a60b73.0 Siemens Issuing CA Intranet Code Signing 2016 > f103366b.0 Siemens Issuing CA Class OneAD 07 > f38f510d.0 Siemens Issuing CA Medium Strength Authentication 2016 So all applications linked to OpenSSL will use this store out of the box w/o any further configuration. Now, when I use fetch/pkg (libfetch) to get files hosted on corporate servers certificate verification fails when some port/package (transitive dependency) requires the security/ca_root_nss pestilence for some reason. The reason is that libfetch has not been modified when certctl(8) introduced. The fauly code is (https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/lib/libfetch/common.c#L1082-L1105): > if (getenv("SSL_NO_VERIFY_PEER") == NULL) { > ca_cert_file = getenv("SSL_CA_CERT_FILE"); > if (ca_cert_file == NULL && > access(LOCAL_CERT_FILE, R_OK) == 0) > ca_cert_file = LOCAL_CERT_FILE; This is installed by the security/ca_root_nss port thus > if (ca_cert_file == NULL && > access(BASE_CERT_FILE, R_OK) == 0) > ca_cert_file = BASE_CERT_FILE; > ca_cert_path = getenv("SSL_CA_CERT_PATH"); > if (verbose) { > fetch_info("Peer verification enabled"); > if (ca_cert_file != NULL) > fetch_info("Using CA cert file: %s", > ca_cert_file); > if (ca_cert_path != NULL) > fetch_info("Using CA cert path: %s", > ca_cert_path); > if (ca_cert_file == NULL && ca_cert_path == NULL) > fetch_info("Using OpenSSL default " > "CA cert file and path"); > } > SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, > fetch_ssl_cb_verify_crt); > if (ca_cert_file != NULL || ca_cert_path != NULL) > SSL_CTX_load_verify_locations(ctx, ca_cert_file, > ca_cert_path); > else > SSL_CTX_set_default_verify_paths(ctx); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This never gets activated. I currently addded a workaround to login.conf SSL_CA_CERT_PATH=/etc/ssl/certs, but it still has problems: https://github.com/BastilleBSD/bastille/issues/410 A user's/admin's expectation is that the default system store is used unless otherwise configured. I did neither install BASE_CERT_FILE nor LOCAL_CERT_FILE, it has been imposed by others. So fetch provides --ca-cert and --ca-path which shall override the default store IF they are provided by the user OR SSL_CA_CERT_PATH/SSL_CA_CERT_FILE are set. kevans@ and we also concluded that security/ca_root_nss has very little use sinse there is /etc/ssl/certs now and it could be reduced/removed at some point in time. There ports > # grep -rl --include='*/Makefile' ca_root_nss . | wc -l > 115 need to be revisited why they require security/ca_root_nss.
fetch output: > root@deblndw013x:/usr/ports > # fetch -v https://deblndw011x.ad001.siemens.net/ > resolving server address: deblndw011x.ad001.siemens.net:443 > SSL options: 82004854 > Peer verification enabled > Using CA cert file: /usr/local/etc/ssl/cert.pem > Certificate verification failed for /C=DE/ST=Bayern/L=Muenchen/O=Siemens/serialNumber=ZZZZZZA1/OU=Siemens Trust Center/CN=Siemens Root CA V3.0 2016 > 34370727936:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915: > fetch: https://deblndw011x.ad001.siemens.net/: Authentication error > root@deblndw013x:/usr/ports > # SSL_CA_CERT_PATH=/etc/ssl/certs fetch -v https://deblndw011x.ad001.siemens.net/ > resolving server address: deblndw011x.ad001.siemens.net:443 > SSL options: 82004854 > Peer verification enabled > Using CA cert file: /usr/local/etc/ssl/cert.pem > Using CA cert path: /etc/ssl/certs > Verify hostname > TLSv1.3 connection established using TLS_AES_256_GCM_SHA384 > Certificate subject: /C=DE/O=Siemens/OU=LDA DW/CN=deblndw011x.ad001.siemens.net > Certificate issuer: /C=DE/ST=Bayern/L=Muenchen/O=Siemens/serialNumber=ZZZZZZB7/OU=Siemens Trust Center/CN=Siemens Issuing CA Intranet Server 2017 > requesting https://deblndw011x.ad001.siemens.net/ > remote size / mtime: 45 / 1623218965 > fetch.out 45 B 811 kBps 00s
Anyone willing to go through this with me?
My 12.4-STABLE has the following symlink mixed into /etc/ssl/certs. lrwxr-xr-x 1 root wheel - 53 Aug 9 08:14 cd8c0d63.0 -> ../../../usr/share/certs/trusted/AC_RAIZ_FNMT-RCM.pem lrwxr-xr-x 1 root wheel - 46 Aug 9 08:14 cd8c0d63.1 -> ../../../usr/local/share/certs/ca-root-nss.crt Aug 9 08:14 is the time I did one of installworld, delete-old or delete-old-libs. Does it have anything to do with this symlink to ca-root-nss.crt being made?
(In reply to Tatsuki Makino from comment #3) Not necessarily the symlink, but the default path to the bundle from NSS libfetch uses. It is a misconception from the past which does not apply anymore with a truststore in base.
(In reply to Michael Osipov from comment #4) I came back again with an understanding :) About this part, > if (ca_cert_file == NULL && > access(LOCAL_CERT_FILE, R_OK) == 0) > ca_cert_file = LOCAL_CERT_FILE; > if (ca_cert_file == NULL && > access(BASE_CERT_FILE, R_OK) == 0) > ca_cert_file = BASE_CERT_FILE; If there is a file that can be accessed by either BASE_CERT_FILE or LOCAL_CERT_FILE, it will not be possible to leave it to the default value of openssl.
(In reply to Tatsuki Makino from comment #5) That is correct because: > 789 #define LOCAL_CERT_FILE "/usr/local/etc/ssl/cert.pem" > 790 #define BASE_CERT_FILE "/etc/ssl/cert.pem" The symlink is created by the port: https://github.com/freebsd/freebsd-ports/blob/86387d99797d313d96ddaf6e216285c87e5a9e1c/security/ca_root_nss/Makefile#L52-L55
(In reply to Michael Osipov from comment #6) That is to say that it will be solved with a turn off of option ETCSYMLINK. But it cannot be the default. This is because the symlink for security/openssl is also created there. However, I think its creation should be done on the security/openssl side. (There is a post of my similar statement on bug 248314 comment 15.) It looks like bug 269473 is actually trying to make changes regarding that. I think those reports should be considered integrated...
(In reply to Tatsuki Makino from comment #7) I am aware of those and I also added a comment regarding them. Ultimately, the port maintainers should link to the system's truststore, that's it.
Not a libfetch bug, please file bugs against the respective ports.
(In reply to Dag-Erling Smørgrav from comment #9) I don't agree with that, I expect libfetch to use default store and not depend on the ports system and which port is installed. It clearly breaks the separation of base and ports especially because ca_root_nss sometimes comes as automatic port.
(In reply to Dag-Erling Smørgrav from comment #9) libfetch needs to drop: https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/lib/libfetch/common.c#L1073-L1074
In my testing, libfetch successfully verifies certificates in FreeBSD 12, 13, 14, and 15 with and without ca_root_nss installed. If that is not the case for you, please provide a test case which can be reproduced outside of your corporate IT environment.
(In reply to Dag-Erling Smørgrav from comment #12) I have described this in the title, as soon as ca_root_nss is installed (which I cannot control) and you have enterprise CA certs in your default store via certctl(8) libfetch will fail to verify the peer. Steps to reproduce: * Create a root CA with openssl * Create a CSR for a server * Process CSR with openssl and sign server cert * Use cert in server * Add that root CA to /etc/ssl/certs via certctl(8) * Issue a fetch request to that server * Watch fetch fail
(In reply to Michael Osipov from comment #13) You can simplify if you want by just adding a self-signed server cert to the system trust store and rehash with certctl(8). If ca_root_nss, this cert won't visible. As soon as it is gone the default store will be picked up.
Created attachment 245226 [details] Git-formatted patch (main)
Created attachment 245227 [details] Git-formatted patch (stable/14)
Created attachment 245228 [details] Git-formatted patch (stable/13)
Created attachment 245229 [details] Git-formatted patch (stable/12)
I have now attached a straight forward patch for all currently open branches. This changes nothing for users with internal URLs in terms of behavior, but simplifies life for users with custom CA certs in the system truststore.
I've rewritten the commit message as: libfetch: don't rely on ca_root_nss for certificate validation Before certctl(8), there was no system trust store, and libfetch relied on the CA certificate bundle from the ca_root_nss port to verify peers. We now have a system trust store and a reliable mechanism for manipulating it (to explicitly add, remove, or revoke certificates), but if ca_root_nss is installed, libfetch will still prefer that to the system trust store. With this change, unless explicitly overridden, libfetch will rely on OpenSSL to pick up the default system trust store. is that ok with you?
(In reply to Dag-Erling Smørgrav from comment #20) Yes, absolutely.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=09f5c1e118bb4eca77b83a0d08f559b20f60aa59 commit 09f5c1e118bb4eca77b83a0d08f559b20f60aa59 Author: Michael Osipov <michael.osipov@siemens.com> AuthorDate: 2023-10-03 05:53:20 +0000 Commit: Dag-Erling Smørgrav <des@FreeBSD.org> CommitDate: 2023-10-03 05:53:20 +0000 libfetch: don't rely on ca_root_nss for certificate validation Before certctl(8), there was no system trust store, and libfetch relied on the CA certificate bundle from the ca_root_nss port to verify peers. We now have a system trust store and a reliable mechanism for manipulating it (to explicitly add, remove, or revoke certificates), but if ca_root_nss is installed, libfetch will still prefer that to the system trust store. With this change, unless explicitly overridden, libfetch will rely on OpenSSL to pick up the default system trust store. PR: 256902 MFC after: 3 days Reviewed by: kevans Differential Revision: https://reviews.freebsd.org/D42059 lib/libfetch/common.c | 8 -------- 1 file changed, 8 deletions(-)
(In reply to commit-hook from comment #22) Fantastic, looking forward to MFC! Thank you!
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=fb058a9a40a5adc82721ed822fb4fba213446a7b commit fb058a9a40a5adc82721ed822fb4fba213446a7b Author: Michael Osipov <michael.osipov@siemens.com> AuthorDate: 2023-10-03 05:53:20 +0000 Commit: Dag-Erling Smørgrav <des@FreeBSD.org> CommitDate: 2023-10-05 00:03:16 +0000 libfetch: don't rely on ca_root_nss for certificate validation Before certctl(8), there was no system trust store, and libfetch relied on the CA certificate bundle from the ca_root_nss port to verify peers. We now have a system trust store and a reliable mechanism for manipulating it (to explicitly add, remove, or revoke certificates), but if ca_root_nss is installed, libfetch will still prefer that to the system trust store. With this change, unless explicitly overridden, libfetch will rely on OpenSSL to pick up the default system trust store. PR: 256902 MFC after: 3 days Reviewed by: kevans Differential Revision: https://reviews.freebsd.org/D42059 (cherry picked from commit 09f5c1e118bb4eca77b83a0d08f559b20f60aa59) lib/libfetch/common.c | 8 -------- 1 file changed, 8 deletions(-)
A commit in branch releng/14.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=4357ae1174f37fa2c10f7de7c05536f23e7439c4 commit 4357ae1174f37fa2c10f7de7c05536f23e7439c4 Author: Michael Osipov <michael.osipov@siemens.com> AuthorDate: 2023-10-03 05:53:20 +0000 Commit: Dag-Erling Smørgrav <des@FreeBSD.org> CommitDate: 2023-10-05 15:52:43 +0000 libfetch: don't rely on ca_root_nss for certificate validation Before certctl(8), there was no system trust store, and libfetch relied on the CA certificate bundle from the ca_root_nss port to verify peers. We now have a system trust store and a reliable mechanism for manipulating it (to explicitly add, remove, or revoke certificates), but if ca_root_nss is installed, libfetch will still prefer that to the system trust store. With this change, unless explicitly overridden, libfetch will rely on OpenSSL to pick up the default system trust store. PR: 256902 MFC after: 3 days Reviewed by: kevans Differential Revision: https://reviews.freebsd.org/D42059 Approved by: re (gjb) (cherry picked from commit 09f5c1e118bb4eca77b83a0d08f559b20f60aa59) (cherry picked from commit fb058a9a40a5adc82721ed822fb4fba213446a7b) lib/libfetch/common.c | 8 -------- 1 file changed, 8 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=baf69f6c997392cde9ae75d3ebc25a8201c7cc99 commit baf69f6c997392cde9ae75d3ebc25a8201c7cc99 Author: Michael Osipov <michael.osipov@siemens.com> AuthorDate: 2023-10-03 05:53:20 +0000 Commit: Dag-Erling Smørgrav <des@FreeBSD.org> CommitDate: 2023-10-05 15:55:33 +0000 libfetch: don't rely on ca_root_nss for certificate validation Before certctl(8), there was no system trust store, and libfetch relied on the CA certificate bundle from the ca_root_nss port to verify peers. We now have a system trust store and a reliable mechanism for manipulating it (to explicitly add, remove, or revoke certificates), but if ca_root_nss is installed, libfetch will still prefer that to the system trust store. With this change, unless explicitly overridden, libfetch will rely on OpenSSL to pick up the default system trust store. PR: 256902 MFC after: 3 days Reviewed by: kevans Differential Revision: https://reviews.freebsd.org/D42059 (cherry picked from commit 09f5c1e118bb4eca77b83a0d08f559b20f60aa59) lib/libfetch/common.c | 8 -------- 1 file changed, 8 deletions(-)
A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=8d939b7d98452c0357e49b090d5a685ea8a0e69a commit 8d939b7d98452c0357e49b090d5a685ea8a0e69a Author: Michael Osipov <michael.osipov@siemens.com> AuthorDate: 2023-10-03 05:53:20 +0000 Commit: Dag-Erling Smørgrav <des@FreeBSD.org> CommitDate: 2023-10-05 15:58:59 +0000 libfetch: don't rely on ca_root_nss for certificate validation Before certctl(8), there was no system trust store, and libfetch relied on the CA certificate bundle from the ca_root_nss port to verify peers. We now have a system trust store and a reliable mechanism for manipulating it (to explicitly add, remove, or revoke certificates), but if ca_root_nss is installed, libfetch will still prefer that to the system trust store. With this change, unless explicitly overridden, libfetch will rely on OpenSSL to pick up the default system trust store. PR: 256902 MFC after: 3 days Reviewed by: kevans Differential Revision: https://reviews.freebsd.org/D42059 (cherry picked from commit 09f5c1e118bb4eca77b83a0d08f559b20f60aa59) lib/libfetch/common.c | 8 -------- 1 file changed, 8 deletions(-)
Thank you des@, at long last this is addressed. Looking forward to 13.3-RELEASE and 14-RELEASE.