Overview: the command 'python3.7 -c "import requests; print(requests.get('https://server:8443').text)"' fails with a 'certificate verify failed' error when additional root certificates from the trust store should be used Steps to Reproduce: 1) import a self-signed root certificate into /usr/share/certs/trusted, i.e. use: cp myroot.pem /usr/share/certs/trusted) 2) get fingerprint from root certificate and add softlink to /etc/ssl/certs, i.e. use: ln -s /usr/share/certs/trusted/myroot.pem /etc/ssl/certs/97efb5b5.0 3) append content of myroot.pem to /etc/ssl/cert.pem, i.e. use: cat myroot.pem | sudo tee -a /etc/ssl/cert.pem) 4) verify that OpenSSL can verify a server certificate that is derived from the newly imported root certificate, i.e. use: openssl s_client -connect server:8443 5) make sure that python is looking into the correct directories for SSL CERTS, i.e. use: python3.7 -c "import ssl; print(ssl.get_default_verify_paths())" 5) try a python request to the server, i.e. use: python3.7 -c "import requests; print(requests.get('https://server:8443').text)" Actual Results: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket cnx.do_handshake() File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1915, in do_handshake self._raise_ssl_error(self._ssl, result) File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error _raise_current_error() File "/usr/local/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue raise exception_type(errors) OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn conn.connect() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 371, in connect ssl_context=context, File "/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 386, in ssl_wrap_socket return context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 494, in wrap_socket raise ssl.SSLError("bad handshake: %r" % e) ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 446, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='server', port=8443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 514, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='server', port=8443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))) Expected Results: HTML content from the server's website (i.e. 'Hello World') Additional Information: When I modify the request so I explicitly tell python to use /etc/ssl/cert.pem for verification, I get my 'Hello World' and no error... python3.7 -c "import requests; print(requests.get('https://server:8443', verify='/etc/ssl/cert.pem').text)" Also, the output of python's ssl.get_default_verify_paths() shows me that it seems to look all in the right places: DefaultVerifyPaths(cafile='/etc/ssl/cert.pem', capath='/etc/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/ssl/certs')
requests uses certifi to get its default trust store. certifi brings an own trust store, so requests won't look in the system one by default. It seems like other OS patch certifi to return the system trust store[0] [0]: https://github.com/psf/requests/issues/2966#issuecomment-614323746
@Reporter, This is a change proposal to have certifi use the system, provided root store, is that correct? Note: What other OS packages do with respect to integration of system provided root store aside, there are other considerations too: - POLA with respect to default (as provided by certifi) behaviour - Consistency with upstream documentation - Alternative options that may be documented upstream for integrating additional trust stores (I haven't looked)
(In reply to Kubilay Kocak from comment #2) Yes, correct. IMHO it would be beneficial to have certifi use the system provided root store. For the sake of security, the main goal should be to encourage as much usage of TLS as possible. As an example: my actual use case is about using certbot. I have the 'py37-certbot' and 'py37-certbot-nginx' packages installed because I run my own ACME server at home. Of course my own ACME server does not have a TLS certificate that could be found in official root stores. I have to add the TLS root certificate of my 'personal little enterprise' to the system provided root store. It is a little bit of extra work, but still no problem. Now, my concern is that if 'private' TLS root certificates have to be added in multiple places, it might make the case for not bothering and rely on the '--no-verify-ssl' options (and equivalents) out there. It unnecessarily raises the bar on both complexity and effort to use TLS and as such, undermines the maximum possible speed in which TLS is being used by everybody. Last but not least, I must admit that I know nothing about Python really and I don't know the magnitude of implications involved to make such change. Regardless, I will be happy to help where I can. Please put me in the direction of tasks to be done and I will try my best.
Apologies for resurrecting this old issue, though I'd like to also request `security/py-certifi` to have a configuration option to use system (or `/usr/local`) SSL store (and subsequently depend on `security/ca_root_nss` I guess). At the moment this "issue" can be worked around by exporting `REQUESTS_CA_BUNDLE=/etc/ssl/cert.pem` into env (given `security/ca_root_nss` is installed) on a per use case basis (or sort of globally via system or user profile), though still is a hassle. Thank you in advance.