Bug 251562 - security/py-certifi: SSLError 'certificate verify failed' despite correct looking /etc/ssl/cert.pem
Summary: security/py-certifi: SSLError 'certificate verify failed' despite correct loo...
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-python (Nobody)
URL:
Keywords: needs-patch, needs-qa
Depends on:
Blocks:
 
Reported: 2020-12-03 13:45 UTC by Andreas Strauch
Modified: 2021-05-24 07:59 UTC (History)
3 users (show)

See Also:
bugzilla: maintainer-feedback? (python)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andreas Strauch 2020-12-03 13:45:28 UTC
Overview:
the command 'python3.7 -c "import requests; print(requests.get('https://server:8443').text)"' fails with a 'certificate verify failed' error when additional root certificates from the trust store should be used

Steps to Reproduce:
1) import a self-signed root certificate into /usr/share/certs/trusted, i.e. use: cp myroot.pem /usr/share/certs/trusted)
2) get fingerprint from root certificate and add softlink to /etc/ssl/certs, i.e. use: ln -s /usr/share/certs/trusted/myroot.pem /etc/ssl/certs/97efb5b5.0
3) append content of myroot.pem to /etc/ssl/cert.pem, i.e. use: cat myroot.pem | sudo tee -a /etc/ssl/cert.pem)
4) verify that OpenSSL can verify a server certificate that is derived from the newly imported root certificate, i.e. use: openssl s_client -connect server:8443
5) make sure that python is looking into the correct directories for SSL CERTS, i.e. use: python3.7 -c "import ssl; print(ssl.get_default_verify_paths())"
5) try a python request to the server, i.e. use: python3.7 -c "import requests; print(requests.get('https://server:8443').text)"

Actual Results:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket
    cnx.do_handshake()
  File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1915, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 371, in connect
    ssl_context=context,
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 386, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 494, in wrap_socket
    raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='server', port=8443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='server', port=8443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

Expected Results:
HTML content from the server's website (i.e. 'Hello World')

Additional Information:
When I modify the request so I explicitly tell python to use /etc/ssl/cert.pem for verification, I get my 'Hello World' and no error...

python3.7 -c "import requests; print(requests.get('https://server:8443', verify='/etc/ssl/cert.pem').text)"

Also, the output of python's ssl.get_default_verify_paths() shows me that it seems to look all in the right places:

DefaultVerifyPaths(cafile='/etc/ssl/cert.pem', capath='/etc/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/ssl/certs')
Comment 1 Benjamin Takacs 2021-01-03 08:44:58 UTC
requests uses certifi to get its default trust store.
certifi brings an own trust store, so requests won't look in the system one by default.

It seems like other OS patch certifi to return the system trust store[0]


[0]: https://github.com/psf/requests/issues/2966#issuecomment-614323746
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2021-05-24 03:21:01 UTC
@Reporter, This is a change proposal to have certifi use the system, provided root store, is that correct?

Note: What other OS packages do with respect to integration of system provided root store aside, there are other considerations too:

- POLA with respect to default (as provided by certifi) behaviour
- Consistency with upstream documentation
- Alternative options that may be documented upstream for integrating additional trust stores (I haven't looked)
Comment 3 Andreas Strauch 2021-05-24 07:59:10 UTC
(In reply to Kubilay Kocak from comment #2)

Yes, correct. IMHO it would be beneficial to have certifi use the system provided root store. For the sake of security, the main goal should be to encourage as much usage of TLS as possible.

As an example: my actual use case is about using certbot. I have the 'py37-certbot' and 'py37-certbot-nginx' packages installed because I run my own ACME server at home. Of course my own ACME server does not have a TLS certificate that could be found in official root stores. I have to add the TLS root certificate of my 'personal little enterprise' to the system provided root store. It is a little bit of extra work, but still no problem.

Now, my concern is that if 'private' TLS root certificates have to be added in multiple places, it might make the case for not bothering and rely on the '--no-verify-ssl' options (and equivalents) out there. It unnecessarily raises the bar on both complexity and effort to use TLS and as such, undermines the maximum possible speed in which TLS is being used by everybody.

Last but not least, I must admit that I know nothing about Python really and I don't know the magnitude of implications involved to make such change. Regardless, I will be happy to help where I can. Please put me in the direction of tasks to be done and I will try my best.