Bug 251562 - security/py-certifi: SSLError 'certificate verify failed' despite correct looking /etc/ssl/cert.pem
Summary: security/py-certifi: SSLError 'certificate verify failed' despite correct loo...
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-python (Nobody)
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2020-12-03 13:45 UTC by Andreas Strauch
Modified: 2021-01-03 09:01 UTC (History)
3 users (show)

See Also:
bugzilla: maintainer-feedback? (python)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andreas Strauch 2020-12-03 13:45:28 UTC
Overview:
the command 'python3.7 -c "import requests; print(requests.get('https://server:8443').text)"' fails with a 'certificate verify failed' error when additional root certificates from the trust store should be used

Steps to Reproduce:
1) import a self-signed root certificate into /usr/share/certs/trusted, i.e. use: cp myroot.pem /usr/share/certs/trusted)
2) get fingerprint from root certificate and add softlink to /etc/ssl/certs, i.e. use: ln -s /usr/share/certs/trusted/myroot.pem /etc/ssl/certs/97efb5b5.0
3) append content of myroot.pem to /etc/ssl/cert.pem, i.e. use: cat myroot.pem | sudo tee -a /etc/ssl/cert.pem)
4) verify that OpenSSL can verify a server certificate that is derived from the newly imported root certificate, i.e. use: openssl s_client -connect server:8443
5) make sure that python is looking into the correct directories for SSL CERTS, i.e. use: python3.7 -c "import ssl; print(ssl.get_default_verify_paths())"
5) try a python request to the server, i.e. use: python3.7 -c "import requests; print(requests.get('https://server:8443').text)"

Actual Results:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket
    cnx.do_handshake()
  File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1915, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 371, in connect
    ssl_context=context,
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 386, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 494, in wrap_socket
    raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='server', port=8443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='server', port=8443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

Expected Results:
HTML content from the server's website (i.e. 'Hello World')

Additional Information:
When I modify the request so I explicitly tell python to use /etc/ssl/cert.pem for verification, I get my 'Hello World' and no error...

python3.7 -c "import requests; print(requests.get('https://server:8443', verify='/etc/ssl/cert.pem').text)"

Also, the output of python's ssl.get_default_verify_paths() shows me that it seems to look all in the right places:

DefaultVerifyPaths(cafile='/etc/ssl/cert.pem', capath='/etc/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/ssl/certs')
Comment 1 nimaje+fbz 2021-01-03 08:44:58 UTC
requests uses certifi to get its default trust store.
certifi brings an own trust store, so requests won't look in the system one by default.

It seems like other OS patch certifi to return the system trust store[0]


[0]: https://github.com/psf/requests/issues/2966#issuecomment-614323746