Bug 245929 - security/py-cryptography: Update to 2.9.2
Summary: security/py-cryptography: Update to 2.9.2
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Kubilay Kocak
URL: https://github.com/pyca/cryptography/...
Keywords: needs-qa
: 247403 (view as bug list)
Depends on: 245928
Blocks: 245930 250337
  Show dependency treegraph
Reported: 2020-04-26 13:20 UTC by daniel.engberg.lists
Modified: 2020-10-20 18:25 UTC (History)
7 users (show)

See Also:
bugzilla: maintainer-feedback? (koobs)

Patch for py-cryptography (4.62 KB, patch)
2020-04-26 13:20 UTC, daniel.engberg.lists
no flags Details | Diff
Patch for py-cryptography v2 (4.94 KB, patch)
2020-04-27 22:19 UTC, daniel.engberg.lists
no flags Details | Diff
py-cryptography-2.9.2-with-openssl-102u.fix (11.60 KB, patch)
2020-08-02 09:03 UTC, Kai Knoblich
kai: maintainer-approval? (koobs)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description daniel.engberg.lists 2020-04-26 13:20:48 UTC
Created attachment 213807 [details]
Patch for py-cryptography

Update to 2.9.2

Remove patch-PR4855, fix committed upstream
Adjust do-test section

Tested on FreeBSD 13.0-CURRENT #0 r358620 (AMD64)
Poudriere OK 12.1-RELEASE (AMD64)

Test results (make test):

===== 3 failed, 103737 passed, 3236 skipped, 26 warnings in 323.10 seconds =====

========== 103849 passed, 3239 skipped, 26 warnings in 350.54 seconds ==========
Comment 1 daniel.engberg.lists 2020-04-27 15:29:19 UTC
I apologize beforehand if this was supposed to go into the other PR.
Anyhow, given https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238968#c3 I'll create a new patch later today or adressing that.
Comment 2 daniel.engberg.lists 2020-04-27 22:19:47 UTC
Created attachment 213873 [details]
Patch for py-cryptography v2

Remove py-asn1crypto dependency, reported by koobs@ in PR 238968
Comment 3 daniel.engberg.lists 2020-05-10 11:41:39 UTC
Hi Kubilay, 

Is there anything else I can do regarding this PR?

Best regards,
Comment 4 Matthias Andree freebsd_committer 2020-06-07 17:20:39 UTC
koobs, ping?
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2020-06-10 03:41:44 UTC
@Matthias This (2.8 -> 2.9.*) update requires substantial QA in particular for dependent ports and the versions they support. The changelogs for 2.8 and onward include several backward incompatible changes, and these need to be qualified for dependent ports.

Making QA somewhat more challenging is that many python ports do not correctly/precisely declare version-specs in their *_DEPENDS, so a simple 'grep of these' isn't sufficient, compounding the challenge is that many ports don't have their test suites hooked up so while builds may succeed, runtime can remain broken.

It's likely that an update to 2.8 first, and then subsequently to 2.9 and onward is the best approach.

Where people can help:

- If anyone wants to run a bulk run for a 2.8 update that would be handy
- If anyone can run through current cryptography dependents to identify any first-order incompatible-with-2.8+ ports, that would be handy.
Comment 6 Alessandro Sagratini 2020-06-19 15:15:04 UTC
*** Bug 247403 has been marked as a duplicate of this bug. ***
Comment 7 daniel.engberg.lists 2020-07-01 09:13:20 UTC
Looking at repology the majority of "recently" released distros they're using 2.8+
( https://repology.org/project/python:cryptography/versions ) so breakage isn't as bad as we think?
lwhsu@ suggested on irc that we could copy the current version and keep it as backup if we run into issues as an alternative solution for troublesome ports.
Comment 8 daniel.engberg.lists 2020-07-23 07:02:48 UTC
3.0 is out, depends on py-cryptography-vectors 3.0
Comment 9 Kai Knoblich freebsd_committer 2020-08-02 08:41:22 UTC
(In reply to Kubilay Kocak from comment #5)

Here's an overview of the actual consumers and the QA results of security/py-cryptography:

> Port                                  Status  Source          Remarks
> ------------------------------------- ------  --------------  -------------------------------------------------------
> cad/uranium                           OK      N/A             Not defined, but needed by Uranium-8d2bada/UM/Trust.py
> databases/py-mycli                    OK      >=1.0.0         setup.py
> databases/py-sqlalchemy-utils         OK      >=0.6           setup.py
> devel/py-adb                          OK      N/A             Noted as 'cryptography' in setup.py
> devel/py-aiortc                       OK      >=2.2           setup.py
> devel/py-apns2                        OK      >=1.7.2         setup.py
> devel/py-azure-keyvault               OK      >=2.1.4         setup.py
> devel/py-azure-multiapi-storage       OK      N/A             Noted as 'cryptography' in setup.py
> devel/py-azure-storage-common         OK      N/A             Noted as 'cryptography' in setup.py
> devel/py-castellan                    OK      >=2.1           requirements.txt
> devel/py-castellan1                   OK      >=2.1           requirements.txt
> devel/py-cursive                      OK      !=1.3.0,>=1.0   requirements.txt
> devel/py-fabric                       OK      N/A             Only referenced in documentation, seems not to be required by code
> devel/py-oci                          OK      ==2.8           Pinned requirements aren't checked at runtime, seems to work, should be patched out
> devel/py-openstacksdk                 OK      >=2.1           requirements.txt
> devel/py-openstacksdk043              OK      >=2.1           requirements.txt
> devel/py-twisted                      OK      >=2.5           src/twisted/python/_setup.py
> dns/py-dns-lexicon                    OK      N/A             Noted as 'cryptography' in setup.py
> finance/py-stripe                     OK      N/A             Not referenced in any file, might be superfluous
> misc/py-cinder                        OK      !=2.0,>=1.9     requirements.txt
> net-mgmt/py-adal                      OK      >=1.1.0         setup.py
> net/py-ripe.atlas.sagan               OK      N/A             Noted as 'cryptography' in setup.py
> net/py-transip                        OK      N/A             Noted as 'cryptography' in setup.py
> net/py-urllib3                        OK      >=1.3.4         setup.py
> news/sabnzbdplus                      OK      >=1.0           INSTALL.txt
> security/cowrie                       OK      >=0.9.1         setup.py
> security/py-SecretStorage             OK      N/A             Noted as 'cryptography' in setup.py
> security/py-acme                      OK      >=1.2.3         setup.py
> security/py-asyncssh                  OK      >=2.6.1         setup.py
> security/py-authlib                   OK      N/A             Noted as 'cryptography' in setup.py
> security/py-certbot                   OK      >=1.2.3         setup.py
> security/py-dfvfs                     OK      >=2.0.2         setup.cfg
> security/py-fido2                     OK      >=1.5           setup.py
> security/py-josepy                    OK      >=0.8           setup.py
> security/py-keystone                  OK      !=2.0,>=1.9     requirements.txt
> security/py-msoffcrypto-tool          OK      >=2.3           setup.py
> security/py-oauthlib                  OK      N/A             Noted as 'cryptography' in setup.py
> security/py-openssl                   OK      >=2.3           src/pyOpenSSL.egg-info/requires.txt (in setup.py as >= 2.2.1)
> security/py-paramiko                  OK      >=2.5           setup.py
> security/py-paramiko1                 OK      N/A             Seems not to be required
> security/py-pgpy                      OK      >=2.6           setup.py
> security/py-pysaml2                   OK      >=1.4           setup.cfg
> security/py-pysaml24                  OK      >=1.4           setup.cfg
> security/py-python-axolotl            OK      N/A             Noted as 'cryptography' in setup.py
> security/py-requests-credssp          OK      N/A             Noted as 'cryptography' in setup.py
> security/py-securesystemslib          OK      >=2.2.2         setup.py
> security/py-service_identity          OK      N/A             Noted as 'cryptography' in setup.py
> security/py-social-auth-core          OK      >=1.4           requirements-base.txt
> security/py-trustme                   OK      N/A             Noted as 'cryptography' in setup.py
> security/py-txtorcon                  OK      N/A             requirements.txt
> sysutils/ansible                      OK      N/A             Noted as 'cryptography' in setup.py
> sysutils/py-azure-cli                 OK      >=2.3.1,<3.0.0  setup.py
> www/buku                              OK      >=1.2.3         setup.py
> www/mitmproxy                         OK      >=2.1.4,<2.4    setup.py (patched out)
> www/py-aiohttp                        OK      N/A             Not referenced in code, listed as TEST_DEPENDS
> www/py-azure-storage                  OK      N/A             Noted as 'cryptography' in setup.py
> www/py-flask-jwt-extended             OK      >=2.3           setup.py
> www/py-pyjwt                          OK      >=1.4           setup.py
> www/py-requests_ntlm                  OK      >=1.3           setup.py
> x11/xpra                              OK      N/A             Noted as 'cryptography' in setup.py

A poudriere bulk run (11.3, 11.4, 12.1-RELEASE, 13.0-CURRENT@r363689) against all consumers listed from above gave no fallouts.

Results of the testsuite via "make test":

11.3-RELEASE / OpenSSL 1.0.2s:
- py27:
  97987 passed, 9101 skipped, 3 warnings in NNN seconds [amd64, i386]

- py35, py36, py37, py38:
  97987 passed, 9101 skipped, 26 warnings in NNN seconds [amd64, i386]

11.4-RELEASE / OpenSSL 1.0.2u:
- py27:
  1 failed, 97986 passed, 9101 skipped, 3 warnings in NNN seconds [amd64, i386]

- py35, py36, py37, py38:
  1 failed, 97986 passed, 9101 skipped, 26 warnings in NNN seconds [amd64, i386]

The test "TestECDSACertificate.test_load_ecdsa_no_named_curve" fails with:

> self = <tests.x509.test_x509.TestECDSACertificate object at 0x859017810>, backend = <cryptography.hazmat.backends.openssl.backend.Backend object at 0x8026c3450>
>     def test_load_ecdsa_no_named_curve(self, backend):
>         _skip_curve_unsupported(backend, ec.SECP256R1())
>         cert = _load_cert(                                                                                                                                                   
>             os.path.join("x509", "custom", "ec_no_named_curve.pem"),
>             x509.load_pem_x509_certificate,
>             backend
>         )
>         with pytest.raises(NotImplementedError):
> >           cert.public_key()
> E           Failed: DID NOT RAISE <class 'NotImplementedError'>
> tests/x509/test_x509.py:4133: Failed

This is because OpenSSL 1.0.2t got some curve matching parameter backported and the checks assume at least OpenSSL 1.1.0+ for this feature. Upstream of security/py-cryptography recently added a workaround to cope with OpenSSL 1.0.2t/u in that situation.

12.1-RELEASE / OpenSSL 1.1.1d:
- py27:
  103849 passed, 3239 skipped, 3 warnings in NNN seconds [amd64]
  103847 passed, 3241 skipped, 3 warnings in NNN seconds [i386]

- py35, py36, py37, py38
  103849 passed, 3239 skipped, 26 warnings in NNN seconds [amd64]
  103847 passed, 3241 skipped, 26 warnings in NNN seconds [i386]

13.0-CURRENT@r363689 [amd64] / OpenSSL 1.1.1g:
- py27:
  103849 passed, 3239 skipped, 3 warnings in NNN seconds [amd64]
  103847 passed, 3241 skipped, 3 warnings in NNN seconds [i386]

- py35, py36, py37, py38
  103849 passed, 3239 skipped, 26 warnings in NNN seconds [amd64]
  103847 passed, 3241 skipped, 26 warnings in NNN seconds [i386]
Comment 10 Kai Knoblich freebsd_committer 2020-08-02 08:53:07 UTC
The upgrade path of security/py-cryptography also contains a few backwards incompatible changes as already noted in comment #5.

Those changes should cause no problems to its consumers and because there are no bug/security fixes it isn't a MFH candidate:

- Support for Python 3.4 has been removed.
        -> Python 3.4 isn't present in the Ports tree
        => OK

- Support for OpenSSL 1.0.1 has been removed
        -> FreeBSD 11.3-RELEASE uses OpenSSL 1.0.2s
        -> FreeBSD 11.4-RELEASE uses OpenSSL 1.0.2u
        -> FreeBSD 12.1-RELEASE uses OpenSSL 1.1.1d
        -> FreeBSD 13.0@r361498 uses OpenSSL 1.1.1g
        => OK

- Support for LibreSSL 2.6.x has been removed
        -> security/libressl is at 3.1.3 in the Ports tree
        => OK

- Reversed the order in which "cryptography.x509.Name.rfc4514_string"
        -> security/py-trustme is the only consumer that uses the "rfc4514_string" method.
        -> test suite of security/py-trustme -> OK, 13 passed
        => OK

- No backwards incompatible changes

- Removed the "cryptography.hazmat.primitives.mac.MACContext" interface.
        -> No port uses the "cryptography.hazmat.primitives.mac.MACContext" interface.
        => OK

Given all those facts I'd suggest to land the release 2.9.2 of security/py-cryptography to have a new milestone.

(In reply to daniel.engberg.lists from comment #8)

The 3.0 release is still somewhat fresh and might break at least sysutils/py-azure-client (see also the related line in comment #9).

We should revise the situation in the next quarter and as far I can tell, support for OpenSSL 1.0.2 will be dropped in the 3.1 release of security/py-cryptography so we need to be careful regarding FreeBSD 11.3/11.4 which use OpenSSL 1.0.2.

(In reply to daniel.engberg.lists from comment #7)

Indeed, for the case if there are problems once the 2.9.2 release is available in the Ports tree it should be not a problem to restore the older release by repo-copying it to security/py-cryptography26 and assign the problematic ports to it.
Comment 11 Kai Knoblich freebsd_committer 2020-08-02 09:03:48 UTC
Created attachment 216949 [details]

Here's an updated patch that is based on Daniel's original patch:

The updated patch adds the workaround for OpenSSL 1.0.2u/t if the package is built for FreeBSD 11.3-STABLE and 11.4-RELEASE/STABLE.

It also keeps the line of the TEST_DEPENDS for security/py-cryptography-vectors unchanged as it's in the current state a good indicator to check the version of that port as well and update it, if required.

With the backported workaround the results of "make test" with FreeBSD 11.4 are:

97987 passed, 9101 skipped, 3 warnings in 370.64 seconds [amd64, i386]

py35, py36, py37, py38:
97987 passed, 9101 skipped, 26 warnings in 424.51 seconds [amd64, i386]

Please let me know if you have ideas or suggestions.  If everyone is fine with the patch then security/py-cryptography should be updated soon.
Comment 12 Kai Knoblich freebsd_committer 2020-08-02 12:10:24 UTC
(In reply to Kai Knoblich from comment #11)

P.S.: The attached patch contains at line #65 a commented out line which is a remnant of a testing session.  I already removed that line in my local repository.
Comment 13 daniel.engberg.lists 2020-08-02 13:31:06 UTC
Great news, thanks for all your work on this!
Comment 14 daniel.engberg.lists 2020-08-25 13:24:06 UTC
Any news on progress?
Comment 15 Matthias Andree freebsd_committer 2020-10-11 13:51:04 UTC
(In reply to daniel.engberg.lists from comment #14)
Koobs, Kai@ being silent, what needs to happen to get this moving again?