Bug 226906 - security/py-cryptography: Update to 2.3 (Fixes build with libressl* 2.7)
Summary: security/py-cryptography: Update to 2.3 (Fixes build with libressl* 2.7)
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: Normal Affects Many People
Assignee: Kubilay Kocak
URL: https://github.com/pyca/cryptography/...
Keywords: security
: 227852 228651 229680 229840 230277 (view as bug list)
Depends on:
Blocks: 226843
  Show dependency treegraph
 
Reported: 2018-03-25 00:55 UTC by Charlie Li
Modified: 2018-08-22 02:16 UTC (History)
31 users (show)

See Also:
koobs: maintainer-feedback+
koobs: merge-quarterly+


Attachments
svn diff for security/py-cryptography (5.53 KB, patch)
2018-03-25 10:22 UTC, Bernard Spil
no flags Details | Diff
Output of make test (221.52 KB, text/plain)
2018-04-08 10:42 UTC, Bernard Spil
no flags Details
svn diff for security/py-cryptography (8.73 KB, patch)
2018-04-30 17:00 UTC, Bernard Spil
koobs: maintainer-approval+
Details | Diff
svn diff for security/py-cryptography (10.94 KB, patch)
2018-05-13 12:00 UTC, Bernard Spil
no flags Details | Diff
Output of make test (408.81 KB, text/plain)
2018-05-13 12:01 UTC, Bernard Spil
no flags Details
2.2.2 with upstream libressl support (44.27 KB, patch)
2018-06-01 07:08 UTC, Charlie Li
no flags Details | Diff
svn diff for security/py-cryptography (8.18 KB, patch)
2018-06-03 10:34 UTC, Bernard Spil
no flags Details | Diff
py-cryptography-2.3 (2.13 KB, patch)
2018-07-23 10:12 UTC, Charlie Li
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Charlie Li freebsd_committer freebsd_triage 2018-03-25 00:55:11 UTC
LibreSSL 2.7.0 was released on 21 March in security/libressl-devel and introduced support for many OpenSSL 1.0.2 and 1.1 APIs. However, this has broken building of this port when DEFAULT_VERSIONS contains ssl=libressl-devel.

Upstream is actively working on a fix. This PR serves to coordinate downstream patching efforts before upstream releases their fix.
Comment 1 Bernard Spil freebsd_committer freebsd_triage 2018-03-25 10:22:44 UTC
Created attachment 191799 [details]
svn diff for security/py-cryptography

```
security/py-cryptography: Fix build with LibreSSL 2.7

From: http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/security/py-cryptography/patches/ 
Author: sthen@OpenBSD.org

PR: 226906
See also: https://github.com/pyca/cryptography/pull/4169
```

QA: https://brnrd.eu/poudriere/data/111libre-default/2018-03-22_21h36m44s/logs/py27-cryptography-2.1.4.log
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2018-03-28 03:02:59 UTC
(In reply to Bernard Spil from comment #1)

Happy for this to land if it passes QA (incl make test (test suite))

Are the build failures upstream (see pull request) anything to worry about, or is it just related to their master (where this is a backport/custom patch to the current ports version) ?
Comment 3 Charlie Li freebsd_committer freebsd_triage 2018-03-28 03:17:34 UTC
It looks like upstream's CI stem from their master, but I don't think this is relevant. The bigger issue found in brnrd@'s follow-up pull request is that their CI is misconfigured [1], and while anyone can edit those configuration files and include them in pull requests, they have no effect unless one of their committers makes the edit (mere commits and merges don't count):
> (Reminder that modifications to the jenkinsfile will not be honored for non-committers)

[1] Example of python 3.6 run: https://travis-ci.org/pyca/cryptography/jobs/358012204
Comment 4 Bernard Spil freebsd_committer freebsd_triage 2018-04-08 10:42:04 UTC
Created attachment 192331 [details]
Output of make test

Not seeing issues when running make test

> 90132 passed, 7403 skipped in 419.70 seconds
Comment 5 jakub_lach 2018-04-28 23:09:50 UTC
I was made aware of this, due to LibreSSL 2.7.0 being default now.
Comment 6 Bernard Spil freebsd_committer freebsd_triage 2018-04-29 19:38:28 UTC
(In reply to Kubilay Kocak from comment #2)
Patch from OpenBSD only worked for LibreSSL 2.7, no longer for other versions. I've just sent a new pull-request to pyca/crypography with a version that retains compatibility with other libcrypto/ssl providers.

https://github.com/pyca/cryptography/pull/4210
Comment 7 Tobias Kortkamp freebsd_committer freebsd_triage 2018-04-30 13:19:52 UTC
*** Bug 227852 has been marked as a duplicate of this bug. ***
Comment 8 Bernard Spil freebsd_committer freebsd_triage 2018-04-30 17:00:13 UTC
Created attachment 192933 [details]
svn diff for security/py-cryptography

This patch is the same as in https://github.com/pyca/cryptography/pull/4210
Comment 9 Bernard Spil freebsd_committer freebsd_triage 2018-04-30 17:36:45 UTC
I've run successful builds with Python 2.7 and 3.6 flavors on
 - 10.4 amd64
 - 10.4 i386
 - 11.1 amd64
 - 11.1 i386
Using the following libcrypto providers
 - base
 - security/libressl
 - (skipped security/libressl-devel)
 - security/openssl
 - security/openssl-devel
 - security/openssl-master (1.1.1-pre5)

Build logs can be found on https://keg.brnrd.eu/ and should be self-explanatory. Look for logs 2018-04-30 around 17:00 (all logs there are UTC)

Upstream is running a verification build https://github.com/pyca/cryptography/pull/4211 for the patch that is attached.
Comment 10 Kubilay Kocak freebsd_committer freebsd_triage 2018-05-01 04:15:54 UTC
Comment on attachment 192933 [details]
svn diff for security/py-cryptography

Approved by: koobs (maintainer),

Please add comments with upstream issue/commit references to any new patches.

@Bernard, commit when you're comfortable/confident on QA
Comment 11 Ivan Rozhuk 2018-05-03 17:44:50 UTC
Patch probably incomplete:
install: security/py-openssl
run: python
import OpenSSL

get:
>>> import OpenSSL
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py", line 16, in <module>
    from OpenSSL._util import (
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 13, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: /usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so: Undefined symbol "DTLS_method"
Comment 12 Peter Putzer 2018-05-04 18:51:04 UTC
As the upstream fix seems to take longer, it would be good if the patch could be committed to the port to have a working system with the new libressl version.
Comment 13 Scott Larson 2018-05-04 20:53:18 UTC
(In reply to Peter Putzer from comment #12)

As a quick workaround for anyone waiting on a permanent solution, grab the attached diff and drop it into a file at patch-issue4210 in /usr/ports or wherever your ports tree is located. Then just run `mkdir security/py-cryptography/files && patch -p0 < patch-issue4210 && rm security/py-cryptography/files/patch-issue4210.orig`.
Comment 14 Gian-Simon Purkert 2018-05-05 08:50:08 UTC
WOW, you saved my Day. 
Works with the patch, thank you very much.
Comment 15 FiLiS 2018-05-05 09:14:01 UTC
What are the problems with just committing the patch if it works?
Comment 16 Matthew Seaman freebsd_committer freebsd_triage 2018-05-05 09:21:04 UTC
Is there any problem with committing this patch?  I can confirm it fixes my own package builds (with python26 and libressl set as defaults) -- and it seems most of the interesting package I want to build seem to depend on py-cryptography.
Comment 17 Matthew Seaman freebsd_committer freebsd_triage 2018-05-05 09:21:58 UTC
(In reply to Matthew Seaman from comment #16)

s/python26/python36/, dammit.
Comment 18 Kubilay Kocak freebsd_committer freebsd_triage 2018-05-05 09:43:57 UTC
Comment 2 is clear on what is required: confirmation of QA. Specifically, this entails (but is not limited to):

- confirming no regressions for all (other) values of ssl=
- Ruling out comment 11 as a symptom of the patch (just replacing libressl with openssl, without recompiling dependents may be the cause).

Additionally, there are indications that an update to the patch is required (which will come from the PR Bernard is working on upstream, if that is the case).

And finally to clarify, the issue is not blocked on upstream accepting or merging the PR.
Comment 19 Kubilay Kocak freebsd_committer freebsd_triage 2018-05-05 09:46:11 UTC
Currently in Bernards capable hands
Comment 20 Ivan Rozhuk 2018-05-06 02:00:42 UTC
It is build OK, but it does not work.
See comment #11 for test case.
Comment 21 Bernard Spil freebsd_committer freebsd_triage 2018-05-06 12:09:22 UTC
(In reply to rozhuk.im from comment #20)
That is a bit in line with the feedback from upstream. There are errors on DTLS_ variables as well. It takes me some time to cycle back to this problem.
Comment 22 FiLiS 2018-05-07 09:41:18 UTC
would it hurt to go back to 2.6.4 for security/libressl and leave 2.7 in security/libressl-devel for a bit longer, until those things are working?
Comment 23 Bernard Spil freebsd_committer freebsd_triage 2018-05-13 12:00:09 UTC
Created attachment 193350 [details]
svn diff for security/py-cryptography

Updated patch that resolves all 'implicit declaration' warnings at build and passes most tests during `make test`. Upstreamed in issue 4210 with additional changes for master, I guess upstream will merge and run another test in issue 4211.
Comment 24 Bernard Spil freebsd_committer freebsd_triage 2018-05-13 12:01:43 UTC
Created attachment 193351 [details]
Output of make test

Updated output of make test

> 27 failed, 90118 passed, 7390 skipped
Comment 25 Ivan Rozhuk 2018-05-21 11:29:18 UTC
Any progress with: https://github.com/pyca/cryptography/pull/4234 ?
Comment 26 gessel 2018-05-27 15:12:27 UTC
The patch seems stable (running in my various jails for month now, no problems).  It would be helpful to get into ports.
Comment 27 Jan Siero 2018-05-29 07:22:35 UTC
I'm not sure if this is on-topic, but with the patched py-cryptography, the script py-certbot (for Let's Encrypt ssl certificates) runs into the following error: 
Undefined symbol "DTLS_method"

Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.24.0', 'console_scripts', 'certbot')()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2693, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2324, in load
    return self.resolve()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.6/site-packages/certbot/main.py", line 10, in <module>
    import josepy as jose
  File "/usr/local/lib/python3.6/site-packages/josepy/__init__.py", line 44, in <module>
    from josepy.interfaces import JSONDeSerializable
  File "/usr/local/lib/python3.6/site-packages/josepy/interfaces.py", line 8, in <module>
    from josepy import errors, util
  File "/usr/local/lib/python3.6/site-packages/josepy/util.py", line 4, in <module>
    import OpenSSL
  File "/usr/local/lib/python3.6/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/local/lib/python3.6/site-packages/OpenSSL/crypto.py", line 16, in <module>
    from OpenSSL._util import (
  File "/usr/local/lib/python3.6/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/usr/local/lib/python3.6/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 13, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: /usr/local/lib/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: Undefined symbol "DTLS_method"
Comment 28 Jan Bramkamp 2018-05-29 09:25:40 UTC
(In reply to Jan Siero from comment #27)

Afaik LibreSSL removed DTLS support.
Comment 29 Charlie Li freebsd_committer freebsd_triage 2018-05-31 23:40:10 UTC
Heads up: upstream finally merged in libressl support.

https://github.com/pyca/cryptography/pull/4270
Comment 30 Kubilay Kocak freebsd_committer freebsd_triage 2018-06-01 05:46:12 UTC
*** Bug 228651 has been marked as a duplicate of this bug. ***
Comment 32 Charlie Li freebsd_committer freebsd_triage 2018-06-01 07:08:02 UTC
Created attachment 193887 [details]
2.2.2 with upstream libressl support

It's way past my bedtime, but here's a patch to the ports tree incorporating upstream's changes to support libressl. This is a simple diff between their master branch and version 2.2.2 on just the relevant src/_cffi_src/openssl directory. The delta between version 2.1.4 and upstream's libressl support was too great, actually refusing to build.

Passes testport, but needs functional testing, as this is an unofficial backport.
Comment 33 Charlie Li freebsd_committer freebsd_triage 2018-06-02 16:40:52 UTC
security/py-certbot works with the upstream changes on top of 2.2.2. Tested on an armv6 system for a change.
Comment 34 Bernard Spil freebsd_committer freebsd_triage 2018-06-03 08:23:08 UTC
Hi koobs,

I believe this update calls for an exp-run?

I can create the PR for that, but will need your blessing on the current patch.
Comment 35 Bernard Spil freebsd_committer freebsd_triage 2018-06-03 10:34:47 UTC
Created attachment 193965 [details]
svn diff for security/py-cryptography

Using the patch I just added:
LibreSSL + 2.7.15: 93566 passed, 7404 skipped, 51 warnings
LibreSSL + 3.6.5:  93566 passed, 7404 skipped, 51 warnings
base r1.0.2k + 2.7.15:  97623 passed, 3347 skipped, 51 warnings

vishwin's patch: 93566 passed, 7404 skipped, 57
Comment 36 Kubilay Kocak freebsd_committer freebsd_triage 2018-06-08 09:40:17 UTC
(In reply to Bernard Spil from comment #34)

Updates to this port don't require an exp-run. However, if the commit(s) to fix this issue require updates to libressl, then perhaps so, though that doesn't appear to be the case per attachment 193965 [details]
Comment 37 Kubilay Kocak freebsd_committer freebsd_triage 2018-06-08 09:42:50 UTC
Comment on attachment 193965 [details]
svn diff for security/py-cryptography

You have/had implicit approval for this change (fix) given the complexity of the background work to get it sorted. You now have explicit approval to handle resolution, with any commit (subsequent to QA of your satisfaction)
Comment 38 Bernard Spil freebsd_committer freebsd_triage 2018-06-09 09:05:29 UTC
Request for exp-run with patch from att 193965
Comment 39 Michael Gmelin freebsd_committer freebsd_triage 2018-07-06 12:34:10 UTC
As I didn't hear anything about exp-run feedback I took the liberty to set the exp-run flag. Also setting merge-quarterly, as we're currently stuck with 2018Q3 not building for our purposes.

@Bernard Maybe you can give some update? Thanks!
Comment 40 Antoine Brodin freebsd_committer freebsd_triage 2018-07-06 12:41:04 UTC
python@ said an exp-run was not needed.
Comment 41 Michael Gmelin freebsd_committer freebsd_triage 2018-07-06 12:45:32 UTC
(In reply to Antoine Brodin from comment #40)

Thanks, so what keeps us from committing and MFHing this to 2018Q3?
Comment 42 Kubilay Kocak freebsd_committer freebsd_triage 2018-07-11 05:27:35 UTC
Comment 36 (koobs) stated an exp-run was not *required*

Comment 37 (koobs) provided approval "with any commit (subsequent to *QA of your satisfaction*)

Comment 38 (brnrd) requested an exp-run

Accordingly:

- Restore assignee to correct last assignee (brnrd)
- Cancel exp-run request (for now, to reset)

@Bernard / Michael,

*If* either of you feel an exp-run is necessary, please explicitly re-request it (assigning to portmgr@ as well)

Otherwise, the change is approved (for either of you) pending your own QA satisfaction. Please assign yourselves (take issue) as necessary.
Comment 43 Kubilay Kocak freebsd_committer freebsd_triage 2018-07-11 05:29:10 UTC
*** Bug 229680 has been marked as a duplicate of this bug. ***
Comment 44 cedric 2018-07-23 08:47:30 UTC
py-cryptography 2.3 has been released with support for LibreSSL, can we update now?
Comment 45 Charlie Li freebsd_committer freebsd_triage 2018-07-23 09:25:28 UTC
The new version obviates the SSL patches, so update title. Currently running testport on the new version with all DEFAULT_VERSIONS settings to verify successful builds; will upload patch when done.
Comment 46 Charlie Li freebsd_committer freebsd_triage 2018-07-23 10:12:34 UTC
Created attachment 195386 [details]
py-cryptography-2.3

No ssl patches, as this release fully supports both major implementations. Passes testport with all DEFAULT_VERSIONS settings, and py-certbot works fine as a runtime test.
Comment 47 Andreas Sommer 2018-07-23 13:33:15 UTC
(In reply to Michael Gmelin from comment #41)
The question is whether an upgrade from 2.1.x to 2.2/2.3 would be eligible for MFH. It's probably not a trivial blanket build fix, since the minor version changes. So do the Python port experts here think a merge-back would be fine after this lands in head?
Comment 48 dewayne 2018-07-30 05:54:16 UTC
(In reply to Charlie Li from comment #46)
Thank-you for the patch, which works nicely on FreeBSD 11.2-STABLE  r336359M & libressl 2.7.4 :)
Comment 49 Goran Mekić 2018-08-02 00:42:16 UTC
*** Bug 230277 has been marked as a duplicate of this bug. ***
Comment 50 Kubilay Kocak freebsd_committer freebsd_triage 2018-08-05 23:44:15 UTC
*** Bug 229840 has been marked as a duplicate of this bug. ***
Comment 51 Kubilay Kocak freebsd_committer freebsd_triage 2018-08-06 02:57:26 UTC
QA'ing now. Fails to build against current openssl-devel (see bug 229223)
Comment 52 Kubilay Kocak freebsd_committer freebsd_triage 2018-08-06 03:10:46 UTC
2.3 also contains a security fix (with CVE) [1]

VuXML addition pending

[1] https://github.com/pyca/cryptography/pull/4360
Comment 53 commit-hook freebsd_committer freebsd_triage 2018-08-06 03:23:36 UTC
A commit references this bug:

Author: koobs
Date: Mon Aug  6 03:23:24 UTC 2018
New revision: 476487
URL: https://svnweb.freebsd.org/changeset/ports/476487

Log:
  security/py-cryptography: Add tag forgery vulnerability

  PR:	226906

Changes:
  head/security/vuxml/vuln.xml
Comment 54 commit-hook freebsd_committer freebsd_triage 2018-08-06 03:25:46 UTC
A commit references this bug:

Author: koobs
Date: Mon Aug  6 03:25:21 UTC 2018
New revision: 476488
URL: https://svnweb.freebsd.org/changeset/ports/476488

Log:
  security/py-cryptography: Update to 2.3

  This version update fixes builds with libressl >= 2.7.

  Builds with openssl-devel are broken due to an underlying lang/python* port
  issue [1].

  Changelog:

    https://github.com/pyca/cryptography/blob/2.3/CHANGELOG.rst

  [1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229223

  PR:             226906
  Submitted by:   Charli Li <ml+freebsd vishwin info>
  Reported by:    many
  MFH:            2018Q3 (fixes package build, security fix)
  Security:	9e2d0dcf-9926-11e8-a92d-0050562a4d7b

Changes:
  head/security/py-cryptography/Makefile
  head/security/py-cryptography/distinfo
Comment 55 commit-hook freebsd_committer freebsd_triage 2018-08-22 02:15:25 UTC
A commit references this bug:

Author: koobs
Date: Wed Aug 22 02:15:06 UTC 2018
New revision: 477764
URL: https://svnweb.freebsd.org/changeset/ports/477764

Log:
  MFH: r476488 security/py-cryptography: Update to 2.3

  This version update fixes builds with libressl >= 2.7.

  Builds with openssl-devel are broken due to an underlying lang/python* port
  issue [1].

  Changelog:

    https://github.com/pyca/cryptography/blob/2.3/CHANGELOG.rst

  [1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229223

  PR:             226906
  Submitted by:   Charli Li <ml+freebsd vishwin info>
  Reported by:    many
  Security:	9e2d0dcf-9926-11e8-a92d-0050562a4d7b

  Approved by:	ports-secteam (miwi)

Changes:
_U  branches/2018Q3/
  branches/2018Q3/security/py-cryptography/Makefile
  branches/2018Q3/security/py-cryptography/distinfo