|Summary:||security/py-cryptography: Update to 2.3 (Fixes build with libressl* 2.7)|
|Product:||Ports & Packages||Reporter:||Charlie Li <ml+freebsd>|
|Component:||Individual Port(s)||Assignee:||Kubilay Kocak <koobs>|
|Severity:||Affects Many People||CC:||amdmi3, andreas.sommer87, arneboeses, bpardini, brnrd, bugzilla.fbsd, cedric, crest, dewayne, freebsd, freebsd, freebsdbugs, georg-bsd, gessel, grembo, gspurki, hostmaster+freebsd, jakub_lach, jan, koobs, matthew, meka, nulani, paul, ports-secteam, romain, rozhuk.im, sseekamp, stl, vidar, w.schwarzenfeld|
|Bug Depends on:|
Description Charlie Li 2018-03-25 00:55:11 UTC
LibreSSL 2.7.0 was released on 21 March in security/libressl-devel and introduced support for many OpenSSL 1.0.2 and 1.1 APIs. However, this has broken building of this port when DEFAULT_VERSIONS contains ssl=libressl-devel. Upstream is actively working on a fix. This PR serves to coordinate downstream patching efforts before upstream releases their fix.
Comment 1 Bernard Spil 2018-03-25 10:22:44 UTC
Created attachment 191799 [details] svn diff for security/py-cryptography ``` security/py-cryptography: Fix build with LibreSSL 2.7 From: http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/security/py-cryptography/patches/ Author: sthen@OpenBSD.org PR: 226906 See also: https://github.com/pyca/cryptography/pull/4169 ``` QA: https://brnrd.eu/poudriere/data/111libre-default/2018-03-22_21h36m44s/logs/py27-cryptography-2.1.4.log
Comment 2 Kubilay Kocak 2018-03-28 03:02:59 UTC
(In reply to Bernard Spil from comment #1) Happy for this to land if it passes QA (incl make test (test suite)) Are the build failures upstream (see pull request) anything to worry about, or is it just related to their master (where this is a backport/custom patch to the current ports version) ?
Comment 3 Charlie Li 2018-03-28 03:17:34 UTC
It looks like upstream's CI stem from their master, but I don't think this is relevant. The bigger issue found in brnrd@'s follow-up pull request is that their CI is misconfigured , and while anyone can edit those configuration files and include them in pull requests, they have no effect unless one of their committers makes the edit (mere commits and merges don't count): > (Reminder that modifications to the jenkinsfile will not be honored for non-committers)  Example of python 3.6 run: https://travis-ci.org/pyca/cryptography/jobs/358012204
Comment 4 Bernard Spil 2018-04-08 10:42:04 UTC
Created attachment 192331 [details] Output of make test Not seeing issues when running make test > 90132 passed, 7403 skipped in 419.70 seconds
Comment 5 jakub_lach 2018-04-28 23:09:50 UTC
I was made aware of this, due to LibreSSL 2.7.0 being default now.
Comment 6 Bernard Spil 2018-04-29 19:38:28 UTC
(In reply to Kubilay Kocak from comment #2) Patch from OpenBSD only worked for LibreSSL 2.7, no longer for other versions. I've just sent a new pull-request to pyca/crypography with a version that retains compatibility with other libcrypto/ssl providers. https://github.com/pyca/cryptography/pull/4210
Comment 7 Tobias Kortkamp 2018-04-30 13:19:52 UTC
*** Bug 227852 has been marked as a duplicate of this bug. ***
Comment 8 Bernard Spil 2018-04-30 17:00:13 UTC
Created attachment 192933 [details] svn diff for security/py-cryptography This patch is the same as in https://github.com/pyca/cryptography/pull/4210
Comment 9 Bernard Spil 2018-04-30 17:36:45 UTC
I've run successful builds with Python 2.7 and 3.6 flavors on - 10.4 amd64 - 10.4 i386 - 11.1 amd64 - 11.1 i386 Using the following libcrypto providers - base - security/libressl - (skipped security/libressl-devel) - security/openssl - security/openssl-devel - security/openssl-master (1.1.1-pre5) Build logs can be found on https://keg.brnrd.eu/ and should be self-explanatory. Look for logs 2018-04-30 around 17:00 (all logs there are UTC) Upstream is running a verification build https://github.com/pyca/cryptography/pull/4211 for the patch that is attached.
Comment 10 Kubilay Kocak 2018-05-01 04:15:54 UTC
Comment on attachment 192933 [details] svn diff for security/py-cryptography Approved by: koobs (maintainer), Please add comments with upstream issue/commit references to any new patches. @Bernard, commit when you're comfortable/confident on QA
Comment 11 rozhuk.im 2018-05-03 17:44:50 UTC
Patch probably incomplete: install: security/py-openssl run: python import OpenSSL get: >>> import OpenSSL Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py", line 16, in <module> from OpenSSL._util import ( File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module> from cryptography.hazmat.bindings.openssl.binding import Binding File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 13, in <module> from cryptography.hazmat.bindings._openssl import ffi, lib ImportError: /usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so: Undefined symbol "DTLS_method"
Comment 12 Peter Putzer 2018-05-04 18:51:04 UTC
As the upstream fix seems to take longer, it would be good if the patch could be committed to the port to have a working system with the new libressl version.
Comment 13 Scott Larson 2018-05-04 20:53:18 UTC
(In reply to Peter Putzer from comment #12) As a quick workaround for anyone waiting on a permanent solution, grab the attached diff and drop it into a file at patch-issue4210 in /usr/ports or wherever your ports tree is located. Then just run `mkdir security/py-cryptography/files && patch -p0 < patch-issue4210 && rm security/py-cryptography/files/patch-issue4210.orig`.
Comment 14 Gian-Simon Purkert 2018-05-05 08:50:08 UTC
WOW, you saved my Day. Works with the patch, thank you very much.
Comment 15 FiLiS 2018-05-05 09:14:01 UTC
What are the problems with just committing the patch if it works?
Comment 16 Matthew Seaman 2018-05-05 09:21:04 UTC
Is there any problem with committing this patch? I can confirm it fixes my own package builds (with python26 and libressl set as defaults) -- and it seems most of the interesting package I want to build seem to depend on py-cryptography.
Comment 17 Matthew Seaman 2018-05-05 09:21:58 UTC
(In reply to Matthew Seaman from comment #16) s/python26/python36/, dammit.
Comment 18 Kubilay Kocak 2018-05-05 09:43:57 UTC
Comment 2 is clear on what is required: confirmation of QA. Specifically, this entails (but is not limited to): - confirming no regressions for all (other) values of ssl= - Ruling out comment 11 as a symptom of the patch (just replacing libressl with openssl, without recompiling dependents may be the cause). Additionally, there are indications that an update to the patch is required (which will come from the PR Bernard is working on upstream, if that is the case). And finally to clarify, the issue is not blocked on upstream accepting or merging the PR.
Comment 19 Kubilay Kocak 2018-05-05 09:46:11 UTC
Currently in Bernards capable hands
Comment 20 rozhuk.im 2018-05-06 02:00:42 UTC
It is build OK, but it does not work. See comment #11 for test case.
Comment 21 Bernard Spil 2018-05-06 12:09:22 UTC
(In reply to rozhuk.im from comment #20) That is a bit in line with the feedback from upstream. There are errors on DTLS_ variables as well. It takes me some time to cycle back to this problem.
Comment 22 FiLiS 2018-05-07 09:41:18 UTC
would it hurt to go back to 2.6.4 for security/libressl and leave 2.7 in security/libressl-devel for a bit longer, until those things are working?
Comment 23 Bernard Spil 2018-05-13 12:00:09 UTC
Created attachment 193350 [details] svn diff for security/py-cryptography Updated patch that resolves all 'implicit declaration' warnings at build and passes most tests during `make test`. Upstreamed in issue 4210 with additional changes for master, I guess upstream will merge and run another test in issue 4211.
Comment 24 Bernard Spil 2018-05-13 12:01:43 UTC
Created attachment 193351 [details] Output of make test Updated output of make test > 27 failed, 90118 passed, 7390 skipped
Comment 25 rozhuk.im 2018-05-21 11:29:18 UTC
Any progress with: https://github.com/pyca/cryptography/pull/4234 ?
Comment 26 gessel 2018-05-27 15:12:27 UTC
The patch seems stable (running in my various jails for month now, no problems). It would be helpful to get into ports.
Comment 27 Jan Siero 2018-05-29 07:22:35 UTC
I'm not sure if this is on-topic, but with the patched py-cryptography, the script py-certbot (for Let's Encrypt ssl certificates) runs into the following error: Undefined symbol "DTLS_method" Traceback (most recent call last): File "/usr/local/bin/certbot", line 11, in <module> load_entry_point('certbot==0.24.0', 'console_scripts', 'certbot')() File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 480, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2693, in load_entry_point return ep.load() File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2324, in load return self.resolve() File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2330, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/usr/local/lib/python3.6/site-packages/certbot/main.py", line 10, in <module> import josepy as jose File "/usr/local/lib/python3.6/site-packages/josepy/__init__.py", line 44, in <module> from josepy.interfaces import JSONDeSerializable File "/usr/local/lib/python3.6/site-packages/josepy/interfaces.py", line 8, in <module> from josepy import errors, util File "/usr/local/lib/python3.6/site-packages/josepy/util.py", line 4, in <module> import OpenSSL File "/usr/local/lib/python3.6/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/usr/local/lib/python3.6/site-packages/OpenSSL/crypto.py", line 16, in <module> from OpenSSL._util import ( File "/usr/local/lib/python3.6/site-packages/OpenSSL/_util.py", line 6, in <module> from cryptography.hazmat.bindings.openssl.binding import Binding File "/usr/local/lib/python3.6/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 13, in <module> from cryptography.hazmat.bindings._openssl import ffi, lib ImportError: /usr/local/lib/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: Undefined symbol "DTLS_method"
Comment 28 Jan Bramkamp 2018-05-29 09:25:40 UTC
(In reply to Jan Siero from comment #27) Afaik LibreSSL removed DTLS support.
Comment 29 Charlie Li 2018-05-31 23:40:10 UTC
Heads up: upstream finally merged in libressl support. https://github.com/pyca/cryptography/pull/4270
Comment 30 Kubilay Kocak 2018-06-01 05:46:12 UTC
*** Bug 228651 has been marked as a duplicate of this bug. ***
Comment 31 Kubilay Kocak 2018-06-01 05:48:05 UTC
https://github.com/pyca/cryptography/pull/4210 closed (in favour of): https://github.com/pyca/cryptography/pull/4270
Comment 32 Charlie Li 2018-06-01 07:08:02 UTC
Created attachment 193887 [details] 2.2.2 with upstream libressl support It's way past my bedtime, but here's a patch to the ports tree incorporating upstream's changes to support libressl. This is a simple diff between their master branch and version 2.2.2 on just the relevant src/_cffi_src/openssl directory. The delta between version 2.1.4 and upstream's libressl support was too great, actually refusing to build. Passes testport, but needs functional testing, as this is an unofficial backport.
Comment 33 Charlie Li 2018-06-02 16:40:52 UTC
security/py-certbot works with the upstream changes on top of 2.2.2. Tested on an armv6 system for a change.
Comment 34 Bernard Spil 2018-06-03 08:23:08 UTC
Hi koobs, I believe this update calls for an exp-run? I can create the PR for that, but will need your blessing on the current patch.
Comment 35 Bernard Spil 2018-06-03 10:34:47 UTC
Created attachment 193965 [details] svn diff for security/py-cryptography Using the patch I just added: LibreSSL + 2.7.15: 93566 passed, 7404 skipped, 51 warnings LibreSSL + 3.6.5: 93566 passed, 7404 skipped, 51 warnings base r1.0.2k + 2.7.15: 97623 passed, 3347 skipped, 51 warnings vishwin's patch: 93566 passed, 7404 skipped, 57
Comment 36 Kubilay Kocak 2018-06-08 09:40:17 UTC
(In reply to Bernard Spil from comment #34) Updates to this port don't require an exp-run. However, if the commit(s) to fix this issue require updates to libressl, then perhaps so, though that doesn't appear to be the case per attachment 193965 [details]
Comment 37 Kubilay Kocak 2018-06-08 09:42:50 UTC
Comment on attachment 193965 [details] svn diff for security/py-cryptography You have/had implicit approval for this change (fix) given the complexity of the background work to get it sorted. You now have explicit approval to handle resolution, with any commit (subsequent to QA of your satisfaction)
Comment 38 Bernard Spil 2018-06-09 09:05:29 UTC
Request for exp-run with patch from att 193965
Comment 39 Michael Gmelin 2018-07-06 12:34:10 UTC
As I didn't hear anything about exp-run feedback I took the liberty to set the exp-run flag. Also setting merge-quarterly, as we're currently stuck with 2018Q3 not building for our purposes. @Bernard Maybe you can give some update? Thanks!
Comment 40 Antoine Brodin 2018-07-06 12:41:04 UTC
python@ said an exp-run was not needed.
Comment 41 Michael Gmelin 2018-07-06 12:45:32 UTC
(In reply to Antoine Brodin from comment #40) Thanks, so what keeps us from committing and MFHing this to 2018Q3?
Comment 42 Kubilay Kocak 2018-07-11 05:27:35 UTC
Comment 36 (koobs) stated an exp-run was not *required* Comment 37 (koobs) provided approval "with any commit (subsequent to *QA of your satisfaction*) Comment 38 (brnrd) requested an exp-run Accordingly: - Restore assignee to correct last assignee (brnrd) - Cancel exp-run request (for now, to reset) @Bernard / Michael, *If* either of you feel an exp-run is necessary, please explicitly re-request it (assigning to portmgr@ as well) Otherwise, the change is approved (for either of you) pending your own QA satisfaction. Please assign yourselves (take issue) as necessary.
Comment 43 Kubilay Kocak 2018-07-11 05:29:10 UTC
*** Bug 229680 has been marked as a duplicate of this bug. ***
Comment 44 cedric 2018-07-23 08:47:30 UTC
py-cryptography 2.3 has been released with support for LibreSSL, can we update now?
Comment 45 Charlie Li 2018-07-23 09:25:28 UTC
The new version obviates the SSL patches, so update title. Currently running testport on the new version with all DEFAULT_VERSIONS settings to verify successful builds; will upload patch when done.
Comment 46 Charlie Li 2018-07-23 10:12:34 UTC
Created attachment 195386 [details] py-cryptography-2.3 No ssl patches, as this release fully supports both major implementations. Passes testport with all DEFAULT_VERSIONS settings, and py-certbot works fine as a runtime test.
Comment 47 Andreas Sommer 2018-07-23 13:33:15 UTC
(In reply to Michael Gmelin from comment #41) The question is whether an upgrade from 2.1.x to 2.2/2.3 would be eligible for MFH. It's probably not a trivial blanket build fix, since the minor version changes. So do the Python port experts here think a merge-back would be fine after this lands in head?
Comment 48 dewayne 2018-07-30 05:54:16 UTC
(In reply to Charlie Li from comment #46) Thank-you for the patch, which works nicely on FreeBSD 11.2-STABLE r336359M & libressl 2.7.4 :)
Comment 49 Goran Mekić 2018-08-02 00:42:16 UTC
*** Bug 230277 has been marked as a duplicate of this bug. ***
Comment 50 Kubilay Kocak 2018-08-05 23:44:15 UTC
*** Bug 229840 has been marked as a duplicate of this bug. ***
Comment 51 Kubilay Kocak 2018-08-06 02:57:26 UTC
QA'ing now. Fails to build against current openssl-devel (see bug 229223)
Comment 52 Kubilay Kocak 2018-08-06 03:10:46 UTC
2.3 also contains a security fix (with CVE)  VuXML addition pending  https://github.com/pyca/cryptography/pull/4360
Comment 53 commit-hook 2018-08-06 03:23:36 UTC
A commit references this bug: Author: koobs Date: Mon Aug 6 03:23:24 UTC 2018 New revision: 476487 URL: https://svnweb.freebsd.org/changeset/ports/476487 Log: security/py-cryptography: Add tag forgery vulnerability PR: 226906 Changes: head/security/vuxml/vuln.xml
Comment 54 commit-hook 2018-08-06 03:25:46 UTC
A commit references this bug: Author: koobs Date: Mon Aug 6 03:25:21 UTC 2018 New revision: 476488 URL: https://svnweb.freebsd.org/changeset/ports/476488 Log: security/py-cryptography: Update to 2.3 This version update fixes builds with libressl >= 2.7. Builds with openssl-devel are broken due to an underlying lang/python* port issue . Changelog: https://github.com/pyca/cryptography/blob/2.3/CHANGELOG.rst  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229223 PR: 226906 Submitted by: Charli Li <ml+freebsd vishwin info> Reported by: many MFH: 2018Q3 (fixes package build, security fix) Security: 9e2d0dcf-9926-11e8-a92d-0050562a4d7b Changes: head/security/py-cryptography/Makefile head/security/py-cryptography/distinfo
Comment 55 commit-hook 2018-08-22 02:15:25 UTC
A commit references this bug: Author: koobs Date: Wed Aug 22 02:15:06 UTC 2018 New revision: 477764 URL: https://svnweb.freebsd.org/changeset/ports/477764 Log: MFH: r476488 security/py-cryptography: Update to 2.3 This version update fixes builds with libressl >= 2.7. Builds with openssl-devel are broken due to an underlying lang/python* port issue . Changelog: https://github.com/pyca/cryptography/blob/2.3/CHANGELOG.rst  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229223 PR: 226906 Submitted by: Charli Li <ml+freebsd vishwin info> Reported by: many Security: 9e2d0dcf-9926-11e8-a92d-0050562a4d7b Approved by: ports-secteam (miwi) Changes: _U branches/2018Q3/ branches/2018Q3/security/py-cryptography/Makefile branches/2018Q3/security/py-cryptography/distinfo