Bug 229322
| Summary: | net/py-urllib3: Update to 1.25.6 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Patrice Clement <monsieurp> | ||||||
| Component: | Individual Port(s) | Assignee: | Kai Knoblich <kai> | ||||||
| Status: | Closed FIXED | ||||||||
| Severity: | Affects Some People | CC: | elastic, kai, koobs, monsieurp, ndowens04, sergey, sunpoet | ||||||
| Priority: | --- | Keywords: | security | ||||||
| Version: | Latest | Flags: | bugzilla:
maintainer-feedback?
(koobs) antoine: merge-quarterly- antoine: exp-run+ |
||||||
| Hardware: | Any | ||||||||
| OS: | Any | ||||||||
| See Also: | https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228770 | ||||||||
| Bug Depends on: | 236283, 239302, 241827, 241874, 241875 | ||||||||
| Bug Blocks: | 234994 | ||||||||
| Attachments: |
|
||||||||
|
Description
Patrice Clement
2018-06-24 23:46:38 UTC
Hi Here's the diff to update py-urllib3 to 1.23. Cheers, Doing this as part of py-requests update. I expect many failures. The following packages seem to depend on an earlier version of urllib3: - py*-requests - py*-pipenv - py*-pip - py*-elasticsearch5 - py*-elasticsearch *** Bug 229951 has been marked as a duplicate of this bug. *** Is there something I can do to help out here? (In reply to Patrice Clement from comment #6) The main area for QA blocking this update is identifying which reverse dependents of urllib in the ports tree won't work with >= 1.23. Subsequent to that, if the list is non zero, identifying upstream commits, released in newer versions or unreleased that add support for >= 1.23 The task is made more difficult because building/packaging successfully (either manually, or during an exp-run) is not sufficient to identify compatibility issues, as the vast majority of Python ports either do not (and/or cant) specify version restrictions in their *_DEPENDS lines that would trigger builds to fail, and/or do not have test targets that could (potentially) be run to produce pkg_resources.VersionConflict errors by setuptools, effectively testing run-time compatibility. A commit references this bug: Author: koobs Date: Tue Jan 22 10:46:12 UTC 2019 New revision: 490937 URL: https://svnweb.freebsd.org/changeset/ports/490937 Log: www/py-requests: Update to 2.21.0 - Update USES comment (Python 3.3 support dropped) - Rebase setup.py patch (idna change released) - Remove comment about failing tests due to httpbin issue which seems to now be fixed. This update includes a pinned urllib3 version bump to < 1.25, which paves the way for a net/urllib3 update to 1.24 [1]. Note: 2.20.0 includes a security vulnerability fix for CVE-2018-18074 Changelog: https://github.com/requests/requests/blob/v2.21.0/HISTORY.md PR: 229322 [1] Security: 50ad9a9a-1e28-11e9-98d7-0050562a4d7b MFH: 2019Q1 Changes: head/www/py-requests/Makefile head/www/py-requests/distinfo head/www/py-requests/files/patch-setup.py urllib3 < 1.23 has a similar (same?) vulnerability as requests < 2.20.0, who's update to 2.21.0 just landed in ports r490937 ... - https://github.com/urllib3/urllib3/issues/1316 - https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-20060 On a somewhat more positive note, after looking through all ports that depend on net/py-urllib3 (their upstream source code), the only ones that pin a max version of urllib3 are: ./www/py-requests: setup.py: 'urllib3>=1.21.1,<1.23' ./textproc/py-elasticsearch5: setup.py: 'urllib3<1.23,>=1.21.1', ./devel/py-botocore: setup.py: requires.append('urllib3>=1.20,<1.25') Of those, py-requests has bumped that to <1.24 as of 2.21.0 (already committed), and py-botocore version is above (1.25) what we'll be updating urllib3 to (1.24). That leaves textproc/py-elasticsearch5 (maintainer CC'd) ... I have a WIP patch to add QA TEST_DEPENDS/test target to py-elasticsearch5, which required switching the sources to GitHub. After patching out the the max version pin, the tests pass [1] after updating urllib3 to 1.24. Finally, with the last py-requests update and a WIP urllib3 1.24 update in place, cmake also does not regress (bug 228770) as expected. [1] ~103 tests pass. Tests that require an local/live elasticsearch server, which I don't have running, aren't run, but don't explicitly fail. For py-elasticsearch5, see also: https://github.com/elastic/elasticsearch-py/issues/807 https://github.com/elastic/elasticsearch-py/commit/7b8976a142ca2ea5685559d6c8345555d9ec6403 (released in >= 5.5.3) And (relevant historically) ... https://github.com/elastic/elasticsearch-py/issues/807 https://github.com/elastic/elasticsearch-py/issues/667 https://github.com/elastic/elasticsearch-py/issues/634 Scary. *** Bug 235261 has been marked as a duplicate of this bug. *** Will request an exp-run when ready. textproc/py-elasticsearch-curator is the only customer of the blocker (textproc/py-elasticsearch5). I've submitted bug #236283 to update it to 5.6.0 which no longer depends on textproc/py-elasticsearch5. After that, we could remove textproc/py-elasticsearch5 and request exp-run for py-urllib3 update. A commit references this bug: Author: koobs Date: Mon Mar 25 07:48:27 UTC 2019 New revision: 496799 URL: https://svnweb.freebsd.org/changeset/ports/496799 Log: MFH: r490937 www/py-requests: Update to 2.21.0 - Update USES comment (Python 3.3 support dropped) - Rebase setup.py patch (idna change released) - Remove comment about failing tests due to httpbin issue which seems to now be fixed. This update includes a pinned urllib3 version bump to < 1.25, which paves the way for a net/urllib3 update to 1.24 [1]. Note: 2.20.0 includes a security vulnerability fix for CVE-2018-18074 Changelog: https://github.com/requests/requests/blob/v2.21.0/HISTORY.md PR: 229322 [1] Security: 50ad9a9a-1e28-11e9-98d7-0050562a4d7b Approved by: ports-secteam (miwi) Changes: _U branches/2019Q1/ branches/2019Q1/www/py-requests/Makefile branches/2019Q1/www/py-requests/distinfo branches/2019Q1/www/py-requests/files/patch-setup.py I think it's time for another exp-run. A commit references this bug: Author: koobs Date: Tue Apr 16 04:06:27 UTC 2019 New revision: 499073 URL: https://svnweb.freebsd.org/changeset/ports/499073 Log: textproc/py-elasticsearch5: Remove pinned urllib3 version elasticsearch5 (this port) unnecessarily pins its urllib dependency to < 1.23, which blocks updating urllib3 to 1.24 [1]: ./textproc/py-elasticsearch5: setup.py: 'urllib3<1.23,>=1.21.1', The package had a history of issues/conflicts/bugs with the urllib3 dependency, ultimately resulting in the maximum version pin being removed [2]: https://github.com/elastic/elasticsearch-py/issues/807 https://github.com/elastic/elasticsearch-py/issues/667 https://github.com/elastic/elasticsearch-py/issues/634 This commit backports that change, a functional noop and sweeping change in advance required for a urllib3 update, and adds TEST_DEPENDS and a test target to support rigorous and confident QA. Switching to GitHub sources was required as the PyPI sdist does not package tests. The packages tests all pass with/against urllib3 1.24 installed, with an intermittent and non-deterministic off-by-one failure in one test: FAIL: test_all_chunks_sent (test_elasticsearch.test_helpers.TestParallelBulk) The issue exists independent of urllib3 version. The flaky test issue was reported upstream [3], but was not resolved. [2] https://github.com/elastic/elasticsearch-py/commit/4352e56174b77560d2f86801cb1ad32440bb2d32 [3] https://github.com/elastic/elasticsearch-py/issues/701 PR: 229322 [1] Approved by: portmgr (blanket: framework compliance, runtime bugfix) Changes: head/textproc/py-elasticsearch5/Makefile head/textproc/py-elasticsearch5/distinfo head/textproc/py-elasticsearch5/files/ head/textproc/py-elasticsearch5/files/patch-setup.py After ports r499073 (required to unblock update), a VuXML entry and further and final QA for the urllib3 WIP is pending, after which point I'll request an ex-run 1.24.2 is out. Please use this one instead. Thanks! (In reply to Sunpoet Po-Chuan Hsieh from comment #18) Will do, thanks! (In reply to Kubilay Kocak from comment #19) Hi, any progress on 1.24.2 or even latest 1.25.3? I need urllib3 1.24+ to unblock py-softlayer update. :) Created attachment 208975 [details] py-urllib-1.25.6.patch Attached is a new patch that updates net/py-urllib3 to 1.25.6. It contains also following modifications: - Convert the dependencies which are declared as extra dependencies in setup.py into OPTIONS - Set the options as default that were used by the previous RUN_DEPENDS - Update the TEST_DEPENDS and add a "do-test" target to make future QA easier - Remove the pkg-message, the related variable and patch as the info about the broken IPv6 support of net/py-socks (was broken with 1.5.7) is obsolete. - Remove the limitation for security/py-certifi. It has no Python version restriction in setup.py and it's more likely a remnant of the time when there were separate versions of www/py-urllib3. See ports r443069 for some details. - Separate USES block QA: ~~~ - poudriere (11.3-, 12.0, 12.1-RELEASE, 13.0-CURRENT@r353466 amd64) for each py27 + py36 flavor -> OK - "Mini" Exp-Runs with 11.3-, 12.0- and 12.1-RELEASE against all direct consumers of net/py-urllib3 and www/py-requests -> OK Results of "make test" with all tests enabled: 11.3-RELEASE, Python 3.6: > 1061 passed, 245 skipped, 121 warnings in 29.94 seconds 11.3-RELEASE, Python 2.7: > 1059 passed, 247 skipped, 86 warnings in 34.71 seconds 12.0-, 12.1-RELEASE, 13.0-CURRENT@r353466, Python 3.6: > 1 failed, 1130 passed, 175 skipped, 125 warnings in 51.85 seconds 12.0-, 12.1-RELEASE, 13.0-CURRENT@r353466, Python 2.7: > 1 failed, 1128 passed, 177 skipped, 86 warnings in 51.86 seconds - With FreeBSD 11.3 there are many skipped tests because of the OpenSSL version in base that has no TLSv3 support. - With FreeBSD >= 12.0 one test permanently fails (= "test_ssl_read_timeout") but IMHO this shouldn't be a blocker because that test also fails with net/py-urllib3 1.22. But I'll do some investigation why it fails but I already excluded that test in the attached patch. TODO: ~~~~~ - In-depth checking - Investigate why 'test_ssl_read_timeout' fails - Request an Exp-Run? Forgot to mention that the tests were all done with www/py-requests 2.22 (from bug #239302). A commit references this bug: Author: kai Date: Fri Nov 8 16:44:11 UTC 2019 New revision: 517078 URL: https://svnweb.freebsd.org/changeset/ports/517078 Log: www/py-requests: Update to 2.22.0 * Backport a patch from upstream that fixes the unittests in conjunction with devel/py-pytest >= 4. * Remove obsolete CONFLICTS_INSTALL entry as www/py-requests1 no longer exists in the Ports tree. This update includes a pinned urllib3 version bump to < 1.26, which clears the way for a net/urllib3 update to 1.25.6 [1]. Changelog: https://github.com/requests/requests/blob/v2.22.0/HISTORY.md PR: 239302, 229322 [1] Submitted by: swills (based on) Approved by: koobs (maintainer) MFH: 2019Q4 Changes: head/www/py-requests/Makefile head/www/py-requests/distinfo head/www/py-requests/files/patch-tests_test__utils.py A commit references this bug: Author: kai Date: Sun Nov 10 14:43:21 UTC 2019 New revision: 517209 URL: https://svnweb.freebsd.org/changeset/ports/517209 Log: MFH: r517078 www/py-requests: Update to 2.22.0 * Backport a patch from upstream that fixes the unittests in conjunction with devel/py-pytest >= 4. * Remove obsolete CONFLICTS_INSTALL entry as www/py-requests1 no longer exists in the Ports tree. This update includes a pinned urllib3 version bump to < 1.26, which clears the way for a net/urllib3 update to 1.25.6 [1]. Changelog: https://github.com/requests/requests/blob/v2.22.0/HISTORY.md PR: 239302, 229322 [1] Submitted by: swills (based on) Approved by: koobs (maintainer) Approved by: ports-secteam (joneum) Changes: _U branches/2019Q4/ branches/2019Q4/www/py-requests/Makefile branches/2019Q4/www/py-requests/distinfo branches/2019Q4/www/py-requests/files/patch-tests_test__utils.py A commit references this bug: Author: kai Date: Sun Nov 10 21:39:05 UTC 2019 New revision: 517227 URL: https://svnweb.freebsd.org/changeset/ports/517227 Log: textproc/py-transifex-client: Relax requirements for urllib3 * Prepare the port for use with urllib 1.25.x [1]. * Bump PORTREVISION for package change. PR: 229322 [1] Approved by: portmgr blanket (runtime bugfix) MFH: 2019Q4 (runtime bugfix blanket) Changes: head/textproc/py-transifex-client/Makefile head/textproc/py-transifex-client/files/patch-requirements.txt A commit references this bug: Author: kai Date: Sun Nov 10 21:40:47 UTC 2019 New revision: 517228 URL: https://svnweb.freebsd.org/changeset/ports/517228 Log: MFH: r517227 textproc/py-transifex-client: Relax requirements for urllib3 * Prepare the port for use with urllib 1.25.x [1]. * Bump PORTREVISION for package change. PR: 229322 [1] Approved by: portmgr blanket (runtime bugfix) Approved by: ports-secteam bugfix blanket Changes: _U branches/2019Q4/ branches/2019Q4/textproc/py-transifex-client/Makefile branches/2019Q4/textproc/py-transifex-client/files/patch-requirements.txt Created attachment 209275 [details] py-urllib3-1.25.6-v2.patch Renamed option SECURE to SSL in the updated patch. Here's an overview of ports that require net/py-urllib3: > Portname Required version Remarks > databases/py-carbon N/A Noted as 'urllib3' in setup.py > devel/py-botocore >=1.20,<1.26 setup.py > devel/py-minio N/A Noted as 'urllib3' in setup.py > devel/py-oslo.vmware >=1.21.1 requirements.txt > net-im/py-telepot >=1.9.1 setup.py > net-mgmt/seafile-client N/A Required by 'scripts/build/build-mac.py' > net-mgmt/seafile-server N/A Required by 'ci/utils.py' > net/py-softlayer >=1.22 setup.py > sysutils/duplicity-devel N/A Noted as 'urllib3' in requirements.txt > sysutils/py-azure-cli ~=1.18 setup.py > textproc/py-elasticsearch >=1.21.1 setup.py > textproc/py-pyes >=1.7 setup.py > textproc/py-transifex-client <1.26 setup.py > www/buku >=1.13.1 setup.py > www/ddgr N/A Code imports only urllib, urllib3 might not be required anymore > www/py-requests <1.26 setup.py > www/py-selenium N/A Noted as 'urllib3' in setup.py Following ports build "fine" so far, but are either already broken at runtime or will be if net/py-urllib3 1.25.6 lands: > Portname Required version Remarks > security/theonionbox >=1.24.2,<1.25 setup.py / Broken at runtime / Fix with bug #241827 > textproc/py-elasticsearch5 >=1.21.1<1.23 Patched out setup.py / Runtime fix with bug #241875 > textproc/py-elasticsearch6 >=1.21.1 setup.py / Runtime fix with bug #241874 (already committed with ports r517541, MFH pending) I did also some own exp-runs against direct and indirect consumers that were all successful so far. Once the three fixes listed above are committed the update for net/py-urllib3 might be ready to land. Asking portmgr@ if the update for net/py-urllib3 requires an exp-run. As already mentioned in comment #27 there are three ports that still require a fix. They build fine but are already broken or will be once net/py-urllib3 is updated to 1.25.6. A commit references this bug: Author: kai Date: Mon Nov 25 17:18:36 UTC 2019 New revision: 518410 URL: https://svnweb.freebsd.org/changeset/ports/518410 Log: textproc/py-elasticsearch5: Prepare for urllib3 >= 1.25 * Backport a patch from the 7.x branch of upstream repository that fixes a possible runtime issue with urllib3 1.25 [1] since that release verifies SSL certificates by default. Disabling SSL certificate verification via "verify_certs" in elasticsearch won't work then as expected thus set "cert_reqs=CERT_NONE" explicitly to restore that behavior. PR: 241875, 229322 [1] Approved by: maintainer timeout (elastic, 14 days) MFH: 2019Q4 Changes: head/textproc/py-elasticsearch5/Makefile head/textproc/py-elasticsearch5/files/patch-elasticsearch_connection_http__urllib3.py With ports r518410 all preparations from my side are now done for the /head branch to get net/py-urllib3 updated to 1.25.6. When my assumptions are correct there are two exp-runs running which are related to urllib3 at the moment (the label PR241621 is a somewhat confusing in that case): http://package23.nyi.freebsd.org/build.html?mastername=113i386-default-PR241624&build=2019-11-25_06h54m48s http://package22.nyi.freebsd.org/build.html?mastername=113amd64-default-PR241624&build=2019-11-25_10h11m45s I have one question/note regarding the "merge-quartely" flag that was set to "-" recently: I'm afraid that a MFH is required because the 1.25.6 release of urllib3 includes fixes for three CVEs (CVE-2018-20060, CVE-2019-11236 and CVE-2019-11324). I plan to commit a related VuXML entry in a few hours. At the moment I'm doing preparations and test-runs for the 2019Q4 branch but that still takes a little while. Maybe another exp-run for the 2019Q4 branch makes sense once urllib 1.25.6 lands in /head? (In reply to Kai Knoblich from comment #30) MFH is not approved, there are too much changes to make it work in the quaterly branch. Exp-run looks fine (only build time was tested, not run time) A commit references this bug: Author: kai Date: Tue Nov 26 11:51:31 UTC 2019 New revision: 518463 URL: https://svnweb.freebsd.org/changeset/ports/518463 Log: security/vuxml: Document net/py-urllib3 issues PR: 229322 Security: CVE-2018-20060 CVE-2019-11236 CVE-2019-11324 Changes: head/security/vuxml/vuln.xml A commit references this bug: Author: kai Date: Tue Nov 26 18:37:59 UTC 2019 New revision: 518476 URL: https://svnweb.freebsd.org/changeset/ports/518476 Log: net/py-urllib3: Update to 1.25.6 * Convert the RUN_DEPENDS into separate OPTIONS as they are listed as extra dependencies in setup.py. Also set those as default that contain the previous RUN_DEPENDS to allow a clean transition. * Remove the Python-specific version limitation for security/py-certifi because it's required for all Python versions. * Also remove the info about the broken IPv6 support of net/py-socks (was broken in 1.5.7) and the relevant patch as both are obsolete. * Update the TEST_DEPENDS and add a "do-test" target to make future QA easier. Please note that a MFH won't be done as it didn't get an approval because there are too much changes to make it work in the 2019Q4 branch. [1] Notable changes since 1.22: * Require and validate certificates by default when using HTTPS. * Add mitigation for BPO-37428 affecting Python < 3.7.4 and OpenSSL 1.1.1+ which caused certificate verification to be enabled when using "cert_reqs=CERT_NONE". * Add TLSv1.3 support to CPython, pyOpenSSL and SecureTransport "SSLContext" implementations. https://github.com/urllib3/urllib3/blob/1.25.6/CHANGES.rst Exp-run by: antoine PR: 229322 [1] Reported by: Patrice Clement <monsieurp@gentoo.org> Security: 87270ba5-03d3-11ea-b81f-3085a9a95629 Changes: head/UPDATING head/net/py-urllib3/Makefile head/net/py-urllib3/distinfo head/net/py-urllib3/files/patch-setup.py head/net/py-urllib3/files/pkg-message.in Technically this PR can be closed now as net/py-urllib3 is updated 1.25.6 in /head and no MFH will be done. I'll leave this PR a open for 1-2 weeks to in case there are some errors/regressions. (In reply to Antoine Brodin from comment #32) Thank you, Antoine, for the exp-run! Close this PR because the current quartely branch (= 2020Q1) has net/py-urlib 1.25.6 now and no issues were reported. |
