Bug 229322 - net/py-urllib3: Update to 1.25.6
Summary: net/py-urllib3: Update to 1.25.6
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Kai Knoblich
URL:
Keywords: needs-qa, security
: 229951 235261 (view as bug list)
Depends on: 236283 239302 241827 241874 241875
Blocks: 234994
  Show dependency treegraph
 
Reported: 2018-06-24 23:46 UTC by Patrice Clement
Modified: 2019-11-26 18:40 UTC (History)
7 users (show)

See Also:
bugzilla: maintainer-feedback? (koobs)
antoine: merge-quarterly-
antoine: exp-run+


Attachments
py-urllib-1.25.6.patch (4.28 KB, patch)
2019-11-08 15:29 UTC, Kai Knoblich
no flags Details | Diff
py-urllib3-1.25.6-v2.patch (4.28 KB, patch)
2019-11-20 10:34 UTC, Kai Knoblich
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Patrice Clement 2018-06-24 23:46:38 UTC

    
Comment 1 Patrice Clement 2018-06-24 23:47:18 UTC
Hi

Here's the diff to update py-urllib3 to 1.23.

Cheers,
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2018-06-25 04:14:32 UTC
Doing this as part of py-requests update.
Comment 3 Antoine Brodin freebsd_committer 2018-06-25 05:12:54 UTC
I expect many failures.
Comment 4 Antoine Brodin freebsd_committer 2018-06-25 08:17:37 UTC
The following packages seem to depend on an earlier version of urllib3:
- py*-requests
- py*-pipenv
- py*-pip
- py*-elasticsearch5
- py*-elasticsearch
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2018-07-23 05:08:50 UTC
*** Bug 229951 has been marked as a duplicate of this bug. ***
Comment 6 Patrice Clement 2018-07-25 17:12:48 UTC
Is there something I can do to help out here?
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2018-08-01 05:25:26 UTC
(In reply to Patrice Clement from comment #6)

The main area for QA blocking this update is identifying which reverse dependents of urllib in the ports tree won't work with >= 1.23. Subsequent to that, if the list is non zero, identifying upstream commits, released in newer versions or unreleased that add support for >= 1.23

The task is made more difficult because building/packaging successfully (either manually, or during an exp-run) is not sufficient to identify compatibility issues, as the vast majority of Python ports either do not (and/or cant) specify version restrictions in their *_DEPENDS lines that would trigger builds to fail, and/or do not have test targets that could (potentially) be run to produce pkg_resources.VersionConflict errors by setuptools, effectively testing run-time compatibility.
Comment 8 commit-hook freebsd_committer 2019-01-22 10:47:02 UTC
A commit references this bug:

Author: koobs
Date: Tue Jan 22 10:46:12 UTC 2019
New revision: 490937
URL: https://svnweb.freebsd.org/changeset/ports/490937

Log:
  www/py-requests: Update to 2.21.0

   - Update USES comment (Python 3.3 support dropped)
   - Rebase setup.py patch (idna change released)
   - Remove comment about failing tests due to httpbin issue which seems
     to now be fixed.

  This update includes a pinned urllib3 version bump to < 1.25, which paves
  the way for a net/urllib3 update to 1.24 [1].

  Note: 2.20.0 includes a security vulnerability fix for CVE-2018-18074

  Changelog:

    https://github.com/requests/requests/blob/v2.21.0/HISTORY.md

  PR: 		229322 [1]
  Security:	50ad9a9a-1e28-11e9-98d7-0050562a4d7b
  MFH:		2019Q1

Changes:
  head/www/py-requests/Makefile
  head/www/py-requests/distinfo
  head/www/py-requests/files/patch-setup.py
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2019-01-22 11:14:18 UTC
urllib3 < 1.23 has a similar (same?) vulnerability as requests < 2.20.0, who's update to 2.21.0 just landed in ports r490937 ...

 - https://github.com/urllib3/urllib3/issues/1316
 - https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-20060

On a somewhat more positive note, after looking through all ports that depend on net/py-urllib3 (their upstream source code), the only ones that pin a max version of urllib3 are: 

./www/py-requests: setup.py: 'urllib3>=1.21.1,<1.23'
./textproc/py-elasticsearch5: setup.py: 'urllib3<1.23,>=1.21.1',
./devel/py-botocore: setup.py: requires.append('urllib3>=1.20,<1.25')

Of those, py-requests has bumped that to <1.24 as of 2.21.0 (already committed), and py-botocore version is above (1.25) what we'll be updating urllib3 to (1.24).

That leaves textproc/py-elasticsearch5 (maintainer CC'd) ...

I have a WIP patch to add QA TEST_DEPENDS/test target to py-elasticsearch5, which required switching the sources to GitHub. After patching out the the max version pin, the tests pass [1] after updating urllib3 to 1.24.

Finally, with the last py-requests update and a WIP urllib3 1.24 update in place, cmake also does not regress (bug 228770) as expected.

[1] ~103 tests pass. Tests that require an local/live elasticsearch server, which I don't have running, aren't run, but don't explicitly fail.
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2019-01-28 03:11:45 UTC
*** Bug 235261 has been marked as a duplicate of this bug. ***
Comment 12 Kubilay Kocak freebsd_committer freebsd_triage 2019-01-28 03:12:34 UTC
Will request an exp-run when ready.
Comment 13 Sunpoet Po-Chuan Hsieh freebsd_committer 2019-03-05 17:13:00 UTC
textproc/py-elasticsearch-curator is the only customer of the blocker (textproc/py-elasticsearch5). I've submitted bug #236283 to update it to 5.6.0 which no longer depends on textproc/py-elasticsearch5. After that, we could remove textproc/py-elasticsearch5 and request exp-run for py-urllib3 update.
Comment 14 commit-hook freebsd_committer 2019-03-25 07:49:15 UTC
A commit references this bug:

Author: koobs
Date: Mon Mar 25 07:48:27 UTC 2019
New revision: 496799
URL: https://svnweb.freebsd.org/changeset/ports/496799

Log:
  MFH: r490937 www/py-requests: Update to 2.21.0

   - Update USES comment (Python 3.3 support dropped)
   - Rebase setup.py patch (idna change released)
   - Remove comment about failing tests due to httpbin issue which seems
     to now be fixed.

  This update includes a pinned urllib3 version bump to < 1.25, which paves
  the way for a net/urllib3 update to 1.24 [1].

  Note: 2.20.0 includes a security vulnerability fix for CVE-2018-18074

  Changelog:

    https://github.com/requests/requests/blob/v2.21.0/HISTORY.md

  PR: 		229322 [1]
  Security:	50ad9a9a-1e28-11e9-98d7-0050562a4d7b

  Approved by:	ports-secteam (miwi)

Changes:
_U  branches/2019Q1/
  branches/2019Q1/www/py-requests/Makefile
  branches/2019Q1/www/py-requests/distinfo
  branches/2019Q1/www/py-requests/files/patch-setup.py
Comment 15 Sunpoet Po-Chuan Hsieh freebsd_committer 2019-03-30 08:36:56 UTC
I think it's time for another exp-run.
Comment 16 commit-hook freebsd_committer 2019-04-16 04:07:01 UTC
A commit references this bug:

Author: koobs
Date: Tue Apr 16 04:06:27 UTC 2019
New revision: 499073
URL: https://svnweb.freebsd.org/changeset/ports/499073

Log:
  textproc/py-elasticsearch5: Remove pinned urllib3 version

  elasticsearch5 (this port) unnecessarily pins its urllib dependency to
  < 1.23, which blocks updating urllib3 to 1.24 [1]:

  ./textproc/py-elasticsearch5: setup.py: 'urllib3<1.23,>=1.21.1',

  The package had a history of issues/conflicts/bugs with the urllib3
  dependency, ultimately resulting in the maximum version pin being
  removed [2]:

    https://github.com/elastic/elasticsearch-py/issues/807
    https://github.com/elastic/elasticsearch-py/issues/667
    https://github.com/elastic/elasticsearch-py/issues/634

  This commit backports that change, a functional noop and sweeping change
  in advance required for a urllib3 update, and adds TEST_DEPENDS and
  a test target to support rigorous and confident QA. Switching to GitHub
  sources was required as the PyPI sdist does not package tests.

  The packages tests all pass with/against urllib3 1.24 installed, with an
  intermittent and non-deterministic off-by-one failure in one test:

  FAIL: test_all_chunks_sent (test_elasticsearch.test_helpers.TestParallelBulk)

  The issue exists independent of urllib3 version. The flaky test issue was
  reported upstream [3], but was not resolved.

  [2] https://github.com/elastic/elasticsearch-py/commit/4352e56174b77560d2f86801cb1ad32440bb2d32
  [3] https://github.com/elastic/elasticsearch-py/issues/701

  PR:		229322 [1]
  Approved by:	portmgr (blanket: framework compliance, runtime bugfix)

Changes:
  head/textproc/py-elasticsearch5/Makefile
  head/textproc/py-elasticsearch5/distinfo
  head/textproc/py-elasticsearch5/files/
  head/textproc/py-elasticsearch5/files/patch-setup.py
Comment 17 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-16 04:09:55 UTC
After ports r499073 (required to unblock update), a VuXML entry and further and final QA for the urllib3 WIP is pending, after which point I'll request an ex-run
Comment 18 Sunpoet Po-Chuan Hsieh freebsd_committer 2019-04-18 17:44:03 UTC
1.24.2 is out. Please use this one instead. Thanks!
Comment 19 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-19 09:48:58 UTC
(In reply to Sunpoet Po-Chuan Hsieh from comment #18)

Will do, thanks!
Comment 20 Sunpoet Po-Chuan Hsieh freebsd_committer 2019-07-09 18:51:19 UTC
(In reply to Kubilay Kocak from comment #19)

Hi, any progress on 1.24.2 or even latest 1.25.3? I need urllib3 1.24+ to unblock py-softlayer update. :)
Comment 21 Kai Knoblich freebsd_committer 2019-11-08 15:29:49 UTC
Created attachment 208975 [details]
py-urllib-1.25.6.patch

Attached is a new patch that updates net/py-urllib3 to 1.25.6. It contains also following modifications:

- Convert the dependencies which are declared as extra dependencies in setup.py into OPTIONS
- Set the options as default that were used by the previous RUN_DEPENDS
- Update the TEST_DEPENDS and add a "do-test" target to make future QA easier
- Remove the pkg-message, the related variable and patch as the info about the broken IPv6 support of net/py-socks (was broken with 1.5.7) is obsolete.
- Remove the limitation for security/py-certifi. It has no Python version restriction in setup.py and it's more likely a remnant of the time when there were separate versions of www/py-urllib3. See ports r443069 for some details.
- Separate USES block

QA:
~~~
- poudriere (11.3-, 12.0, 12.1-RELEASE, 13.0-CURRENT@r353466 amd64) for each py27 + py36 flavor -> OK
- "Mini" Exp-Runs with 11.3-, 12.0- and 12.1-RELEASE against all direct consumers of net/py-urllib3 and www/py-requests -> OK

Results of "make test" with all tests enabled:

11.3-RELEASE, Python 3.6:
> 1061 passed, 245 skipped, 121 warnings in 29.94 seconds

11.3-RELEASE, Python 2.7:
> 1059 passed, 247 skipped, 86 warnings in 34.71 seconds

12.0-, 12.1-RELEASE, 13.0-CURRENT@r353466, Python 3.6:
> 1 failed, 1130 passed, 175 skipped, 125 warnings in 51.85 seconds

12.0-, 12.1-RELEASE, 13.0-CURRENT@r353466, Python 2.7: 
> 1 failed, 1128 passed, 177 skipped, 86 warnings in 51.86 seconds

- With FreeBSD 11.3 there are many skipped tests because of the OpenSSL version in base that has no TLSv3 support.

- With FreeBSD >= 12.0 one test permanently fails (= "test_ssl_read_timeout") but IMHO this shouldn't be a blocker because that test also fails with net/py-urllib3 1.22. But I'll do some investigation why it fails but I already excluded that test in the attached patch.


TODO:
~~~~~
- In-depth checking
- Investigate why 'test_ssl_read_timeout' fails
- Request an Exp-Run?
Comment 22 Kai Knoblich freebsd_committer 2019-11-08 15:51:19 UTC
Forgot to mention that the tests were all done with www/py-requests 2.22 (from bug #239302).
Comment 23 commit-hook freebsd_committer 2019-11-08 16:44:36 UTC
A commit references this bug:

Author: kai
Date: Fri Nov  8 16:44:11 UTC 2019
New revision: 517078
URL: https://svnweb.freebsd.org/changeset/ports/517078

Log:
  www/py-requests: Update to 2.22.0

  * Backport a patch from upstream that fixes the unittests in conjunction
    with devel/py-pytest >= 4.

  * Remove obsolete CONFLICTS_INSTALL entry as www/py-requests1 no longer
    exists in the Ports tree.

  This update includes a pinned urllib3 version bump to < 1.26, which clears
  the way for a net/urllib3 update to 1.25.6 [1].

  Changelog:

  https://github.com/requests/requests/blob/v2.22.0/HISTORY.md

  PR:		239302, 229322 [1]
  Submitted by:	swills (based on)
  Approved by:	koobs (maintainer)
  MFH:		2019Q4

Changes:
  head/www/py-requests/Makefile
  head/www/py-requests/distinfo
  head/www/py-requests/files/patch-tests_test__utils.py
Comment 24 commit-hook freebsd_committer 2019-11-10 14:43:58 UTC
A commit references this bug:

Author: kai
Date: Sun Nov 10 14:43:21 UTC 2019
New revision: 517209
URL: https://svnweb.freebsd.org/changeset/ports/517209

Log:
  MFH: r517078

  www/py-requests: Update to 2.22.0

  * Backport a patch from upstream that fixes the unittests in conjunction
    with devel/py-pytest >= 4.

  * Remove obsolete CONFLICTS_INSTALL entry as www/py-requests1 no longer
    exists in the Ports tree.

  This update includes a pinned urllib3 version bump to < 1.26, which clears
  the way for a net/urllib3 update to 1.25.6 [1].

  Changelog:

  https://github.com/requests/requests/blob/v2.22.0/HISTORY.md

  PR:		239302, 229322 [1]
  Submitted by:	swills (based on)
  Approved by:	koobs (maintainer)

  Approved by:	ports-secteam (joneum)

Changes:
_U  branches/2019Q4/
  branches/2019Q4/www/py-requests/Makefile
  branches/2019Q4/www/py-requests/distinfo
  branches/2019Q4/www/py-requests/files/patch-tests_test__utils.py
Comment 25 commit-hook freebsd_committer 2019-11-10 21:39:31 UTC
A commit references this bug:

Author: kai
Date: Sun Nov 10 21:39:05 UTC 2019
New revision: 517227
URL: https://svnweb.freebsd.org/changeset/ports/517227

Log:
  textproc/py-transifex-client: Relax requirements for urllib3

  * Prepare the port for use with urllib 1.25.x [1].

  * Bump PORTREVISION for package change.

  PR:		229322 [1]
  Approved by:	portmgr blanket (runtime bugfix)
  MFH:		2019Q4 (runtime bugfix blanket)

Changes:
  head/textproc/py-transifex-client/Makefile
  head/textproc/py-transifex-client/files/patch-requirements.txt
Comment 26 commit-hook freebsd_committer 2019-11-10 21:41:33 UTC
A commit references this bug:

Author: kai
Date: Sun Nov 10 21:40:47 UTC 2019
New revision: 517228
URL: https://svnweb.freebsd.org/changeset/ports/517228

Log:
  MFH: r517227

  textproc/py-transifex-client: Relax requirements for urllib3

  * Prepare the port for use with urllib 1.25.x [1].

  * Bump PORTREVISION for package change.

  PR:		229322 [1]
  Approved by:	portmgr blanket (runtime bugfix)

  Approved by:	ports-secteam bugfix blanket

Changes:
_U  branches/2019Q4/
  branches/2019Q4/textproc/py-transifex-client/Makefile
  branches/2019Q4/textproc/py-transifex-client/files/patch-requirements.txt
Comment 27 Kai Knoblich freebsd_committer 2019-11-20 10:34:49 UTC
Created attachment 209275 [details]
py-urllib3-1.25.6-v2.patch

Renamed option SECURE to SSL in the updated patch.

Here's an overview of ports that require net/py-urllib3:

> Portname                      Required version        Remarks
> databases/py-carbon           N/A                     Noted as 'urllib3' in setup.py
> devel/py-botocore             >=1.20,<1.26            setup.py
> devel/py-minio                N/A                     Noted as 'urllib3' in setup.py
> devel/py-oslo.vmware          >=1.21.1                requirements.txt
> net-im/py-telepot             >=1.9.1                 setup.py
> net-mgmt/seafile-client       N/A                     Required by 'scripts/build/build-mac.py'
> net-mgmt/seafile-server       N/A                     Required by 'ci/utils.py'
> net/py-softlayer              >=1.22                  setup.py
> sysutils/duplicity-devel      N/A                     Noted as 'urllib3' in requirements.txt
> sysutils/py-azure-cli         ~=1.18                  setup.py
> textproc/py-elasticsearch     >=1.21.1                setup.py
> textproc/py-pyes              >=1.7                   setup.py
> textproc/py-transifex-client  <1.26                   setup.py
> www/buku                      >=1.13.1                setup.py
> www/ddgr                      N/A                     Code imports only urllib, urllib3 might not be required anymore
> www/py-requests               <1.26                   setup.py
> www/py-selenium               N/A                     Noted as 'urllib3' in setup.py

Following ports build "fine" so far, but are either already broken at runtime or will be if net/py-urllib3 1.25.6 lands:

> Portname                      Required version        Remarks
> security/theonionbox          >=1.24.2,<1.25          setup.py / Broken at runtime / Fix with bug #241827
> textproc/py-elasticsearch5    >=1.21.1<1.23           Patched out setup.py / Runtime fix with bug #241875
> textproc/py-elasticsearch6    >=1.21.1                setup.py / Runtime fix with bug #241874 (already committed with ports r517541, MFH pending)

I did also some own exp-runs against direct and indirect consumers that were all successful so far. Once the three fixes listed above are committed the update for net/py-urllib3 might be ready to land.
Comment 28 Kai Knoblich freebsd_committer 2019-11-20 10:46:24 UTC
Asking portmgr@ if the update for net/py-urllib3 requires an exp-run.

As already mentioned in comment #27 there are three ports that still require a fix. They build fine but are already broken or will be once net/py-urllib3 is updated to 1.25.6.
Comment 29 commit-hook freebsd_committer 2019-11-25 17:19:20 UTC
A commit references this bug:

Author: kai
Date: Mon Nov 25 17:18:36 UTC 2019
New revision: 518410
URL: https://svnweb.freebsd.org/changeset/ports/518410

Log:
  textproc/py-elasticsearch5: Prepare for urllib3 >= 1.25

  * Backport a patch from the 7.x branch of upstream repository that fixes a
    possible runtime issue with urllib3 1.25 [1] since that release verifies
    SSL certificates by default.

    Disabling SSL certificate verification via "verify_certs" in elasticsearch
    won't work then as expected thus set "cert_reqs=CERT_NONE" explicitly to
    restore that behavior.

  PR:		241875, 229322 [1]
  Approved by:	maintainer timeout (elastic, 14 days)
  MFH:		2019Q4

Changes:
  head/textproc/py-elasticsearch5/Makefile
  head/textproc/py-elasticsearch5/files/patch-elasticsearch_connection_http__urllib3.py
Comment 30 Kai Knoblich freebsd_committer 2019-11-25 17:57:21 UTC
With ports r518410 all preparations from my side are now done for the /head branch to get net/py-urllib3 updated to 1.25.6.

When my assumptions are correct there are two exp-runs running which are related to urllib3 at the moment (the label PR241621 is a somewhat confusing in that case):

http://package23.nyi.freebsd.org/build.html?mastername=113i386-default-PR241624&build=2019-11-25_06h54m48s

http://package22.nyi.freebsd.org/build.html?mastername=113amd64-default-PR241624&build=2019-11-25_10h11m45s

I have one question/note regarding the "merge-quartely" flag that was set to "-" recently: 

I'm afraid that a MFH is required because the 1.25.6 release of urllib3 includes fixes for three CVEs (CVE-2018-20060, CVE-2019-11236 and CVE-2019-11324). I plan to commit a related VuXML entry in a few hours.

At the moment I'm doing preparations and test-runs for the 2019Q4 branch but that still takes a little while. Maybe another exp-run for the 2019Q4 branch makes sense once urllib 1.25.6 lands in /head?
Comment 31 Antoine Brodin freebsd_committer 2019-11-25 18:21:26 UTC
(In reply to Kai Knoblich from comment #30)
MFH is not approved,  there are too much changes to make it work in the quaterly branch.
Comment 32 Antoine Brodin freebsd_committer 2019-11-26 06:49:27 UTC
Exp-run looks fine (only build time was tested, not run time)
Comment 33 commit-hook freebsd_committer 2019-11-26 11:51:56 UTC
A commit references this bug:

Author: kai
Date: Tue Nov 26 11:51:31 UTC 2019
New revision: 518463
URL: https://svnweb.freebsd.org/changeset/ports/518463

Log:
  security/vuxml: Document net/py-urllib3 issues

  PR:		229322
  Security:	CVE-2018-20060
  		CVE-2019-11236
  		CVE-2019-11324

Changes:
  head/security/vuxml/vuln.xml
Comment 34 commit-hook freebsd_committer 2019-11-26 18:38:39 UTC
A commit references this bug:

Author: kai
Date: Tue Nov 26 18:37:59 UTC 2019
New revision: 518476
URL: https://svnweb.freebsd.org/changeset/ports/518476

Log:
  net/py-urllib3: Update to 1.25.6

  * Convert the RUN_DEPENDS into separate OPTIONS as they are listed as extra
    dependencies in setup.py.  Also set those as default that contain the
    previous RUN_DEPENDS to allow a clean transition.

  * Remove the Python-specific version limitation for security/py-certifi
    because it's required for all Python versions.

  * Also remove the info about the broken IPv6 support of net/py-socks (was
    broken in 1.5.7) and the relevant patch as both are obsolete.

  * Update the TEST_DEPENDS and add a "do-test" target to make future QA easier.

  Please note that a MFH won't be done as it didn't get an approval because
  there are too much changes to make it work in the 2019Q4 branch. [1]

  Notable changes since 1.22:

  * Require and validate certificates by default when using HTTPS.

  * Add mitigation for BPO-37428 affecting Python < 3.7.4 and OpenSSL 1.1.1+
    which caused certificate verification to be enabled when using
    "cert_reqs=CERT_NONE".

  * Add TLSv1.3 support to CPython, pyOpenSSL and SecureTransport "SSLContext"
    implementations.

  https://github.com/urllib3/urllib3/blob/1.25.6/CHANGES.rst

  Exp-run by:	antoine
  PR:		229322 [1]
  Reported by:	Patrice Clement <monsieurp@gentoo.org>
  Security:	87270ba5-03d3-11ea-b81f-3085a9a95629

Changes:
  head/UPDATING
  head/net/py-urllib3/Makefile
  head/net/py-urllib3/distinfo
  head/net/py-urllib3/files/patch-setup.py
  head/net/py-urllib3/files/pkg-message.in
Comment 35 Kai Knoblich freebsd_committer 2019-11-26 18:40:57 UTC
Technically this PR can be closed now as net/py-urllib3 is updated 1.25.6 in /head and no MFH will be done.

I'll leave this PR a open for 1-2 weeks to in case there are some errors/regressions.

(In reply to Antoine Brodin from comment #32)

Thank you, Antoine, for the exp-run!