Bug 229322 - net/py-urllib3: Update to 1.24.2
Summary: net/py-urllib3: Update to 1.24.2
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Kubilay Kocak
URL:
Keywords: needs-patch, needs-qa, security
: 229951 235261 (view as bug list)
Depends on: 236283
Blocks: 234994
  Show dependency treegraph
 
Reported: 2018-06-24 23:46 UTC by Patrice Clement
Modified: 2019-07-09 18:51 UTC (History)
5 users (show)

See Also:
bugzilla: maintainer-feedback? (python)
koobs: merge-quarterly?


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Patrice Clement 2018-06-24 23:46:38 UTC

    
Comment 1 Patrice Clement 2018-06-24 23:47:18 UTC
Hi

Here's the diff to update py-urllib3 to 1.23.

Cheers,
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2018-06-25 04:14:32 UTC
Doing this as part of py-requests update.
Comment 3 Antoine Brodin freebsd_committer 2018-06-25 05:12:54 UTC
I expect many failures.
Comment 4 Antoine Brodin freebsd_committer 2018-06-25 08:17:37 UTC
The following packages seem to depend on an earlier version of urllib3:
- py*-requests
- py*-pipenv
- py*-pip
- py*-elasticsearch5
- py*-elasticsearch
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2018-07-23 05:08:50 UTC
*** Bug 229951 has been marked as a duplicate of this bug. ***
Comment 6 Patrice Clement 2018-07-25 17:12:48 UTC
Is there something I can do to help out here?
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2018-08-01 05:25:26 UTC
(In reply to Patrice Clement from comment #6)

The main area for QA blocking this update is identifying which reverse dependents of urllib in the ports tree won't work with >= 1.23. Subsequent to that, if the list is non zero, identifying upstream commits, released in newer versions or unreleased that add support for >= 1.23

The task is made more difficult because building/packaging successfully (either manually, or during an exp-run) is not sufficient to identify compatibility issues, as the vast majority of Python ports either do not (and/or cant) specify version restrictions in their *_DEPENDS lines that would trigger builds to fail, and/or do not have test targets that could (potentially) be run to produce pkg_resources.VersionConflict errors by setuptools, effectively testing run-time compatibility.
Comment 8 commit-hook freebsd_committer 2019-01-22 10:47:02 UTC
A commit references this bug:

Author: koobs
Date: Tue Jan 22 10:46:12 UTC 2019
New revision: 490937
URL: https://svnweb.freebsd.org/changeset/ports/490937

Log:
  www/py-requests: Update to 2.21.0

   - Update USES comment (Python 3.3 support dropped)
   - Rebase setup.py patch (idna change released)
   - Remove comment about failing tests due to httpbin issue which seems
     to now be fixed.

  This update includes a pinned urllib3 version bump to < 1.25, which paves
  the way for a net/urllib3 update to 1.24 [1].

  Note: 2.20.0 includes a security vulnerability fix for CVE-2018-18074

  Changelog:

    https://github.com/requests/requests/blob/v2.21.0/HISTORY.md

  PR: 		229322 [1]
  Security:	50ad9a9a-1e28-11e9-98d7-0050562a4d7b
  MFH:		2019Q1

Changes:
  head/www/py-requests/Makefile
  head/www/py-requests/distinfo
  head/www/py-requests/files/patch-setup.py
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2019-01-22 11:14:18 UTC
urllib3 < 1.23 has a similar (same?) vulnerability as requests < 2.20.0, who's update to 2.21.0 just landed in ports r490937 ...

 - https://github.com/urllib3/urllib3/issues/1316
 - https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-20060

On a somewhat more positive note, after looking through all ports that depend on net/py-urllib3 (their upstream source code), the only ones that pin a max version of urllib3 are: 

./www/py-requests: setup.py: 'urllib3>=1.21.1,<1.23'
./textproc/py-elasticsearch5: setup.py: 'urllib3<1.23,>=1.21.1',
./devel/py-botocore: setup.py: requires.append('urllib3>=1.20,<1.25')

Of those, py-requests has bumped that to <1.24 as of 2.21.0 (already committed), and py-botocore version is above (1.25) what we'll be updating urllib3 to (1.24).

That leaves textproc/py-elasticsearch5 (maintainer CC'd) ...

I have a WIP patch to add QA TEST_DEPENDS/test target to py-elasticsearch5, which required switching the sources to GitHub. After patching out the the max version pin, the tests pass [1] after updating urllib3 to 1.24.

Finally, with the last py-requests update and a WIP urllib3 1.24 update in place, cmake also does not regress (bug 228770) as expected.

[1] ~103 tests pass. Tests that require an local/live elasticsearch server, which I don't have running, aren't run, but don't explicitly fail.
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2019-01-28 03:11:45 UTC
*** Bug 235261 has been marked as a duplicate of this bug. ***
Comment 12 Kubilay Kocak freebsd_committer freebsd_triage 2019-01-28 03:12:34 UTC
Will request an exp-run when ready.
Comment 13 Sunpoet Po-Chuan Hsieh freebsd_committer 2019-03-05 17:13:00 UTC
textproc/py-elasticsearch-curator is the only customer of the blocker (textproc/py-elasticsearch5). I've submitted bug #236283 to update it to 5.6.0 which no longer depends on textproc/py-elasticsearch5. After that, we could remove textproc/py-elasticsearch5 and request exp-run for py-urllib3 update.
Comment 14 commit-hook freebsd_committer 2019-03-25 07:49:15 UTC
A commit references this bug:

Author: koobs
Date: Mon Mar 25 07:48:27 UTC 2019
New revision: 496799
URL: https://svnweb.freebsd.org/changeset/ports/496799

Log:
  MFH: r490937 www/py-requests: Update to 2.21.0

   - Update USES comment (Python 3.3 support dropped)
   - Rebase setup.py patch (idna change released)
   - Remove comment about failing tests due to httpbin issue which seems
     to now be fixed.

  This update includes a pinned urllib3 version bump to < 1.25, which paves
  the way for a net/urllib3 update to 1.24 [1].

  Note: 2.20.0 includes a security vulnerability fix for CVE-2018-18074

  Changelog:

    https://github.com/requests/requests/blob/v2.21.0/HISTORY.md

  PR: 		229322 [1]
  Security:	50ad9a9a-1e28-11e9-98d7-0050562a4d7b

  Approved by:	ports-secteam (miwi)

Changes:
_U  branches/2019Q1/
  branches/2019Q1/www/py-requests/Makefile
  branches/2019Q1/www/py-requests/distinfo
  branches/2019Q1/www/py-requests/files/patch-setup.py
Comment 15 Sunpoet Po-Chuan Hsieh freebsd_committer 2019-03-30 08:36:56 UTC
I think it's time for another exp-run.
Comment 16 commit-hook freebsd_committer 2019-04-16 04:07:01 UTC
A commit references this bug:

Author: koobs
Date: Tue Apr 16 04:06:27 UTC 2019
New revision: 499073
URL: https://svnweb.freebsd.org/changeset/ports/499073

Log:
  textproc/py-elasticsearch5: Remove pinned urllib3 version

  elasticsearch5 (this port) unnecessarily pins its urllib dependency to
  < 1.23, which blocks updating urllib3 to 1.24 [1]:

  ./textproc/py-elasticsearch5: setup.py: 'urllib3<1.23,>=1.21.1',

  The package had a history of issues/conflicts/bugs with the urllib3
  dependency, ultimately resulting in the maximum version pin being
  removed [2]:

    https://github.com/elastic/elasticsearch-py/issues/807
    https://github.com/elastic/elasticsearch-py/issues/667
    https://github.com/elastic/elasticsearch-py/issues/634

  This commit backports that change, a functional noop and sweeping change
  in advance required for a urllib3 update, and adds TEST_DEPENDS and
  a test target to support rigorous and confident QA. Switching to GitHub
  sources was required as the PyPI sdist does not package tests.

  The packages tests all pass with/against urllib3 1.24 installed, with an
  intermittent and non-deterministic off-by-one failure in one test:

  FAIL: test_all_chunks_sent (test_elasticsearch.test_helpers.TestParallelBulk)

  The issue exists independent of urllib3 version. The flaky test issue was
  reported upstream [3], but was not resolved.

  [2] https://github.com/elastic/elasticsearch-py/commit/4352e56174b77560d2f86801cb1ad32440bb2d32
  [3] https://github.com/elastic/elasticsearch-py/issues/701

  PR:		229322 [1]
  Approved by:	portmgr (blanket: framework compliance, runtime bugfix)

Changes:
  head/textproc/py-elasticsearch5/Makefile
  head/textproc/py-elasticsearch5/distinfo
  head/textproc/py-elasticsearch5/files/
  head/textproc/py-elasticsearch5/files/patch-setup.py
Comment 17 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-16 04:09:55 UTC
After ports r499073 (required to unblock update), a VuXML entry and further and final QA for the urllib3 WIP is pending, after which point I'll request an ex-run
Comment 18 Sunpoet Po-Chuan Hsieh freebsd_committer 2019-04-18 17:44:03 UTC
1.24.2 is out. Please use this one instead. Thanks!
Comment 19 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-19 09:48:58 UTC
(In reply to Sunpoet Po-Chuan Hsieh from comment #18)

Will do, thanks!
Comment 20 Sunpoet Po-Chuan Hsieh freebsd_committer 2019-07-09 18:51:19 UTC
(In reply to Kubilay Kocak from comment #19)

Hi, any progress on 1.24.2 or even latest 1.25.3? I need urllib3 1.24+ to unblock py-softlayer update. :)