I'm opening this PR at wen@'s request as a way to resolve bug 216953. This upgrade appears to be non-trivial because setuptools un-vendored its dependencies as of v34.0.0 [0]. In particular, completing this upgrade may require breaking the setuptools dependency of devel/py-pyparsing devel/py-six devel/py-appdirs and/or devel/py-pip on setuptools in order for setuptools to depend on them. Also, devel/py-packaging doesn't exist yet, and it might have to. [0] https://github.com/pypa/setuptools/blob/master/CHANGES.rst#v3400
Here's a candidate strategy. 1. Teach USES=python about USE_PYTHON=wheel [0] 2. Convert devel/py-pyparsing, devel/py-six, and devel/py-appdirs to wheel 3. Implement devel/py-packaging to wheel 4. Convert devel/py-setuptools to wheel and depend on the foregoing The first step would be the only relatively heavy lift. Fortunately, the wheel feature would initially only need limited PEP-427 compliance since all ports that need it for this purpose are pure python, support both py2 and py3, set "Root-Is-Purelib: true", and don't require most steps of the "spread" phase because they all lack a .data directory. It should be reasonably sane to retain autoplist support for the wheel feature by using the output of `unzip -l`. Jason Coombs writes, "Instead of vendoring the growing list of dependencies that Setuptools requires to function, Setuptools now requires these dependencies just like any other project." This "growing list" makes me wonder how quickly Setuptools has been accumulating dependencies. The answer is not that fast, at least so far. 2014-09-04 packaging [1] 2015-12-10 pyparsing [2] 2016-01-07 six [3] 2016-08-27 appdirs [4] This gives me a basis to expect that a nascent implementation of #1 above given the minimal requirements described isn't likely to be broken right away. If there are any major problems or pitfalls with this approach I haven't thought of or encountered them yet. I shall, therefore, set off down this road in search of fame and fortune. [0] https://www.python.org/dev/peps/pep-0427/ [1] https://github.com/pypa/setuptools/commit/84c9006110e53c84296a05741edb7b9edd305f12 [2] https://github.com/pypa/setuptools/commit/a718819d2849196e902808301c9a95724510c5c1 [3] https://github.com/pypa/setuptools/commit/3bd5118eaa87b4f6598e0a473fec3623b6579d3b [4] https://github.com/pypa/setuptools/commit/691e6ac03339d6aef045e05872b286d91e9f49b9
What about removing all the setuptools / pip ports and use the ones bundled with the python ports instead (with --with-ensurepip=install)?
(In reply to Antoine Brodin from comment #2) I'm glad you asked. :) If we were to do as you suggest, and essentially expand lang/pythonXY to install the bundled pip and setuptools during the stage phase, then we would lock the pip and setuptools upgrade cycle to the corresponding lang/pythonXY upgrades. All current python versions bundle pip 9.0.1 (latest) and setuptools 28.8.0 (from 2016-11-04, 27 releases earlier than latest). Furthermore, since pip and setuptools are bundled as wheels, they could presumably be overwritten with updated wheels during the lang/pythonXY patch phase. I can think of a few drawbacks to that, but no deal breakers. That seems like a fine alternative, and I would be glad to work on developing a patch to implement it if the list prefers.
(In reply to Antoine Brodin from comment #2) Upon further reflection... A solution that uses ensurepip *without* a way to upgrade pip and setuptools (P&S) more frequently than python itself would be a step backwards, because we could not straight forwardly patch either one of them. The ports machinery wants patching to happen between extraction and configuration, while --with-ensurepip=install wouldn't make the guts of P&S available for patching until the midst of staging. Furthermore, the unpatched, bundled version of S would still suffer from the lack of globby grafts that led to bug 216953, because globbing reportedly broke after v28.4.0 [0] which affects the bundled v28.8.0. Therefore, we *must* retain the ability to upgrade P&S between python releases *or* to patch it locally (or both). So, what options are available for doing that? Let's start with my suggestion from comment #3: drop in new wheels during production of a lang/pythonXY package that are fetched as distfiles or patches and inserted into WRKSRC somewhere between post-fetch and pre-stage. There are two problems with this. First, it still doesn't solve the local patching limitation because P&S would still appear as wheels until after the stage phase. Second, in order to upgrade setuptools to v34.0.1 or later to thoroughly satisfy to goals of bug 216953, or for some other reason that may present itself before upstream python decides how to bundle v34.0.1 or later for ensurepip and releases a new version, we would have to make that decision locally and implement it. Another disadvantage of tying P&S to lang/pythonXY is that any any patch or upgrade to P or S will induce a PORTREVISION bump to lang/pythonXY and associated rebuild of all ports that depend on it, which is a good deal more than the ports that depend on P or S. This pushes me back toward wanting to retain P&S as independent ports, and to think about other strategies for dealing with tricky dependencies. Taking a page from the ensurepip book, a third family of options could be based on implementing a wheel-oriented install for a small number of ports (probably all of those mentioned in comment #1) but treating devel/py-pip as the root of the dependency tree and depending upon it as a BUILD_DEPENDS for as much of the wheel installation implementation as possible (instead of DIY in Mk/Uses/python.mk). The key integration points would be: 1) Retain the ability to patch one of those ports locally, possibly by moving the patching to a post-stage target, or moving the pip-managed wheel installation to a pre-patch target with a WRKSRC destination and implementing the stage target as a dumb bunch of cp commands. 2) To the extent that wheels are involved, fetch and verify them with the ports machinery and then feed to pip to preserve the integrity and authenticity of the ports. 3) Generate PYC/PYO files. 4) Draft a packing list (a la autoplist) from pip's work and merge the bytecode files into it. I think that's all I've got for now, and welcome reactions. [0] https://github.com/pypa/setuptools/issues/935#issue-202591204
To update setuptools to 34.1.1 , I suggest: 1 change the install way of devel/py-pyparsing, devel/py-six and devel/py-appdirs, install these ports with setup from distutils.core, instead of setuptools. Then we can avoid cycle-depend. 2 create the new port devel/py-packaging with the same install way. 3 update setuptools to 34.1.1 I upload the patch file for it. Any suggestions ? wen
Created attachment 179982 [details] patch to update devel/py-setuptools to 34.1.1. patch to update devel/py-setuptools to 34.1.1.
A problem is that we currently have 6 ports, so that poudriere builds a setuptools package for all versions of python in the ports tree: devel/py-setuptools devel/py-setuptools27 devel/py-setuptools33 devel/py-setuptools34 devel/py-setuptools35 devel/py-setuptools36 With your patch, only 1 version of py-parsing, py-packaging etc. will be build and the dependencies of py-setuptools3* won't be satisfied in poudriere builds. Other problems: - install should be run with "-c -O1" to create pyo files - the plist won't work with python 3, py3kplist may help
(In reply to Wen Heping from comment #5) Nice work. pyparsing [0] and six [1] need patch-setup.py setuptools needs RUN_DEPENDS+=BUILD_DEPENDS pkg-plist needs %%PORTVERSION%%, %%PYTHON_VER%% consistently, or just %%PYDISTUTILS_EGGINFO%% (right?) packaging needs {BUILD,RUN}_DEPENDS+=${blah}:devel/py-pyparsing [0] https://sourceforge.net/p/pyparsing/code/HEAD/tree/tags/pyparsing_2.1.10/src/setup.py#l4 [1] https://github.com/benjaminp/six/blob/master/setup.py#L23
(In reply to Antoine Brodin from comment #2) Now I support use the ones bundled with the python ports instead (with --with-ensurepip=install).
Created attachment 180610 [details] USES=wheel, remove py33 slaves, add slave ports for setuptools dependencies, update setuptools to 34.3.1 Attached is a PoC: - USES=wheel to install a pure python wheel using pip bundled in python ; the urls to fetch pure python wheels are quite cryptic, I don't know if there's a better way - unfortunately python33 doesn't have ensurepip, so I removed py33 slaves - add slaves for setuptools dependencies (appdirs, packaging, pyparsing and six) and use USES=wheel to install them - update setuptools to 34.3.1 ; I used USES=wheel but USE_PYTHON=setuptools can probably be used
Is it possible to change py-setuptoolsXX to pyXX-setuptools along with this update?
(In reply to Po-Chuan Hsieh from comment #11) Probably, does anyone remember why those ports were named this way?
(In reply to Antoine Brodin from comment #12) I proposed a patch (bug #187091) to allow coexistence of python ports with different python versions. I wish I came up with pyXX- prefix at that time.
Created attachment 181404 [details] USES=wheel, remove py33 slaves, add slave ports for setuptools dependencies, update setuptools to 34.3.3
Hi, I've tested the latest patch from antoine, so far ever works and builds, tested on 2 machines. I think its ready to go.
For what it's worth, and albeit belatedly, I'm strongly -1 on using ensurepip to boostrap instead of the setuptools/pip ports for the same and additional reasons as John someone captures in comment 3 and comment 3. Namely: - static versions - per python port (currently 4, but at best 3 (the number of concurrently supported upstream bugfix branches at any one time) - that receive no bugfixes (unless we backport them) and - unless they are manually hacked/updated in the pythonXY ports by ourselves Instead I would have the pythonXY ports depend on the setuptools/pip ports so that they are guaranteed to be provided in a new installation (as PEP453 [1] recommends): "Ensure that whenever Python is installed pip is either installed or is otherwise made readily available to end users. " It appears that the only requirement here is that setuptools/pip is guaranteed to be available for the framework (USES=wheel or otherwise) to use. There is no apparent or explicitly mentioned requirement for that to happen via the private/vendored setuptools/pip (ensurepip) module. Further, having the framework utilise one version of setuptools/pip and other parts of a users environment (site-packages and/or virtual environments) potentially or actually use another is a major issue. Addressing that issue by manually upgrading private/vendored copies of pip/setuptools in lang/pythonXY ports with patches or framework in-place upgrades is also likely to cause issues. It also means multiple (one per python port) combinations of setuptools/pip to maintain, bug check, test and upgrade than one. Finally, setuptools/pip (the upstream separate PyPI packages) are intended to support all currently supported Python branches at once within the same version, and breakages are considered bugs/regressions. We should leverage that level of support by using them. [1] https://www.python.org/dev/peps/pep-0453/
Additionally, related to the removal of python33, id like that to be created as a blocking issue, so it can be dealt with and land separately (before this issue) so as to allow for announcing its removal (along with 3.4) as part of a new python policy (yet to propose) to only support those Python versions that upstream supports with bugfix/security backports/merges. Currently and for a long time this means 2.7 + 3.latest + 3.latest-1. I'm happy to create the issue for this, write up the (short) proposal, announce to list/users and take care of the removal of these ports in accordance with it.
Hi, Actually I was about to commit that wheel framework to ports the other day, but decided to keep it on hold, for 2 reasons, 1) flavors will enter the ports tree very soon, which will solve many problems in particular with multiple concurrent py versions which means bigger changes to the python framework is needed. 2) I agree with John that bundled pip/setuptools might be a step backwards in terms of security because if pip or setuptools got a security problem we will have to wait till pythonxX catches up on that. Which is most likely a bad idea. That all said I think it's better to wait till flavors enters the portstree and address this issue together with the needed changes to the python framework. - Martin
unbundling of pyparsing / six / appdirs was reverted in version 36.0.0: #980 and others: Once again, Setuptools vendors all of its dependencies. It seems to be the case that in the Python ecosystem, all build tools must run without any dependencies (build, runtime, or otherwise). At such a point that a mechanism exists that allows build tools to have dependencies, Setuptools will adopt it. The port has been updated to version 36.0.1