Bug 205308 - devel/py-pip and devel/py-virtualenv don't aggree on version suffix
Summary: devel/py-pip and devel/py-virtualenv don't aggree on version suffix
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-python mailing list
URL: https://github.com/servo/servo/issues...
Keywords: needs-qa, regression
Depends on:
Blocks:
 
Reported: 2015-12-14 08:22 UTC by Jan Beich
Modified: 2018-01-17 09:34 UTC (History)
3 users (show)

See Also:
koobs: maintainer-feedback? (nivit)
koobs: merge-quarterly?


Attachments
Use pip2.7 over pip-2.7 (a la PkgSrc) (842 bytes, patch)
2016-07-14 08:56 UTC, Jan Beich
no flags Details | Diff
Use pip2.7 over pip-2.7 (a la PkgSrc), v2 (628 bytes, patch)
2016-07-14 13:06 UTC, Jan Beich
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jan Beich freebsd_committer 2015-12-14 08:22:11 UTC
Due to inconsistent suffix upstream cannot rely on it: pip-2.7 vs. pip2.7. Regressed by ports r390183 which also removed pip2. See URL for an impact. Here's a simplified version:

$ virtualenv-2.7 python/_virtualenv
New python executable in python/_virtualenv/bin/python2.7
Also creating executable in python/_virtualenv/bin/python
Installing setuptools, pip, wheel...done.

$ . python/_virtualenv/bin/activate

(_virtualenv)$ pip-2.7 install blessings
Collecting blessings
  Using cached blessings-1.6.tar.gz
Installing collected packages: blessings
  Running setup.py install for blessings
    Complete output from command /usr/local/bin/python2.7 -c "import setuptools, tokenize;__file__='/tmp/pip-build-JLlt1P/blessings/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-JhU9dM-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/blessings
    copying blessings/tests.py -> build/lib/blessings
    copying blessings/__init__.py -> build/lib/blessings
    running egg_info
    writing blessings.egg-info/PKG-INFO
    writing top-level names to blessings.egg-info/top_level.txt
    writing dependency_links to blessings.egg-info/dependency_links.txt
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'blessings.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'blessings.egg-info/SOURCES.txt'
    running install_lib
    creating /usr/local/lib/python2.7/site-packages/blessings
    error: could not create '/usr/local/lib/python2.7/site-packages/blessings': Permission denied

    ----------------------------------------
Command "/usr/local/bin/python2.7 -c "import setuptools, tokenize;__file__='/tmp/pip-build-JLlt1P/blessings/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-JhU9dM-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-JLlt1P/blessings
zsh: exit 1     pip-2.7 install blessings
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2016-07-14 07:59:33 UTC
@Jan, is this still an issue?
Comment 2 Jan Beich freebsd_committer 2016-07-14 08:56:11 UTC
Created attachment 172499 [details]
Use pip2.7 over pip-2.7 (a la PkgSrc)

It is. Did you try the steps to reproduce before asking? virtualenv still fails to shadow system pip.

Here's a fix based on review D5844. pip-2.7 is relatively rare, only MacPorts and Homebrew use it besides FreeBSD. Most distributions also provide pip2 symlink but it's not currently possible to implement with USES=uniquefiles.
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2016-07-14 10:52:44 UTC
(In reply to Jan Beich from comment #2)

I didn't run through the steps, I was asking with hat @triage not @python, apologies for not being less ambiguous.

A few notes:

The prefixes/suffixes set in python.mk are intended to standardize script filenames for all python ports, which is the reason it's not override'able.

The pip* scripts provided by devel/py-pip, when invoked (either by the user, or by blessings in this case), attempt to install the packages in the system site-packages location, which requires root. This is the reason for the permission denied error message. Using pip to manage system site-packages is not supported, as per the devel/py-pip pkg-message.

virtualenv (the project) vendors (bundes) setuptools, pip & wheel packages, and installs those versions when creating new environments. The devel/py-virtualenv port does not unbundle them.

Once a virtualenv has been created, using the pip that's installed in the virtualenv, correctly installs the desired packages:

(_virtualenv) [user@CURRENT-amd64:/usr/home/user/test/python/_virtualenv/bin] pip install blessings

pip install blessings
Collecting blessings
  Using cached blessings-1.6.tar.gz
Building wheels for collected packages: blessings
  Running setup.py bdist_wheel for blessings ... done
  Stored in directory: /home/koobs/.cache/pip/wheels/a5/7c/ba/e52b14bbd1b8cdeeb0771b03e1cca9b9d438c6e0054144a508
Successfully built blessings
Installing collected packages: blessings
Successfully installed blessings-1.6
You are using pip version 8.0.0, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

This also works outside of _virtualenv/bin as virtualenv modifies the PATH to pick up the scripts from the virtualenv that has been activated, before system scripts.

(_virtualenv) [user@CURRENT-amd64:/usr/home/user/test] pip install requests
Collecting requests
  Downloading requests-2.10.0-py2.py3-none-any.whl (506kB)
    100% |████████████████████████████████| 507kB 224kB/s
Installing collected packages: requests
Successfully installed requests-2.10.0
You are using pip version 8.0.0, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

While one can acknowledge devel/pip does not use the default script names, it's not yet clear how blessings finding pip-x.y before others implies that devel/py-pip should change them, given it appears blessings would find  "pip2.7", "pip2", "pip" from the system after the change anyway, resulting in the same permission denied error (as expected), or why virtualenv must follow the same naming as pip, given it bundles and uses its own copies of pip.

Lastly, its possible, though in this case I'm not necessarily advocating for, a port to use USES=uniquefiles directly, rather than python.mk's 'concurrent' feature. This is the reason uniquefiles was created separately, to allow for any ports (instead of just python ones) to use it.
Comment 4 Jan Beich freebsd_committer 2016-07-14 12:44:29 UTC
During build Servo installs a few python 2.x modules via virtualenv. Not knowing which version suffix is used by the distribution it tries to figure it out using PATH. However, the build breaks as virtualenv activation leaks system pip through PATH. This worked fine before ports r390183 and I don't think Servo logic is wrong.

  # Possible names of executables, sorted from most to least specific
  PYTHON_NAMES = ["python-2.7", "python2.7", "python2", "python"]
  VIRTUALENV_NAMES = ["virtualenv-2.7", "virtualenv2.7", "virtualenv2", "virtualenv"]
  PIP_NAMES = ["pip-2.7", "pip2.7", "pip2", "pip"]

https://github.com/servo/servo/blob/master/python/mach_bootstrap.py

As pip bundled in virtualenv doesn't support pip-X.Y variant the following patch has no effect. The issue has to be fixed in devel/py-pip.

  --- pip/wheel.py	2016-01-21 23:48:28 UTC
  +++ pip/wheel.py
  @@ -465,6 +465,9 @@ if __name__ == '__main__':

	   spec = 'pip%s = %s' % (sys.version[:3], pip_script)
	   generated.extend(maker.make(spec))
  +        # Also shadow pip-X.Y variant used by FreeBSD package
  +        spec = 'pip-%s = %s' % (sys.version[:3], pip_script)
  +        generated.extend(maker.make(spec))
	   # Delete any other versioned pip entry points
	   pip_ep = [k for k in console if re.match(r'pip(\d(\.\d)?)?$', k)]
Comment 5 Jan Beich freebsd_committer 2016-07-14 13:06:51 UTC
Created attachment 172508 [details]
Use pip2.7 over pip-2.7 (a la PkgSrc), v2

- Converted to use USES=uniquefiles directly.

This remains a workaround because the proper fix requires coordination with pip upstream then waiting for virtualenv upstream then for consumers to update bundled virtualenv. Why would anyone care about pip-X.Y if its purely downstream invention?
Comment 6 Walter Schwarzenfeld freebsd_triage 2018-01-17 09:34:14 UTC
Is this still relevant?