Bug 191865 - math/py-numpy: Builds and stages egg-info directory but silently fails to install it
math/py-numpy: Builds and stages egg-info directory but silently fails to ins...
Status: Closed FIXED
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s)
amd64 Any
: Normal Affects Some People
Assigned To: Kubilay Kocak
: easy
Depends on:
Blocks: 191857
  Show dependency treegraph
Reported: 2014-07-14 15:52 UTC by John W. O'Brien
Modified: 2014-07-24 23:52 UTC (History)
2 users (show)

See Also:

Recursive listing of python3.3 site-packages directory (95.64 KB, text/plain)
2014-07-14 17:47 UTC, John W. O'Brien
no flags Details
Test environment with DEFAULT_VERSIONS=python=3.4 (513 bytes, text/plain)
2014-07-14 18:10 UTC, John W. O'Brien
no flags Details
Demonstation of problem under python3.4 (772 bytes, text/plain)
2014-07-14 18:11 UTC, John W. O'Brien
no flags Details
Recursive listing of python3.4 site-packages directory (95.56 KB, text/plain)
2014-07-14 18:12 UTC, John W. O'Brien
no flags Details
Patch math/py-numpy to install egg-info directory (408 bytes, patch)
2014-07-16 00:51 UTC, John W. O'Brien
no flags Details | Diff
Patch math/py-numpy to install egg-info directory and bump PORTREVISION (579 bytes, patch)
2014-07-16 01:05 UTC, John W. O'Brien
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description John W. O'Brien 2014-07-14 15:52:15 UTC
Test environment:

    root@teratoid:~ # uname -a
    FreeBSD teratoid.n.saltant.net 10.0-STABLE FreeBSD 10.0-STABLE #0 r266389: Sat May 17 23:21:39 EDT 2014     root@drivel.saltant.net:/usr/obj/usr/src/sys/NARB  amd64
    root@teratoid:~ # pkg info -aq

How to repeat:

    root@teratoid:~ # python3.3
    Python 3.3.5 (default, Jul 14 2014, 11:17:18)
    [GCC 4.2.1 Compatible FreeBSD Clang 3.4 (tags/RELEASE_34/final 197956)] on freebsd10
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pkg_resources
    >>> import sys
    >>> for d in sys.path:
    ...     print("Searching {}".format(d))
    ...     for dist in pkg_resources.find_distributions(d):
    ...         print(dist)
    Searching /usr/local/lib/python33.zip
    Searching /usr/local/lib/python3.3
    Searching /usr/local/lib/python3.3/plat-freebsd10
    Searching /usr/local/lib/python3.3/lib-dynload
    Searching /root/.local/lib/python3.3/site-packages
    Searching /usr/local/lib/python3.3/site-packages
    setuptools 5.1
    nose 1.3.0
    >>> import numpy
    >>> numpy.__file__

This also manifests itself when setup.py runs with setup(setup_requires="numpy >= 1.7.0b2"), for example, because the underlying pkg_resources.require("numpy >= 1.7.0b2") raises DistributionNotFound.
Comment 1 Kubilay Kocak freebsd_committer 2014-07-14 16:12:10 UTC
An update to devel/py-setuptools to 5.4.1 is about to land shortly (pending a small QA issue).

Can you take a look at the upstream changes [1] and let me know if there's anything in there that might be relevent.


If possible, it would also be handy to reproduce the issue on lang/python34, to isolate a root cause to a specific port
Comment 2 Kubilay Kocak freebsd_committer 2014-07-14 16:16:53 UTC
Also, a directory listing of the contents of /usr/local/lib/python3.3/site-packages could prove handy (please attach it, rather than in a comment)
Comment 3 John W. O'Brien 2014-07-14 17:47:15 UTC
Created attachment 144664 [details]
Recursive listing of python3.3 site-packages directory
Comment 4 John W. O'Brien 2014-07-14 18:00:58 UTC
koobs: I see nothing relevant in the upstream changelog for setuptools. I'm working on reproducing in 3.4 right now.
Comment 5 John W. O'Brien 2014-07-14 18:10:19 UTC
Created attachment 144667 [details]
Test environment with DEFAULT_VERSIONS=python=3.4

To upgrade the test environment, I pkg remove'd the three py33-* packages plus python itself, updated /etc/make.conf, then built and installed lang/python3.4, devel/py-setuptools34, devel/py-nose, and math/py-numpy.
Comment 6 John W. O'Brien 2014-07-14 18:11:42 UTC
Created attachment 144668 [details]
Demonstation of problem under python3.4

This terminal capture shows the same behavior under python3.4.
Comment 7 John W. O'Brien 2014-07-14 18:12:36 UTC
Created attachment 144669 [details]
Recursive listing of python3.4 site-packages directory
Comment 8 John W. O'Brien 2014-07-14 18:18:44 UTC
See also bug 191857, comment 1, regarding a previous notice to freebsd-pkg-fallout@, which suggests that this bug affects 2.7 too, at least on 11.0-CURRENT.
Comment 9 John W. O'Brien 2014-07-14 18:44:04 UTC
Oh, that's interesting and unexpected... I just reproduced this with 2.7 in one test environment, but could not in another. I wonder what's different?
Comment 10 John W. O'Brien 2014-07-14 19:45:51 UTC
What was different is that on one machine, /usr/local/lib/python2.7/site-packages/numpy-1.8.1-py2.7.egg-info existed, and on the other it didn't. This is a problem because the pkg_resources.find_distributions function ends up using the pkg_resources.find_on_path finder for the site-packages directory, which only pays attention to directory entries ending in one of ['.egg-info', '.egg', '.egg-link']. When I copied the .egg-info directory from one machine to the other, pkg_resources.find_distributions found numpy.

So, this is looking more like a numpy or ports infrastructure bug, not so much like a setuptools bug.
Comment 11 John W. O'Brien 2014-07-15 20:54:22 UTC
The plot thickens.

I've been building and rebuilding math/py-numpy, and been unable to generate a build where the .egg-info directory appears in .PLIST.mktmp, with any version of python. Of course that means that despite having .egg-info in the staging directory, it doesn't make it into the package let alone into the installed system. However, the binary package that the FreeBSD repos are shipping contains .egg-info, as does a binary package my own poudriere instance built for me of the same numpy version on or about 2014-06-03. When I rebuild with that poudriere instance from the latest rev of the ports tree, the package is missing .egg-info.

math/py-numpy doesn't use PYDISTUTILS_AUTOPLIST (could it?), so while I would expect to find entries in pkg-plist for %%PYDISTUTILS_EGGINFO%% there are none. It doesn't look like math/py-numpy ever had those lines, though, so maybe that isn't the problem.
Comment 12 John W. O'Brien 2014-07-16 00:51:22 UTC
Created attachment 144707 [details]
Patch math/py-numpy to install egg-info directory

From Mk/bsd.python.mk --
        .if !defined(PYDISTUTILS_NOEGGINFO) && \
        !defined(PYDISTUTILS_AUTOPLIST) && \
        defined(INSTALLS_EGGINFO) && \
        # [add .egg-info entries to plist]

math/py-numpy does not use AUTOPLIST, nor does it it inhibit with NOEGGINFO. Therefore it needs INSTALLS_EGGINFO.
Comment 13 John W. O'Brien 2014-07-16 01:04:35 UTC
Patch tested successfully with poudriere on 10.0-RELEASE against python{27,33,34}.
Comment 14 John W. O'Brien 2014-07-16 01:05:09 UTC
Created attachment 144708 [details]
Patch math/py-numpy to install egg-info directory and bump PORTREVISION
Comment 15 Kubilay Kocak freebsd_committer 2014-07-16 01:06:49 UTC
I'll take it, thanks for the fantastic diagnostic & isolation effort
Comment 16 commit-hook freebsd_committer 2014-07-16 13:17:12 UTC
A commit references this bug:

Author: koobs
Date: Wed Jul 16 13:16:18 UTC 2014
New revision: 362060
URL: http://svnweb.freebsd.org/changeset/ports/362060

  math/py-numpy: Use INSTALLS_EGGINFO, fixes pkg_resources detection

  Python packages installs .egg-info metadata directories when
  --single-version-externally managed is specified, as the Python ports
  framework does currently for all ports when USE_PYDISTUTILS is defined.

  In this case the py-numpy port does *not* currently define
  PYDISTUTILS_AUTOPLIST nor INSTALLS_EGGINFO, which precludes the contents of
  the add-plist-egginfo: target from being executed.

  This results in an .egg-info metadata directory being built, and staged, but
  not installed.

  Correctly specify INSTALLS_EGGINFO to compensate.

  Additionally add an "Open" Goal Wiki entry [1] to look at automatically detecting the
  presence of .egg-info metadata directories, since all ports leveraging
  USE_PYDISTUTILS also use the --record mechanism for generating installed
  file lists, and potentially deprecating INSTALLS_EGGINFO and related variables.

  [1] https://wiki.freebsd.org/Python#Goals

  PR:		191865
  Submitted by:	 John W. O'Brien <john at saltant dot com>