Test environment: root@teratoid:~ # uname -a FreeBSD teratoid.n.saltant.net 10.0-STABLE FreeBSD 10.0-STABLE #0 r266389: Sat May 17 23:21:39 EDT 2014 root@drivel.saltant.net:/usr/obj/usr/src/sys/NARB amd64 root@teratoid:~ # pkg info -aq binutils-2.24 blas-3.4.2_2 dialog4ports-0.1.5_2 gcc-4.7.3_1 gcc-ecj-4.5 gettext-0.18.3.1_1 gmake-3.82_1 gmp-5.1.3_2 indexinfo-0.2 lapack-3.4.2_2 mpc-1.0.2 mpfr-3.1.2_2 pkg-1.2.7_4 py33-nose-1.3.0_1 py33-numpy-1.8.1,1 py33-setuptools33-5.1 python33-3.3.5_1 suitesparse-4.0.2_3 How to repeat: root@teratoid:~ # python3.3 Python 3.3.5 (default, Jul 14 2014, 11:17:18) [GCC 4.2.1 Compatible FreeBSD Clang 3.4 (tags/RELEASE_34/final 197956)] on freebsd10 Type "help", "copyright", "credits" or "license" for more information. >>> import pkg_resources >>> import sys >>> for d in sys.path: ... print("Searching {}".format(d)) ... for dist in pkg_resources.find_distributions(d): ... print(dist) ... Searching Searching /usr/local/lib/python33.zip Searching /usr/local/lib/python3.3 Searching /usr/local/lib/python3.3/plat-freebsd10 Searching /usr/local/lib/python3.3/lib-dynload Searching /root/.local/lib/python3.3/site-packages Searching /usr/local/lib/python3.3/site-packages setuptools 5.1 nose 1.3.0 >>> import numpy >>> numpy.__file__ '/usr/local/lib/python3.3/site-packages/numpy/__init__.py' >>> This also manifests itself when setup.py runs with setup(setup_requires="numpy >= 1.7.0b2"), for example, because the underlying pkg_resources.require("numpy >= 1.7.0b2") raises DistributionNotFound.
An update to devel/py-setuptools to 5.4.1 is about to land shortly (pending a small QA issue). Can you take a look at the upstream changes [1] and let me know if there's anything in there that might be relevent. https://bitbucket.org/pypa/setuptools/src/b7581d108b877e360bf8311089e8dff27a53c6f0/CHANGES.txt?at=default If possible, it would also be handy to reproduce the issue on lang/python34, to isolate a root cause to a specific port
Also, a directory listing of the contents of /usr/local/lib/python3.3/site-packages could prove handy (please attach it, rather than in a comment)
Created attachment 144664 [details] Recursive listing of python3.3 site-packages directory
koobs: I see nothing relevant in the upstream changelog for setuptools. I'm working on reproducing in 3.4 right now.
Created attachment 144667 [details] Test environment with DEFAULT_VERSIONS=python=3.4 To upgrade the test environment, I pkg remove'd the three py33-* packages plus python itself, updated /etc/make.conf, then built and installed lang/python3.4, devel/py-setuptools34, devel/py-nose, and math/py-numpy.
Created attachment 144668 [details] Demonstation of problem under python3.4 This terminal capture shows the same behavior under python3.4.
Created attachment 144669 [details] Recursive listing of python3.4 site-packages directory
See also bug 191857, comment 1, regarding a previous notice to freebsd-pkg-fallout@, which suggests that this bug affects 2.7 too, at least on 11.0-CURRENT.
Oh, that's interesting and unexpected... I just reproduced this with 2.7 in one test environment, but could not in another. I wonder what's different?
What was different is that on one machine, /usr/local/lib/python2.7/site-packages/numpy-1.8.1-py2.7.egg-info existed, and on the other it didn't. This is a problem because the pkg_resources.find_distributions function ends up using the pkg_resources.find_on_path finder for the site-packages directory, which only pays attention to directory entries ending in one of ['.egg-info', '.egg', '.egg-link']. When I copied the .egg-info directory from one machine to the other, pkg_resources.find_distributions found numpy. So, this is looking more like a numpy or ports infrastructure bug, not so much like a setuptools bug.
The plot thickens. I've been building and rebuilding math/py-numpy, and been unable to generate a build where the .egg-info directory appears in .PLIST.mktmp, with any version of python. Of course that means that despite having .egg-info in the staging directory, it doesn't make it into the package let alone into the installed system. However, the binary package that the FreeBSD repos are shipping contains .egg-info, as does a binary package my own poudriere instance built for me of the same numpy version on or about 2014-06-03. When I rebuild with that poudriere instance from the latest rev of the ports tree, the package is missing .egg-info. math/py-numpy doesn't use PYDISTUTILS_AUTOPLIST (could it?), so while I would expect to find entries in pkg-plist for %%PYDISTUTILS_EGGINFO%% there are none. It doesn't look like math/py-numpy ever had those lines, though, so maybe that isn't the problem.
Created attachment 144707 [details] Patch math/py-numpy to install egg-info directory From Mk/bsd.python.mk -- add-plist-egginfo: .if !defined(PYDISTUTILS_NOEGGINFO) && \ !defined(PYDISTUTILS_AUTOPLIST) && \ defined(INSTALLS_EGGINFO) && \ defined(PYTHON_REL) # [add .egg-info entries to plist] math/py-numpy does not use AUTOPLIST, nor does it it inhibit with NOEGGINFO. Therefore it needs INSTALLS_EGGINFO.
Patch tested successfully with poudriere on 10.0-RELEASE against python{27,33,34}.
Created attachment 144708 [details] Patch math/py-numpy to install egg-info directory and bump PORTREVISION
I'll take it, thanks for the fantastic diagnostic & isolation effort
A commit references this bug: Author: koobs Date: Wed Jul 16 13:16:18 UTC 2014 New revision: 362060 URL: http://svnweb.freebsd.org/changeset/ports/362060 Log: math/py-numpy: Use INSTALLS_EGGINFO, fixes pkg_resources detection Python packages installs .egg-info metadata directories when --single-version-externally managed is specified, as the Python ports framework does currently for all ports when USE_PYDISTUTILS is defined. In this case the py-numpy port does *not* currently define PYDISTUTILS_AUTOPLIST nor INSTALLS_EGGINFO, which precludes the contents of the add-plist-egginfo: target from being executed. This results in an .egg-info metadata directory being built, and staged, but not installed. Correctly specify INSTALLS_EGGINFO to compensate. Additionally add an "Open" Goal Wiki entry [1] to look at automatically detecting the presence of .egg-info metadata directories, since all ports leveraging USE_PYDISTUTILS also use the --record mechanism for generating installed file lists, and potentially deprecating INSTALLS_EGGINFO and related variables. [1] https://wiki.freebsd.org/Python#Goals PR: 191865 Submitted by: John W. O'Brien <john at saltant dot com> Changes: head/math/py-numpy/Makefile