Created attachment 184682 [details] Update math/py-numpy to 1.13.1 Update math/py-numpy to 1.13.1
Exp-run looks fine.
A commit references this bug: Author: wen Date: Wed Aug 16 06:17:48 UTC 2017 New revision: 448024 URL: https://svnweb.freebsd.org/changeset/ports/448024 Log: - Update to 1.13.1 PR: 220986 Submitted by: wen@(myself) Exp-run by: antoine@ Changes: head/math/py-numpy/Makefile head/math/py-numpy/distinfo head/math/py-numpy/files/patch-numpy_core_include_numpy_npy__endian.h head/math/py-numpy/files/patch-numpy_core_setup.py head/math/py-numpy/pkg-plist
Packaging *fails* with Python 3.6: # make PYTHON_VERSION=python3.6 package pkg-static: Unable to access file /usr/ports/math/py-numpy/work/stage/usr/local/lib/python3.6/site-packages/numpy/__pycache__/__config__.opt-1.cpython-36.pyc:No such file or directory ... pkg-static: Unable to access file /usr/ports/math/py-numpy/work/stage/usr/local/lib/python3.6/site-packages/numpy/__pycache__/version.opt-1.cpython-36.pyc:No such file or directory It seems like the cpython and opt extensions are reversed: ls /usr/ports/math/py-numpy/work/stage/usr/local/lib/python3.6/site-packages/numpy/__pycache__/__config__.*.pyc /usr/ports/math/py-numpy/work/stage/usr/local/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.opt-1.pyc /usr/ports/math/py-numpy/work/stage/usr/local/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc
(In reply to rsmith from comment #3) Thank your report, I shall fix it. wen
Created attachment 185496 [details] [patch] fetch files to version-specific location After the update to 1.13.1, I was getting fetch failures... % make fetch ===> License BSD3CLAUSE accepted by the user ===> py27-numpy-1.13.1,1 depends on file: /usr/local/sbin/pkg - found => reference.pdf doesn't seem to exist in /usr/ports/math/py-numpy/tmp/dist/. => Attempting to fetch https://downloads.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf fetch: https://downloads.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf: Not Found => Attempting to fetch https://cytranet.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf fetch: https://cytranet.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf: Not Found => Attempting to fetch https://dronedata.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf fetch: https://dronedata.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf: No address record => Attempting to fetch https://excellmedia.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf fetch: https://excellmedia.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf: Not Found => Attempting to fetch https://freefr.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf fetch: https://freefr.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf: Not Found . . and so on. The docs for 1.13.1 have not been uploaded to SF yet. To better catch this and not have mismatch failures if you have an old reference.pdf or userguide.pdf, consider fetching to a version-specific subdir. Some other ports use the same technique. See attached patch. Note this patch won't fix the problem of missing docs on the distribution sites. It looks like that has been addressed by caching a local copy on freebsd servers of the old (1.11.2) doc files. If this patch is applied, those could be copied to <distdir>/numpy/1.13.1
Created attachment 185602 [details] Use autoplist This patch applies to the port Makefile as of 448024. To work around the plist problems, I've added 'autoplist' to USE_PYTHON. This made it necessary to add a pre-package target, to remove a stray directory from the plist.
(In reply to rsmith from comment #6) Can't you patch setup.py? pre-package is the wrong target for this.
(In reply to Antoine Brodin from comment #7) I don't really understand how the stray directory ends up in the plist in the first place. So I am unclear as to how patching setup.py would fix this. At first I tried patching the plist in the post-install target. But that didn't work. So my suspicion was that something was updating the plist after post-install. Just for my enlightenment, why is using pre-package wrong?
(In reply to rsmith from comment #8) The plist should be finished at the end of the STAGE phase. check-plist is ran between the STAGE phase and the PACKAGE phase for instance.
(In reply to Antoine Brodin from comment #9) I tried moving it to post-stage, but that didn't work either. Just like post-install.
(In reply to rsmith from comment #10) You may try: POST_PLIST= remove-src-from-plist remove-src-from-plist: ${REINPLACE_CMD} '/src$$/d' ${TMPPLIST}
(In reply to Antoine Brodin from comment #11) Thanks! That did the trick. But now 'make check-orphans' complains. :-( > make check-orphans ====> Checking for pkg-plist issues (check-plist) ===> Parsing plist ===> Checking for items in STAGEDIR missing from pkg-plist Error: Orphaned: man/man1/f2py-%%PYTHON_VER%%.1.gz Error: Orphaned: man/man1/f2py.1.gz ===> Checking for items in pkg-plist which are not in STAGEDIR ===> Error: Plist issues found. *** Error code 1 Stop. Not sure what has to be done here. Does the manpage have to be manually added to ${TMPPLIST} in the post-install target, just like the manpage itself is manually installed and linked?
(In reply to rsmith from comment #12) Try to add back the PLIST_FILES line you removed
Created attachment 185630 [details] Updated autoplist patch Updated the patch to fix plist issues. Additionally, re-built the patch files and the distinfo file to pet portlint. This patch was re-built upon r448445.
(In reply to rsmith from comment #14) The %%FC%% changed to gfortran5 in the patches look wrong. Also, PLIST_FILES+= is preferred over appending to ${TMPPLIST}
Created attachment 185635 [details] Autoplist patch updated again. (In reply to Antoine Brodin from comment #15) Ah. I forgot to merge the patched file. I fixed that. And I switched to using PLIST_FILES+=. Portlint is happy now: "looks fine". I have run some tests. `make package` and `make check-orphans` run fine for Python 2.7 and 3.6. I can install and uninstall the packages. They seem to work OK. Patch has been updated.
(In reply to rsmith from comment #16) Thank you ! wen
Created attachment 185658 [details] Updated patch again to acount for r448473 Merged changes for r448473 in the patch. Wen, can you perhaps look at this an merge it before there are more port changes? Portling is still happy.
*** Bug 221738 has been marked as a duplicate of this bug. ***
Can you remove duplicate lines from distinfo?
Created attachment 185726 [details] Partially revert r436459 After r436459 it is not possible to build math/py-numpy with option OPENBLAS=on. Partial reverting r436459 fixes this problem for me.
Created attachment 185751 [details] Updated patch: removed duplicate distfile lines. (in reply to Antoine Brodin in comment #20) Patch has been updated. Sorry about the noise.
(In reply to Anton Yuzhaninov from comment #21) I'm building with OPENBLAS=on on Python 2.7 and 3.6 on amd64 without problems...
(In reply to rsmith from comment #23) Yes, I was mistaken about build error (probably was caused by other changes in my working copy), but there is a bigger problem: this patch affects blas usage at runtime. Test script: http://termbin.com/w6sl Test script output when numpy built with openblas and patch-numpy-distutils-system_info.py as currently committed: numpy version: 1.13.1 ... a lot of errors skipped ... BLAS info: dot: 31.780785 sec same, but patch-numpy-distutils-system_info.py with r436459 reverted: numpy version: 1.13.1 /usr/local/lib/python3.6/site-packages/numpy/distutils/system_info.py:664: UserWarning: Specified path /usr/local/include/suitesparse is invalid. return self.get_paths(self.section, key) BLAS info: * libraries ['openblasp', 'gfortran', 'openblasp', 'gfortran'] * library_dirs ['/usr/local/lib', '/usr/local/lib/gcc5/gcc/x86_64-portbld-freebsd11.0/5.4.0/../../../'] * language c * define_macros [('HAVE_CBLAS', None)] dot: 1.601915 sec As I can see patch change in r436459 is not really need to build without openblas, at least I was able to build port with netlib blas and without openblas installed (python3). So I still think that partially revert r436459 is a good idea.
Created attachment 185758 [details] configure with OPENBLAS fails (In reply to Anton Yuzhaninov from comment #21) (In reply to rsmith from comment #23) 'make configure' fails for me as well when using WITH=OPENBLAS WITHOUT=NETLIB. This is with gcc5 installed on freebsd 10.3/amd64. See attachment. I don't know if this is the same error Anton is seeing, but reverting the last hunk in allows 'make configure' (and build) to succeed for me. Anton, if you have a different error, attach the build output.
(In reply to John Hein from comment #25) I currently don't see any build errors. Build log (WITH=OPENBLAS WITHOUT=NETLIB) with current ports tree: http://termbin.com/xl4i (but openblas is not used by installed module at runtime) Build log (WITH=OPENBLAS WITHOUT=NETLIB) with patch-numpy-distutils-system_info.py reverted: http://termbin.com/mi5f
Created attachment 185769 [details] Fix packaging, openblas (In reply to Anton Yuzhaninov from comment #24) (In reply to John Hein from comment #25) I concur with Anton's findings. I see the same behavior with the test program on 11.1-STABLE AMD64. I've updated my port patch to include Anton's patch. John, can you try with my updated patch applied to the port? It looks like the build doesn't find openblas (which isn't strange since the test for it was removed) but still continues with cblas and then fails.
(In reply to rsmith from comment #27) Why does update5 include a change to use a hard-coded gfortran5 instead of %FC% in patch-numpy-distutils-fcompiler-gnu.py?
Created attachment 185770 [details] plist fixes, 6th version. (in reply to John Hein in comment #28) I couldn't get Anton's patch to apply cleanly, so I re-created its effects and ran `make makepatch`. Losing %%FC%% was a side effect of that, I think. Should be fixed now. That'll teach me to be careful with `make makepatch`. Ports are complicated. :-/
After the last update I'm not able to compile numpy anymore. Seems something is borking out with gcc? Attaching build log for it.
Created attachment 186063 [details] Build log for failing numpy build after last update
(In reply to Daniel Ylitalo from comment #30) Yes, it doesn't package on amd64. Please apply my patch (plist fixes, 6th version) to the port and try again. That should fix this problem by using autoplist.
(In reply to rsmith from comment #32) autoplist is just a work-around, but we must understand why some files are missing.
(In reply to Thierry Thomas from comment #33) Basically, they're not missing but misnamed. See comment #3. I *think* it goes wrong when the substitution for PYTHON_PYOEXTENSION is done in the plist. Where else could it happen? But looking at /usr/ports/Mk/Uses/python.mk it escapes me *why* it goes wrong.
(In reply to rsmith from comment #34) py3kplist and PYTHON_PYOEXTENSION are not intended to be used together, they should be exclusive
(In reply to Antoine Brodin from comment #35) So should revision 448024 be partially (w.r.t. pkg-plist) reverted then? https://svnweb.freebsd.org/ports/head/math/py-numpy/pkg-plist?r1=423445&r2=448024
(In reply to Thierry Thomas from comment #33) Using autoplist is more than a workaround. It also means not having to maintain a pkg-plist anymore. Having less files to maintain for a port is a worthy goal in itself, IMHO.
(In reply to rsmith from comment #37) When there is an error in the packaging (some missing files, or some files installed in an incorrect directory), it is difficult to notice it. It becomes also impossible to see the diff in the MANIFEST between two versions.
A commit references this bug: Author: antoine Date: Tue Sep 5 18:50:20 UTC 2017 New revision: 449309 URL: https://svnweb.freebsd.org/changeset/ports/449309 Log: Fix packaging with python3*, PYTHON_PYOEXTENSION and py3kplist should not be used together PR: 220986 With hat: portmgr Changes: head/math/py-numpy/pkg-plist
(In reply to commit-hook from comment #39) This fixes the issue for me. The package builds, installs and runs fine now.
py-sip is still broken -- probably, others too. Do we really need to fix them individually or, perhaps, some special massaging of pkg-plist can be done to automatically fix references to .pyo? The way we already fix man-pages (with/without .gz)?
I think this can be closed now, right - no problems left?
(In reply to John Hein from comment #42) The port has been updated again anyway, so this can be closed, I think.
We have version 1.13.3. See comment42 and 43. If there still problems, please new PR.