Bug 220986 - [exp-run] Update math/py-numpy to 1.13.1
Summary: [exp-run] Update math/py-numpy to 1.13.1
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Wen Heping
: 221738 (view as bug list)
Depends on:
Reported: 2017-07-25 02:31 UTC by Wen Heping
Modified: 2018-02-18 19:10 UTC (History)
11 users (show)

See Also:
bugzilla: maintainer-feedback? (python)

Update math/py-numpy to 1.13.1 (51.43 KB, text/plain)
2017-07-25 02:31 UTC, Wen Heping
no flags Details
[patch] fetch files to version-specific location (1.75 KB, patch)
2017-08-16 19:55 UTC, John Hein
no flags Details | Diff
Use autoplist (45.23 KB, patch)
2017-08-20 12:02 UTC, rsmith
no flags Details | Diff
Updated autoplist patch (53.18 KB, patch)
2017-08-21 08:00 UTC, rsmith
no flags Details | Diff
Autoplist patch updated again. (50.69 KB, patch)
2017-08-21 10:43 UTC, rsmith
no flags Details | Diff
Updated patch again to acount for r448473 (50.71 KB, patch)
2017-08-22 09:45 UTC, rsmith
no flags Details | Diff
Partially revert r436459 (1.18 KB, patch)
2017-08-24 18:56 UTC, Anton Yuzhaninov
no flags Details | Diff
Updated patch: removed duplicate distfile lines. (50.10 KB, patch)
2017-08-25 10:00 UTC, rsmith
no flags Details | Diff
configure with OPENBLAS fails (28.14 KB, text/plain)
2017-08-25 16:17 UTC, John Hein
no flags Details
Fix packaging, openblas (52.26 KB, patch)
2017-08-25 21:24 UTC, rsmith
no flags Details | Diff
plist fixes, 6th version. (50.23 KB, patch)
2017-08-25 22:48 UTC, rsmith
no flags Details | Diff
Build log for failing numpy build after last update (404.99 KB, text/plain)
2017-09-04 14:51 UTC, Daniel Ylitalo
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Wen Heping freebsd_committer 2017-07-25 02:31:36 UTC
Created attachment 184682 [details]
Update math/py-numpy to 1.13.1

Update math/py-numpy to 1.13.1
Comment 1 Antoine Brodin freebsd_committer 2017-08-16 05:17:13 UTC
Exp-run looks fine.
Comment 2 commit-hook freebsd_committer 2017-08-16 06:18:38 UTC
A commit references this bug:

Author: wen
Date: Wed Aug 16 06:17:48 UTC 2017
New revision: 448024
URL: https://svnweb.freebsd.org/changeset/ports/448024

  - Update to 1.13.1

  PR:		220986
  Submitted by:	wen@(myself)
  Exp-run by:	antoine@

Comment 3 rsmith 2017-08-16 09:05:06 UTC
Packaging *fails* with Python 3.6:

# make PYTHON_VERSION=python3.6 package
pkg-static: Unable to access file /usr/ports/math/py-numpy/work/stage/usr/local/lib/python3.6/site-packages/numpy/__pycache__/__config__.opt-1.cpython-36.pyc:No such file or directory


pkg-static: Unable to access file /usr/ports/math/py-numpy/work/stage/usr/local/lib/python3.6/site-packages/numpy/__pycache__/version.opt-1.cpython-36.pyc:No such file or directory

It seems like the cpython and opt extensions are reversed:

ls /usr/ports/math/py-numpy/work/stage/usr/local/lib/python3.6/site-packages/numpy/__pycache__/__config__.*.pyc
Comment 4 Wen Heping freebsd_committer 2017-08-16 09:11:31 UTC
(In reply to rsmith from comment #3)
Thank your report, I shall fix it.

Comment 5 John Hein 2017-08-16 19:55:29 UTC
Created attachment 185496 [details]
[patch] fetch files to version-specific location

After the update to 1.13.1, I was getting fetch failures...

% make fetch
===>  License BSD3CLAUSE accepted by the user
===>   py27-numpy-1.13.1,1 depends on file: /usr/local/sbin/pkg - found
=> reference.pdf doesn't seem to exist in /usr/ports/math/py-numpy/tmp/dist/.
=> Attempting to fetch https://downloads.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf
fetch: https://downloads.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf: Not Found
=> Attempting to fetch https://cytranet.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf
fetch: https://cytranet.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf: Not Found
=> Attempting to fetch https://dronedata.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf
fetch: https://dronedata.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf: No address record
=> Attempting to fetch https://excellmedia.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf
fetch: https://excellmedia.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf: Not Found
=> Attempting to fetch https://freefr.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf
fetch: https://freefr.dl.sourceforge.net/project/numpy/NumPy/1.13.1/reference.pdf: Not Found

and so on.

The docs for 1.13.1 have not been uploaded to SF yet.

To better catch this and not have mismatch failures if you have an old reference.pdf or userguide.pdf, consider fetching to a version-specific subdir.  Some other ports use the same technique.

See attached patch.  Note this patch won't fix the problem of missing docs on the distribution sites.  It looks like that has been addressed by caching a local copy on freebsd servers of the old (1.11.2) doc files.  If this patch is applied, those could be copied to <distdir>/numpy/1.13.1
Comment 6 rsmith 2017-08-20 12:02:55 UTC
Created attachment 185602 [details]
Use autoplist

This patch applies to the port Makefile as of 448024.

To work around the plist problems, I've added 'autoplist' to USE_PYTHON.
This made it necessary to add a pre-package target, to remove a stray directory from the plist.
Comment 7 Antoine Brodin freebsd_committer 2017-08-20 14:30:29 UTC
(In reply to rsmith from comment #6)
Can't you patch setup.py?   pre-package is the wrong target for this.
Comment 8 rsmith 2017-08-20 15:02:21 UTC
(In reply to Antoine Brodin from comment #7)

I don't really understand how the stray directory ends up in the plist in the first place.

So I am unclear as to how patching setup.py would fix this.

At first I tried patching the plist in the post-install target. But that didn't work. So my suspicion was that something was updating the plist after post-install.

Just for my enlightenment, why is using pre-package wrong?
Comment 9 Antoine Brodin freebsd_committer 2017-08-20 15:33:54 UTC
(In reply to rsmith from comment #8)
The plist should be finished at the end of the STAGE phase.
check-plist is ran between the STAGE phase and the PACKAGE phase for instance.
Comment 10 rsmith 2017-08-20 18:33:15 UTC
(In reply to Antoine Brodin from comment #9)

I tried moving it to post-stage, but that didn't work either.
Just like post-install.
Comment 11 Antoine Brodin freebsd_committer 2017-08-20 19:29:33 UTC
(In reply to rsmith from comment #10)
You may try:
POST_PLIST= remove-src-from-plist

    ${REINPLACE_CMD} '/src$$/d' ${TMPPLIST}
Comment 12 rsmith 2017-08-20 22:09:22 UTC
(In reply to Antoine Brodin from comment #11)

Thanks! That did the trick.

But now 'make check-orphans' complains. :-(

> make check-orphans
====> Checking for pkg-plist issues (check-plist)
===> Parsing plist
===> Checking for items in STAGEDIR missing from pkg-plist
Error: Orphaned: man/man1/f2py-%%PYTHON_VER%%.1.gz
Error: Orphaned: man/man1/f2py.1.gz
===> Checking for items in pkg-plist which are not in STAGEDIR
===> Error: Plist issues found.
*** Error code 1

Not sure what has to be done here.

Does the manpage have to be manually added to ${TMPPLIST} in the post-install target, just like the manpage itself is manually installed and linked?
Comment 13 Antoine Brodin freebsd_committer 2017-08-21 05:46:23 UTC
(In reply to rsmith from comment #12)
Try to add back the PLIST_FILES line you removed
Comment 14 rsmith 2017-08-21 08:00:24 UTC
Created attachment 185630 [details]
Updated autoplist patch

Updated the patch to fix plist issues.

Additionally, re-built the patch files and the distinfo file to pet portlint.

This patch was re-built upon r448445.
Comment 15 Antoine Brodin freebsd_committer 2017-08-21 08:32:40 UTC
(In reply to rsmith from comment #14)
The %%FC%% changed to gfortran5 in the patches look wrong.
Also,  PLIST_FILES+= is preferred over appending to ${TMPPLIST}
Comment 16 rsmith 2017-08-21 10:43:01 UTC
Created attachment 185635 [details]
Autoplist patch updated again.

(In reply to Antoine Brodin from comment #15)

Ah. I forgot to merge the patched file. I fixed that.
And I switched to using PLIST_FILES+=.

Portlint is happy now: "looks fine".

I have run some tests. `make package` and `make check-orphans` run fine for Python 2.7 and 3.6. I can install and uninstall the packages. They seem to work OK.

Patch has been updated.
Comment 17 Wen Heping freebsd_committer 2017-08-21 10:45:50 UTC
(In reply to rsmith from comment #16)
Thank you !

Comment 18 rsmith 2017-08-22 09:45:21 UTC
Created attachment 185658 [details]
Updated patch again to acount for r448473

Merged changes for r448473 in the patch.

Wen, can you perhaps look at this an merge it before there are more port changes?

Portling is still happy.
Comment 19 Antoine Brodin freebsd_committer 2017-08-23 12:53:03 UTC
*** Bug 221738 has been marked as a duplicate of this bug. ***
Comment 20 Antoine Brodin freebsd_committer 2017-08-24 18:50:20 UTC
Can you remove duplicate lines from distinfo?
Comment 21 Anton Yuzhaninov 2017-08-24 18:56:12 UTC
Created attachment 185726 [details]
Partially revert r436459

After r436459 it is not possible to build math/py-numpy with option OPENBLAS=on. Partial reverting r436459 fixes this problem for me.
Comment 22 rsmith 2017-08-25 10:00:47 UTC
Created attachment 185751 [details]
Updated patch: removed duplicate distfile lines.

(in reply to Antoine Brodin in comment #20)

Patch has been updated. Sorry about the noise.
Comment 23 rsmith 2017-08-25 10:04:27 UTC
(In reply to Anton Yuzhaninov from comment #21)

I'm building with OPENBLAS=on on Python 2.7 and 3.6 on amd64 without problems...
Comment 24 Anton Yuzhaninov 2017-08-25 16:05:24 UTC
(In reply to rsmith from comment #23)
Yes, I was mistaken about build error (probably was caused by other changes in my working copy), but there is a bigger problem: this patch affects blas usage at runtime.

Test script:

Test script output when numpy built with openblas and patch-numpy-distutils-system_info.py as currently committed:

numpy version: 1.13.1
... a lot of errors skipped ...
BLAS info:

dot: 31.780785 sec

same, but patch-numpy-distutils-system_info.py with r436459 reverted:
numpy version: 1.13.1
/usr/local/lib/python3.6/site-packages/numpy/distutils/system_info.py:664: UserWarning: Specified path /usr/local/include/suitesparse is invalid.
  return self.get_paths(self.section, key)
BLAS info:
  * libraries ['openblasp', 'gfortran', 'openblasp', 'gfortran']
  * library_dirs ['/usr/local/lib', '/usr/local/lib/gcc5/gcc/x86_64-portbld-freebsd11.0/5.4.0/../../../']
  * language c
  * define_macros [('HAVE_CBLAS', None)]

dot: 1.601915 sec

As I can see patch change in r436459 is not really need to build without openblas, at least I was able to build port with netlib blas and without openblas installed (python3). So I still think that partially revert r436459 is a good idea.
Comment 25 John Hein 2017-08-25 16:17:31 UTC
Created attachment 185758 [details]
configure with OPENBLAS fails

(In reply to Anton Yuzhaninov from comment #21)
(In reply to rsmith from comment #23)
'make configure' fails for me as well when using WITH=OPENBLAS WITHOUT=NETLIB.  This is with gcc5 installed on freebsd 10.3/amd64.  See attachment.

I don't know if this is the same error Anton is seeing, but reverting the last hunk in allows 'make configure' (and build) to succeed for me.

Anton, if you have a different error, attach the build output.
Comment 26 Anton Yuzhaninov 2017-08-25 18:01:17 UTC
(In reply to John Hein from comment #25)
I currently don't see any build errors.

Build log (WITH=OPENBLAS WITHOUT=NETLIB) with current ports tree:
(but openblas is not used by installed module at runtime)

Build log (WITH=OPENBLAS WITHOUT=NETLIB) with patch-numpy-distutils-system_info.py reverted:
Comment 27 rsmith 2017-08-25 21:24:23 UTC
Created attachment 185769 [details]
Fix packaging, openblas

(In reply to Anton Yuzhaninov from comment #24)
(In reply to John Hein from comment #25)

I concur with Anton's findings. I see the same behavior with the test program on 11.1-STABLE AMD64. I've updated my port patch to include Anton's patch.

John, can you try with my updated patch applied to the port? It looks like the build doesn't find openblas (which isn't strange since the test for it was removed) but still continues with cblas and then fails.
Comment 28 John Hein 2017-08-25 22:08:55 UTC
(In reply to rsmith from comment #27)
Why does update5 include a change to use a hard-coded gfortran5 instead of %FC% in patch-numpy-distutils-fcompiler-gnu.py?
Comment 29 rsmith 2017-08-25 22:48:39 UTC
Created attachment 185770 [details]
plist fixes, 6th version.

(in reply to John Hein in comment #28)

I couldn't get Anton's patch to apply cleanly, so I re-created its effects and ran `make makepatch`. Losing %%FC%% was a side effect of that, I think. Should be fixed now. 

That'll teach me to be careful with `make makepatch`.
Ports are complicated. :-/
Comment 30 Daniel Ylitalo 2017-09-04 14:50:22 UTC
After the last update I'm not able to compile numpy anymore.

Seems something is borking out with gcc? Attaching build log for it.
Comment 31 Daniel Ylitalo 2017-09-04 14:51:02 UTC
Created attachment 186063 [details]
Build log for failing numpy build after last update
Comment 32 rsmith 2017-09-04 17:39:50 UTC
(In reply to Daniel Ylitalo from comment #30)

Yes, it doesn't package on amd64. Please apply my patch (plist fixes, 6th version) to the port and try again. That should fix this problem by using autoplist.
Comment 33 Thierry Thomas freebsd_committer 2017-09-04 18:59:55 UTC
(In reply to rsmith from comment #32)
autoplist is just a work-around, but we must understand why some files are missing.
Comment 34 rsmith 2017-09-04 20:26:22 UTC
(In reply to Thierry Thomas from comment #33)

Basically, they're not missing but misnamed. See comment #3.

I *think* it goes wrong when the substitution for PYTHON_PYOEXTENSION is done in the plist. Where else could it happen?

But looking at /usr/ports/Mk/Uses/python.mk it escapes me *why* it goes wrong.
Comment 35 Antoine Brodin freebsd_committer 2017-09-04 20:30:18 UTC
(In reply to rsmith from comment #34)
py3kplist and PYTHON_PYOEXTENSION are not intended to be used together,  they should be exclusive
Comment 36 rsmith 2017-09-04 23:03:35 UTC
(In reply to Antoine Brodin from comment #35)

So should revision 448024 be partially (w.r.t. pkg-plist) reverted then?

Comment 37 rsmith 2017-09-05 17:47:57 UTC
(In reply to Thierry Thomas from comment #33)

Using autoplist is more than a workaround.

It also means not having to maintain a pkg-plist anymore. Having less files to maintain for a port is a worthy goal in itself, IMHO.
Comment 38 Thierry Thomas freebsd_committer 2017-09-05 17:53:59 UTC
(In reply to rsmith from comment #37)
When there is an error in the packaging (some missing files, or some files installed in an incorrect directory), it is difficult to notice it.
It becomes also impossible to see the diff in the MANIFEST between two versions.
Comment 39 commit-hook freebsd_committer 2017-09-05 18:51:19 UTC
A commit references this bug:

Author: antoine
Date: Tue Sep  5 18:50:20 UTC 2017
New revision: 449309
URL: https://svnweb.freebsd.org/changeset/ports/449309

  Fix packaging with python3*, PYTHON_PYOEXTENSION and py3kplist should not
  be used together

  PR:		220986
  With hat:	portmgr

Comment 40 rsmith 2017-09-05 23:01:27 UTC
(In reply to commit-hook from comment #39)

This fixes the issue for me. 
The package builds, installs and runs fine now.
Comment 41 Mikhail T. 2017-09-26 14:58:32 UTC
py-sip is still broken -- probably, others too. Do we really need to fix them individually or, perhaps, some special massaging of pkg-plist can be done to automatically fix references to .pyo?

The way we already fix man-pages (with/without .gz)?
Comment 42 John Hein 2017-10-20 22:03:42 UTC
I think this can be closed now, right - no problems left?
Comment 43 rsmith 2017-10-21 09:12:20 UTC
(In reply to John Hein from comment #42)

The port has been updated again anyway, so this can be closed, I think.
Comment 44 Walter Schwarzenfeld freebsd_triage 2018-02-18 19:10:38 UTC
We have version 1.13.3. See comment42 and 43. If there still problems, please new PR.