Created attachment 220299 [details] numpy1194.patch The version of numpy currently in ports doesn't work with python 3.9, here's an update that does. --- notes --- The `${REINPLACE_CMD} -e '\|_examples/$$|d' ${_PYTHONPKGLIST}` line prevents the automatic plist from distutils from including a directory, which would crash pkg, see https://github.com/freebsd/pkg/issues/1911 / https://github.com/freebsd/pkg/pull/1912 The big chunk of files/patch-numpy-distutils-system_info.py that doesn't apply anymore has been removed — I don't think any change like that is necessary now? I have tried both openblas and atlas, make test does run tests with both.
Thanks Greg, AFAIK, NumPy >= 1.19.0 is python 3 only and you should add USES= python:3.6+ instead of USES= python
(In reply to Loïc Bartoletti from comment #1) And if it's the case, all ports depending on it have to be changed to python:3.6+ too
Created attachment 220988 [details] Numpy 1.9.0 and bump dependent ports
You're right, I updated all the ports. A note for math/plot, I tested this port with python:3.6+ and it compiles by adapting the plist. Can you run an exp-run, please? Thanks
More ports that need to be adjusted: [00:01:53] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package vera++-1.3.0_14 from devel/vera++ -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR? [00:01:55] Error: compute_deps_pkg failed to lookup pkgname for math/py-matplotlib2@py27 processing package vtk6-6.2.0_13 from math/vtk6 -- Is SUBDIR+=py-matplotlib2@py27 missing in math/Makefile and does the port provide the 'py27' FLAVOR? [00:01:56] Error: compute_deps_pkg failed to lookup pkgname for devel/py-game@py27 processing package py27-kaa-base-0.6.0_12 from multimedia/py-kaa-base -- Is SUBDIR+=py-game@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR? [00:01:56] Error: compute_deps_pkg failed to lookup existing pkgname for devel/py-game@py27 processing package py27-kaa-base-0.6.0_12 [00:01:56] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package ceph12-12.2.12_2 from net/ceph12 -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR? [00:02:01] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package py27-tagpy-2013.1_21 from audio/py-tagpy@py27 -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR? [00:02:01] Error: compute_deps_pkg failed to lookup pkgname for math/py-pandas@py27 processing package py27-pandas-charm-0.3.0_1 from biology/py-pandas-charm@py27 -- Is SUBDIR+=py-pandas@py27 missing in math/Makefile and does the port provide the 'py27' FLAVOR? [00:02:01] Error: compute_deps_pkg failed to lookup existing pkgname for math/py-pandas@py27 processing package py27-pandas-charm-0.3.0_1 [00:02:01] Error: compute_deps_pkg failed to lookup pkgname for cad/py-gdspy@py27 processing package py27-phidl-1.0.1_1 from cad/py-phidl@py27 -- Is SUBDIR+=py-gdspy@py27 missing in cad/Makefile and does the port provide the 'py27' FLAVOR? [00:02:01] Error: compute_deps_pkg failed to lookup existing pkgname for cad/py-gdspy@py27 processing package py27-phidl-1.0.1_1 [00:02:01] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package py27-YouCompleteMe-g20191130_1 from devel/youcompleteme@py27 -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR? [00:02:01] Error: compute_deps_pkg failed to lookup pkgname for devel/py-game@py27 processing package py27-hypatia_engine-0.3.1_4 from games/hypatia_engine@py27 -- Is SUBDIR+=py-game@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR? [00:02:01] Error: compute_deps_pkg failed to lookup existing pkgname for devel/py-game@py27 processing package py27-hypatia_engine-0.3.1_4 [00:02:01] Error: compute_deps_pkg failed to lookup pkgname for devel/py-game@py27 processing package py27-pyspacewar-1.1.1_1 from games/pyspacewar@py27 -- Is SUBDIR+=py-game@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR? [00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for devel/py-game@py27 processing package py27-pyspacewar-1.1.1_1 [00:02:02] Error: compute_deps_pkg failed to lookup pkgname for graphics/py-gdal@py27 processing package py27-Fiona-1.8.18 from graphics/py-fiona@py27 -- Is SUBDIR+=py-gdal@py27 missing in graphics/Makefile and does the port provide the 'py27' FLAVOR? [00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for graphics/py-gdal@py27 processing package py27-Fiona-1.8.18 [00:02:02] Error: compute_deps_pkg failed to lookup pkgname for devel/py-game@py27 processing package py27-pyganim-0.9.2_3 from graphics/py-pyganim@py27 -- Is SUBDIR+=py-game@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR? [00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for devel/py-game@py27 processing package py27-pyganim-0.9.2_3 [00:02:02] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package py27-piranha-0.11_7 from math/py-piranha@py27 -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR? [00:02:02] Error: compute_deps_pkg failed to lookup pkgname for math/py-pandas@py27 processing package py27-pandas-datareader-0.8.1 from math/py-pandas-datareader@py27 -- Is SUBDIR+=py-pandas@py27 missing in math/Makefile and does the port provide the 'py27' FLAVOR? [00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for math/py-pandas@py27 processing package py27-pandas-datareader-0.8.1 [00:02:02] Error: compute_deps_pkg failed to lookup pkgname for misc/py-onnx@py27 processing package py27-onnx-tf-1.6.0 from misc/py-onnx-tf@py27 -- Is SUBDIR+=py-onnx@py27 missing in misc/Makefile and does the port provide the 'py27' FLAVOR? [00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for misc/py-onnx@py27 processing package py27-onnx-tf-1.6.0 [00:02:02] Error: compute_deps_pkg failed to lookup pkgname for devel/py-flatbuffers@py27 processing package py27-tflite-2.3.0 from misc/py-tflite@py27 -- Is SUBDIR+=py-flatbuffers@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR? [00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for devel/py-flatbuffers@py27 processing package py27-tflite-2.3.0 [00:02:02] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package py27-libtorrent-rasterbar-1.2.10 from net-p2p/py-libtorrent-rasterbar@py27 -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR? [00:02:02] Error: compute_deps_pkg failed to lookup pkgname for science/py-netCDF4@py27 processing package py27-netcdf-flattener-1.2.0 from science/py-netcdf-flattener@py27 -- Is SUBDIR+=py-netCDF4@py27 missing in science/Makefile and does the port provide the 'py27' FLAVOR? [00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for science/py-netCDF4@py27 processing package py27-netcdf-flattener-1.2.0 [00:02:02] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package py27-dlib-19.21 from science/py-dlib@py27 -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR? [00:02:02] Error: compute_deps_pkg failed to lookup pkgname for textproc/py-elasticsearch6@py27 processing package py27-elasticsearch-dsl-6.4.0_1 from textproc/py-elasticsearch-dsl@py27 -- Is SUBDIR+=py-elasticsearch6@py27 missing in textproc/Makefile and does the port provide the 'py27' FLAVOR? [00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for textproc/py-elasticsearch6@py27 processing package py27-elasticsearch-dsl-6.4.0_1 [00:02:02] Error: compute_deps_pkg failed to lookup pkgname for textproc/py-elasticsearch6@py27 processing package py27-elasticsearch-curator-5.6.0_1 from textproc/py-elasticsearch-curator@py27 -- Is SUBDIR+=py-elasticsearch6@py27 missing in textproc/Makefile and does the port provide the 'py27' FLAVOR? [00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for textproc/py-elasticsearch6@py27 processing package py27-elasticsearch-curator-5.6.0_1
Created attachment 221265 [details] numpy 1.9 and dependent ports v2 Here is the updated version which includes the latest python commits.
Some new failure logs: http://package18.nyi.freebsd.org/data/121amd64-default-PR244494/2021-01-04_17h31m54s/logs/py37-pandas-0.24.2_2,1.log http://package18.nyi.freebsd.org/data/121amd64-default-PR244494/2021-01-04_17h31m54s/logs/orange3-3.19.0_5.log
(In reply to Antoine Brodin from comment #7) Regarding the log file for py-pandas: There is a bug #251412 with an update for py-pandas from 0.24.2 to 1.15.0, ready to commit ;) Maybe the error with the undeclared identifier 'NUMPY_IMPORT_ARRAY_RETVAL' does not occur with this version of py-pandas (untested)?
(In reply to Rainer Hurling from comment #8) I tried your patch of py-pandas and it built fine with numpy 1.9, please commit it :)
Created attachment 223412 [details] py-numpy 1.19.4 v4 - update numpy to 1.19.4 - While here replace `${PYTHON_PKGNAMEPREFIX}numpy.*:math/py-numpy@${PY_FLAVOR}` by `${PYNUMPY}` - replace `USES=python:3.[4-5]` by `USES=python:3.6` - bump portrevision - replace range version in `Mk/Uses/python.mk` (workaround for https://github.com/freebsd/pkg/pull/1944)
Something is wrong, all ports depending on numpy are failing Installing py37-numpy-1.19.4... the most recent version of py37-numpy-1.19.4 is already installed ===> py37-metpy-1.0 depends on package: py37-numpy>=1.16.0,1 - not found
Created attachment 223519 [details] py-numpy 1.19.4 v5 antoine@ koobs@ The problem seems related to https://github.com/freebsd/pkg/pull/1944 I fix it removing portepoch (,1) in the version of numpy requirements. Tested against matplotlib on updated poudriere and cleaned packages.
You can remove PORTEPOCH=1 from math/py-numpy/Makefile , you have to add it back
Please consider updating to a newer release. openblas-0.3.13 does have a much larger memory footprint which causes numpy to segfault on import (virtual memory is limited per user in our environment) A more recent release of numpy fixes this by setting the BUFFERSIZE back to 20. Also see: https://github.com/numpy/numpy/pull/17924 https://github.com/xianyi/OpenBLAS/issues/3180 https://github.com/xianyi/OpenBLAS/issues/2970
Created attachment 224438 [details] numpy1202.patch Here's my version of the patch @ 1.20.2
This fails to fetch: => Attempting to fetch https://numpy.org/doc/1.20/numpy-ref.pdf fetch: https://numpy.org/doc/1.20/numpy-ref.pdf: size mismatch: expected 7127190, actual 7362241
Also, the doc should be put in a versioned directory or with a version in the filename
Created attachment 225259 [details] Upgrade NUMPY to 1.20.3
Comment on attachment 225259 [details] Upgrade NUMPY to 1.20.3 Sorry, this patch is not viable. Will try to provide another one.
Created attachment 225274 [details] Upgrade NUMPY to 1.20.3 Adding a new patch: - autoplist does not work in this case, a plist has been added; - some tests fail, but this is not FreeBSD-specific, see https://github.com/numpy/numpy/issues/18471#issuecomment-848684221
Something is missing, all ports fail with: ===> py38-foo depends on package: py38-numpy>=1.16,1<1.20,1 - not found
(In reply to Antoine Brodin from comment #21) Indeed, Mk/Uses/python.mk must be adapted: PYNUMPY= ${PYTHON_PKGNAMEPREFIX}numpy>=1.16,1<1.20,1:math/py-numpy@${PY_FLAVOR} @python team, what about this patch? diff --git a/Mk/Uses/python.mk b/Mk/Uses/python.mk index fcaf479d8076..5e2682d8efd6 100644 --- a/Mk/Uses/python.mk +++ b/Mk/Uses/python.mk @@ -640,7 +640,7 @@ CMAKE_ARGS+= -DPython_ADDITIONAL_VERSIONS=${PYTHON_VER} # Python 3rd-party modules PYGAME= ${PYTHON_PKGNAMEPREFIX}game>0:devel/py-game@${PY_FLAVOR} -PYNUMPY= ${PYTHON_PKGNAMEPREFIX}numpy>=1.16,1<1.20,1:math/py-numpy@${PY_FLAVOR} +PYNUMPY= ${PYTHON_PKGNAMEPREFIX}numpy>=1.16,1<1.21,1:math/py-numpy@${PY_FLAVOR} # Common Python modules that can be needed but only for some versions of Python. .if ${PYTHON_REL} < 30500
Shared macro's (depends lines) in python.mk are fundamentally incompatible with the way Python dependency declarations (version-specs) work (packages can depend individually and arbitrarily on any version of a dependency they way), and will be removed going forward.
Thank you for the update Greg. This can land and be merged with QA confirmation.
(In reply to Thierry Thomas from comment #20) What happens/errors with autoplist?
Exp-run is still ongoing with the additional patch Thierry provided.
(In reply to Kubilay Kocak from comment #25) It produces an uncomplete plist.
^Triage: portmgr doesn't need issue assignment to run an exp-run when the flag is requested (In reply to Thierry Thomas from comment #27) incomplete autoplists are almost always indicative of bugs. Can you attach a file containing the list of whats missing? Also note that one can use autoplist in tandem with pkg-plist (for those missing files, if we cant identify the issue) With the manual plist, there also appears to be some potentially problematic entries, specifically those that need to be installed with python-version suffixed filenames: +bin/f2py +bin/f2py3
(In reply to Kubilay Kocak from comment #28) Apologies, forgot to explicit mention that the problematic aspect is with the use of 'concurrent'. The only package version that should install a 'suffix-less' filename is the 'default (python) version'. I'm not sure how the semantics of this change with 'allflavors' usage, which also begs the question, what are we using allflavors for?
py37-numpy fails to package: http://package18.nyi.freebsd.org/data/122amd64-default-foo/2021-05-31_08h30m42s/logs/errors/py37-numpy-1.20.3,1.log New failure logs: http://package18.nyi.freebsd.org/data/122amd64-default-foo/2021-05-31_08h30m42s/logs/errors/py38-qutip-4.6.1.log http://package18.nyi.freebsd.org/data/122amd64-default-foo/2021-05-31_08h30m42s/logs/errors/py38-geometer-0.2.3.log
(In reply to Kubilay Kocak from comment #28) Adding autoplist to USE_PYTHON and removing pkg-plist produces this error: $ make check-plist ====> Checking for pkg-plist issues (check-plist) ===> Parsing plist ===> Checking for items in STAGEDIR missing from pkg-plist ===> Checking for items in pkg-plist which are not in STAGEDIR Error: Missing: %%PYTHON_SITELIBDIR%%/numpy/random/_examples/ ===> Error: Plist issues found. *** Error code 1 Stop. make: stopped in /usr/ports/math/py-numpy Does it work for you?
(In reply to Antoine Brodin from comment #30) 1) science/py-geometer This port does not use PYNUMPY but a custom line restricted to numpy>=1.15,1<1.20,1 I guess that this restriction is somewhat arbitrary… See the attached patch science_py-geometer.diff 2) devel/py-qutip The restriction is not in the Makefile of the port, but in the setup.cfg file. The attached patch devel_py-qutip.diff fixed the problem for me.
Created attachment 225466 [details] Fix science/py-geometer with a recent Numpy Fix science/py-geometer with a recent Numpy
Created attachment 225467 [details] Fix devel/py-qutip with a recent Numpy Fix devel/py-qutip with a recent Numpy.
(In reply to Antoine Brodin from comment #30) py37-numpy-1.20.3,1 builds OK, but the staging fails: all optimized .pyc are missing. Maybe someone with more Python clue could explain that and fix it?
(In reply to Thierry Thomas from comment #31) Interesting, thanks for that. This appears to be a common bug in upstreams where a data_dir is added to the --record output. The best way to address this is to remove the entry (modify the .pymodtmp pkg-plist before its written), or fix the offending setup.py numpy/distutils issue
(In reply to Kubilay Kocak from comment #36) Kubilay, As much as I can admit the usage of autoplist for lazy porting of a minor port which installs 5 or 6 files, for a port as important as Numpy, I don't understand it , and IMHO it would be a mistake, especially if you have to patch it in order to fix autoplist! - a static plist allows you to simply answer the question "which port installs this particular file?" with a grep even if not installed; - during an upgrade, this is easy to see if some important files are missing due to an unoticed error. For example, the case reported by Antoine in Comment 35 (optimized files missing with Python-3.7) would not even have been noticed with an autoplist!
(In reply to Thierry Thomas from comment #35) I think that the plist issue is due to %%PYTHON_EXT_SUFFIX%% being used instead of cpython-%%PYTHON_SUFFIX%% in pyc lines ( %%PYTHON_EXT_SUFFIX%% should only be used in .so lines )
Created attachment 225511 [details] math/py-numpy: upgrade to 1.20.3 Thanks Antoine! This new patch adopted your idea. Note: the previous plist had been generated from `make makeplist', so makeplist must be fixed too!
(In reply to Thierry Thomas from comment #39) Can you clarify where this is at wrt QA? The exp-run has not been ack'd, the latest patch (and we have three), does it pass QA? Does it need another exp-run? With no open questions, python has no issue accepting an explicitly QA'd changeset. Other open questions: With the other port fixes: +- numpy>=1.16.6,<1.20 ++ numpy>=1.16.6,<1.21 Does the software pass tests with 1.21 ? Are there refs upstream that <1.20 is unnecessarily limiting, or has it been bumped upstream? -RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}numpy>=1.15,1<1.20,1:math/py-numpy@${PY_FLAVOR} \ +RUN_DEPENDS= ${PYNUMPY} \ Shared macros don't work with Python package dependency declarations, and we're removing instances of these. autoplist is preferred. I agree there are some cases where autoplists can cause lack of observability, but there are plenty of cases where autoplists assist immensely. We should have a discussion about improving the situation (possibly to have our cake and eat it to) offline. In the meantime autoplist is preferred, especially for complex ports, and particularly for C extension based python ports. It picks up enough upstream packaging issues that cause other issues downstream for us that its beneficial overall.
(In reply to Kubilay Kocak from comment #40) There is many points in your comment. For what I'm concerned with: - the proposed patch (the latest, of course) is OK for the ports I tested, but of course I have not tested all of them, and an exp-run is necessary (you asked Antoine to stop it) - for what I'm seeing, most ports are limiting to a version unnecessarily, just because they have not been tested with a higher version. If we remove shared macros for ports like numpy, it means that we must support several versions of these ports, like py-numpy-legacy and py-numpy, without conflicts. - I do not see any value in auto-plist, excepted lazy-porting, but if you want it, just do it, you are the maintainer. I have not been able to make it work with autoplist, this is why I have generated a static one.
(In reply to Thierry Thomas from comment #41) It is an issue, but it is the price we pay for wanting to package Python programs We can address the issue in several ways, none resolving it completely. This is a conversation for the Python team that we'd love interested people to participate in. Shared macros though, don't fix the problem, they make the problem worse. Ports build because the 'version spec' is apparently 'satisfied', but then are broken at rutime, because they're actually not. There is no such thing as a shared version spec in the Python world. Every upstream package declares their dependencies individually and arbitrarily. We can pretend they're 'shared' to make builds work, but this doesn't mean 'usable' software for ports and package users. I'll respond to the exp-run issue offline
QA: as requested by koobs, I launched poudrière for the ports math/py-numpy, devel/py-qutip and science/py-geometer in the following environments: - 13i386{defaultpython} - 12amd64 - 12amd64{python3.9) The resulting logs are available at: https://people.freebsd.org/~thierry/12amd64_py38-geometer-0.2.3.log.gz https://people.freebsd.org/~thierry/12amd64_py38-numpy-1.20.3,1.log.gz https://people.freebsd.org/~thierry/12amd64_py38-qutip-4.6.2.log.gz https://people.freebsd.org/~thierry/12amd64_py39-geometer-0.2.3.log.gz https://people.freebsd.org/~thierry/12amd64_py39-numpy-1.20.3,1.log.gz https://people.freebsd.org/~thierry/12amd64_py39-qutip-4.6.2.log.gz https://people.freebsd.org/~thierry/13i386_py38-geometer-0.2.3.log.gz https://people.freebsd.org/~thierry/13i386_py38-numpy-1.20.3,1.log.gz https://people.freebsd.org/~thierry/13i386_py38-qutip-4.6.2.log.gz
(In reply to Thierry Thomas from comment #33) PYGAME and PYNUMPY should be removed some day. Unlike other PY_foo, they do not bring any benefit but make things more complex. For example, it's getting much harder to check version requirements of dependent ports. Please do not revert the change for removing PYNUMPY. The correct way is to patch the source (setup.py, setup.cfg, requestments.txt or other files) and update RUN_DEPENDS. In this case, it's setup.py. (In reply to Thierry Thomas from comment #39) Note that numpy 1.20.* requires Python 3.7. For a complete patch, you forgot to change USES=python:3.6+ to USES=python:3.7+ and update all dependent ports as well.