Bug 251626 - math/py-numpy: Update to 1.20.3 (compatible with Python 3.9)
Summary: math/py-numpy: Update to 1.20.3 (compatible with Python 3.9)
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Wen Heping
URL: https://github.com/numpy/numpy/releas...
Keywords:
Depends on: 251412 252540
Blocks: 257476 258493 256814 257723
  Show dependency treegraph
 
Reported: 2020-12-06 00:11 UTC by Val Packett
Modified: 2021-10-26 23:56 UTC (History)
11 users (show)

See Also:
bugzilla: maintainer-feedback? (python)
antoine: exp-run?


Attachments
numpy1194.patch (7.53 KB, patch)
2020-12-06 00:11 UTC, Val Packett
no flags Details | Diff
Numpy 1.9.0 and bump dependent ports (145.82 KB, patch)
2020-12-27 10:18 UTC, Loïc Bartoletti
no flags Details | Diff
numpy 1.9 and dependent ports v2 (119.43 KB, patch)
2021-01-04 06:21 UTC, Loïc Bartoletti
no flags Details | Diff
py-numpy 1.19.4 v4 (159.76 KB, patch)
2021-03-19 05:28 UTC, Loïc Bartoletti
no flags Details | Diff
py-numpy 1.19.4 v5 (159.72 KB, patch)
2021-03-23 10:59 UTC, Loïc Bartoletti
no flags Details | Diff
numpy1202.patch (7.99 KB, patch)
2021-04-26 11:16 UTC, Val Packett
no flags Details | Diff
Upgrade NUMPY to 1.20.3 (9.84 KB, patch)
2021-05-25 18:48 UTC, Thierry Thomas
no flags Details | Diff
Upgrade NUMPY to 1.20.3 (112.65 KB, patch)
2021-05-26 11:27 UTC, Thierry Thomas
no flags Details | Diff
Fix science/py-geometer with a recent Numpy (531 bytes, patch)
2021-06-01 16:47 UTC, Thierry Thomas
no flags Details | Diff
Fix devel/py-qutip with a recent Numpy (502 bytes, patch)
2021-06-01 16:48 UTC, Thierry Thomas
no flags Details | Diff
math/py-numpy: upgrade to 1.20.3 (116.75 KB, patch)
2021-06-02 20:14 UTC, Thierry Thomas
no flags Details | Diff
Update to 1.20.3 and add missing TEST_DEPENDS (116.73 KB, patch)
2021-08-09 03:03 UTC, Wen Heping
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Val Packett 2020-12-06 00:11:11 UTC
Created attachment 220299 [details]
numpy1194.patch

The version of numpy currently in ports doesn't work with python 3.9, here's an update that does.

--- notes ---

The `${REINPLACE_CMD} -e '\|_examples/$$|d' ${_PYTHONPKGLIST}` line prevents the automatic plist from distutils from including a directory, which would crash pkg, see https://github.com/freebsd/pkg/issues/1911 / https://github.com/freebsd/pkg/pull/1912

The big chunk of files/patch-numpy-distutils-system_info.py that doesn't apply anymore has been removed — I don't think any change like that is necessary now? I have tried both openblas and atlas, make test does run tests with both.
Comment 1 Loïc Bartoletti freebsd_committer freebsd_triage 2020-12-24 10:02:53 UTC
Thanks Greg,

AFAIK, NumPy >= 1.19.0 is python 3 only and you should add USES= python:3.6+ instead of USES= python
Comment 2 Antoine Brodin freebsd_committer freebsd_triage 2020-12-24 10:28:08 UTC
(In reply to Loïc Bartoletti from comment #1)
And if it's the case,  all ports depending on it have to be changed to python:3.6+ too
Comment 3 Loïc Bartoletti freebsd_committer freebsd_triage 2020-12-27 10:18:03 UTC
Created attachment 220988 [details]
Numpy 1.9.0 and bump dependent ports
Comment 4 Loïc Bartoletti freebsd_committer freebsd_triage 2020-12-27 10:25:16 UTC
You're right, I updated all the ports. 

A note for math/plot, I tested this port with python:3.6+ and it compiles by adapting the plist.

Can you run an exp-run, please? Thanks
Comment 5 Antoine Brodin freebsd_committer freebsd_triage 2020-12-28 13:53:47 UTC
More ports that need to be adjusted:

[00:01:53] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package vera++-1.3.0_14 from devel/vera++ -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR?

[00:01:55] Error: compute_deps_pkg failed to lookup pkgname for math/py-matplotlib2@py27 processing package vtk6-6.2.0_13 from math/vtk6 -- Is SUBDIR+=py-matplotlib2@py27 missing in math/Makefile and does the port provide the 'py27' FLAVOR?

[00:01:56] Error: compute_deps_pkg failed to lookup pkgname for devel/py-game@py27 processing package py27-kaa-base-0.6.0_12 from multimedia/py-kaa-base -- Is SUBDIR+=py-game@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR?

[00:01:56] Error: compute_deps_pkg failed to lookup existing pkgname for devel/py-game@py27 processing package py27-kaa-base-0.6.0_12

[00:01:56] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package ceph12-12.2.12_2 from net/ceph12 -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:01] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package py27-tagpy-2013.1_21 from audio/py-tagpy@py27 -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:01] Error: compute_deps_pkg failed to lookup pkgname for math/py-pandas@py27 processing package py27-pandas-charm-0.3.0_1 from biology/py-pandas-charm@py27 -- Is SUBDIR+=py-pandas@py27 missing in math/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:01] Error: compute_deps_pkg failed to lookup existing pkgname for math/py-pandas@py27 processing package py27-pandas-charm-0.3.0_1

[00:02:01] Error: compute_deps_pkg failed to lookup pkgname for cad/py-gdspy@py27 processing package py27-phidl-1.0.1_1 from cad/py-phidl@py27 -- Is SUBDIR+=py-gdspy@py27 missing in cad/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:01] Error: compute_deps_pkg failed to lookup existing pkgname for cad/py-gdspy@py27 processing package py27-phidl-1.0.1_1

[00:02:01] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package py27-YouCompleteMe-g20191130_1 from devel/youcompleteme@py27 -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:01] Error: compute_deps_pkg failed to lookup pkgname for devel/py-game@py27 processing package py27-hypatia_engine-0.3.1_4 from games/hypatia_engine@py27 -- Is SUBDIR+=py-game@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:01] Error: compute_deps_pkg failed to lookup existing pkgname for devel/py-game@py27 processing package py27-hypatia_engine-0.3.1_4

[00:02:01] Error: compute_deps_pkg failed to lookup pkgname for devel/py-game@py27 processing package py27-pyspacewar-1.1.1_1 from games/pyspacewar@py27 -- Is SUBDIR+=py-game@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for devel/py-game@py27 processing package py27-pyspacewar-1.1.1_1

[00:02:02] Error: compute_deps_pkg failed to lookup pkgname for graphics/py-gdal@py27 processing package py27-Fiona-1.8.18 from graphics/py-fiona@py27 -- Is SUBDIR+=py-gdal@py27 missing in graphics/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for graphics/py-gdal@py27 processing package py27-Fiona-1.8.18

[00:02:02] Error: compute_deps_pkg failed to lookup pkgname for devel/py-game@py27 processing package py27-pyganim-0.9.2_3 from graphics/py-pyganim@py27 -- Is SUBDIR+=py-game@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for devel/py-game@py27 processing package py27-pyganim-0.9.2_3

[00:02:02] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package py27-piranha-0.11_7 from math/py-piranha@py27 -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:02] Error: compute_deps_pkg failed to lookup pkgname for math/py-pandas@py27 processing package py27-pandas-datareader-0.8.1 from math/py-pandas-datareader@py27 -- Is SUBDIR+=py-pandas@py27 missing in math/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for math/py-pandas@py27 processing package py27-pandas-datareader-0.8.1

[00:02:02] Error: compute_deps_pkg failed to lookup pkgname for misc/py-onnx@py27 processing package py27-onnx-tf-1.6.0 from misc/py-onnx-tf@py27 -- Is SUBDIR+=py-onnx@py27 missing in misc/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for misc/py-onnx@py27 processing package py27-onnx-tf-1.6.0

[00:02:02] Error: compute_deps_pkg failed to lookup pkgname for devel/py-flatbuffers@py27 processing package py27-tflite-2.3.0 from misc/py-tflite@py27 -- Is SUBDIR+=py-flatbuffers@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for devel/py-flatbuffers@py27 processing package py27-tflite-2.3.0

[00:02:02] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package py27-libtorrent-rasterbar-1.2.10 from net-p2p/py-libtorrent-rasterbar@py27 -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:02] Error: compute_deps_pkg failed to lookup pkgname for science/py-netCDF4@py27 processing package py27-netcdf-flattener-1.2.0 from science/py-netcdf-flattener@py27 -- Is SUBDIR+=py-netCDF4@py27 missing in science/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for science/py-netCDF4@py27 processing package py27-netcdf-flattener-1.2.0

[00:02:02] Error: compute_deps_pkg failed to lookup pkgname for devel/boost-python-libs@py27 processing package py27-dlib-19.21 from science/py-dlib@py27 -- Is SUBDIR+=boost-python-libs@py27 missing in devel/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:02] Error: compute_deps_pkg failed to lookup pkgname for textproc/py-elasticsearch6@py27 processing package py27-elasticsearch-dsl-6.4.0_1 from textproc/py-elasticsearch-dsl@py27 -- Is SUBDIR+=py-elasticsearch6@py27 missing in textproc/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for textproc/py-elasticsearch6@py27 processing package py27-elasticsearch-dsl-6.4.0_1

[00:02:02] Error: compute_deps_pkg failed to lookup pkgname for textproc/py-elasticsearch6@py27 processing package py27-elasticsearch-curator-5.6.0_1 from textproc/py-elasticsearch-curator@py27 -- Is SUBDIR+=py-elasticsearch6@py27 missing in textproc/Makefile and does the port provide the 'py27' FLAVOR?

[00:02:02] Error: compute_deps_pkg failed to lookup existing pkgname for textproc/py-elasticsearch6@py27 processing package py27-elasticsearch-curator-5.6.0_1
Comment 6 Loïc Bartoletti freebsd_committer freebsd_triage 2021-01-04 06:21:39 UTC
Created attachment 221265 [details]
numpy 1.9 and dependent ports v2

Here is the updated version which includes the latest python commits.
Comment 8 Rainer Hurling freebsd_committer freebsd_triage 2021-01-06 08:20:35 UTC
(In reply to Antoine Brodin from comment #7)
Regarding the log file for py-pandas: There is a bug #251412 with an update for py-pandas from 0.24.2 to 1.15.0, ready to commit ;)

Maybe the error with the undeclared identifier 'NUMPY_IMPORT_ARRAY_RETVAL' does not occur with this version of py-pandas (untested)?
Comment 9 Loïc Bartoletti freebsd_committer freebsd_triage 2021-01-09 07:32:08 UTC
(In reply to Rainer Hurling from comment #8)
I tried your patch of py-pandas and it built fine with numpy 1.9, please commit it :)
Comment 10 Loïc Bartoletti freebsd_committer freebsd_triage 2021-03-19 05:28:44 UTC
Created attachment 223412 [details]
py-numpy 1.19.4 v4

- update numpy to 1.19.4
- While here replace `${PYTHON_PKGNAMEPREFIX}numpy.*:math/py-numpy@${PY_FLAVOR}` by `${PYNUMPY}`
- replace `USES=python:3.[4-5]` by `USES=python:3.6`
- bump portrevision
- replace range version in `Mk/Uses/python.mk` (workaround for https://github.com/freebsd/pkg/pull/1944)
Comment 11 Antoine Brodin freebsd_committer freebsd_triage 2021-03-21 08:02:39 UTC
Something is wrong,  all ports depending on numpy are failing

Installing py37-numpy-1.19.4...
the most recent version of py37-numpy-1.19.4 is already installed
===>   py37-metpy-1.0 depends on package: py37-numpy>=1.16.0,1 - not found
Comment 12 Loïc Bartoletti freebsd_committer freebsd_triage 2021-03-23 10:59:06 UTC
Created attachment 223519 [details]
py-numpy 1.19.4 v5

antoine@ koobs@ The problem seems related to https://github.com/freebsd/pkg/pull/1944

I fix it removing portepoch (,1) in the version of numpy requirements.

Tested against matplotlib on updated poudriere and cleaned packages.
Comment 13 Antoine Brodin freebsd_committer freebsd_triage 2021-03-24 21:48:47 UTC
You can remove PORTEPOCH=1 from math/py-numpy/Makefile ,  you have to add it back
Comment 14 vincent.jancso 2021-04-13 14:38:13 UTC
Please consider updating to a newer release. openblas-0.3.13 does have a much larger memory footprint which causes numpy to segfault on import (virtual memory is limited per user in our environment)
A more recent release of numpy fixes this by setting the BUFFERSIZE back to 20.

Also see:
https://github.com/numpy/numpy/pull/17924
https://github.com/xianyi/OpenBLAS/issues/3180
https://github.com/xianyi/OpenBLAS/issues/2970
Comment 15 Val Packett 2021-04-26 11:16:09 UTC
Created attachment 224438 [details]
numpy1202.patch

Here's my version of the patch @ 1.20.2
Comment 16 Antoine Brodin freebsd_committer freebsd_triage 2021-04-26 12:45:00 UTC
This fails to fetch:

=> Attempting to fetch https://numpy.org/doc/1.20/numpy-ref.pdf
fetch: https://numpy.org/doc/1.20/numpy-ref.pdf: size mismatch: expected 7127190, actual 7362241
Comment 17 Antoine Brodin freebsd_committer freebsd_triage 2021-04-26 12:46:43 UTC
Also,  the doc should be put in a versioned directory or with a version in the filename
Comment 18 Thierry Thomas freebsd_committer freebsd_triage 2021-05-25 18:48:31 UTC
Created attachment 225259 [details]
Upgrade NUMPY to 1.20.3
Comment 19 Thierry Thomas freebsd_committer freebsd_triage 2021-05-25 20:54:45 UTC
Comment on attachment 225259 [details]
Upgrade NUMPY to 1.20.3

Sorry, this patch is not viable. Will try to provide another one.
Comment 20 Thierry Thomas freebsd_committer freebsd_triage 2021-05-26 11:27:04 UTC
Created attachment 225274 [details]
Upgrade NUMPY to 1.20.3

Adding a new patch:

- autoplist does not work in this case, a plist has been added;

- some tests fail, but this is not FreeBSD-specific, see
https://github.com/numpy/numpy/issues/18471#issuecomment-848684221
Comment 21 Antoine Brodin freebsd_committer freebsd_triage 2021-05-27 10:43:07 UTC
Something is missing,  all ports fail with:

===>   py38-foo depends on package: py38-numpy>=1.16,1<1.20,1 - not found
Comment 22 Thierry Thomas freebsd_committer freebsd_triage 2021-05-27 11:37:00 UTC
(In reply to Antoine Brodin from comment #21)

Indeed, Mk/Uses/python.mk must be adapted:
PYNUMPY=   ${PYTHON_PKGNAMEPREFIX}numpy>=1.16,1<1.20,1:math/py-numpy@${PY_FLAVOR}

@python team, what about this patch?

diff --git a/Mk/Uses/python.mk b/Mk/Uses/python.mk
index fcaf479d8076..5e2682d8efd6 100644
--- a/Mk/Uses/python.mk
+++ b/Mk/Uses/python.mk
@@ -640,7 +640,7 @@ CMAKE_ARGS+=        -DPython_ADDITIONAL_VERSIONS=${PYTHON_VER}
 
 # Python 3rd-party modules
 PYGAME=                ${PYTHON_PKGNAMEPREFIX}game>0:devel/py-game@${PY_FLAVOR}
-PYNUMPY=       ${PYTHON_PKGNAMEPREFIX}numpy>=1.16,1<1.20,1:math/py-numpy@${PY_FLAVOR}
+PYNUMPY=       ${PYTHON_PKGNAMEPREFIX}numpy>=1.16,1<1.21,1:math/py-numpy@${PY_FLAVOR}
 
 # Common Python modules that can be needed but only for some versions of Python.
 .if ${PYTHON_REL} < 30500
Comment 23 Kubilay Kocak freebsd_committer freebsd_triage 2021-05-31 02:20:34 UTC
Shared macro's (depends lines) in python.mk are fundamentally incompatible with the way Python dependency declarations (version-specs) work (packages can depend individually and arbitrarily on any version of a dependency they way), and will be removed going forward.
Comment 24 Kubilay Kocak freebsd_committer freebsd_triage 2021-05-31 02:22:26 UTC
Thank you for the update Greg. This can land and be merged with QA confirmation.
Comment 25 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-01 09:31:48 UTC
(In reply to Thierry Thomas from comment #20)

What happens/errors with autoplist?
Comment 26 Antoine Brodin freebsd_committer freebsd_triage 2021-06-01 09:45:13 UTC
Exp-run is still ongoing with the additional patch Thierry provided.
Comment 27 Thierry Thomas freebsd_committer freebsd_triage 2021-06-01 09:47:32 UTC
(In reply to Kubilay Kocak from comment #25)
It produces an uncomplete plist.
Comment 28 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-01 09:51:40 UTC
^Triage: portmgr doesn't need issue assignment to run an exp-run when the flag is requested

(In reply to Thierry Thomas from comment #27)

incomplete autoplists are almost always indicative of bugs. Can you attach a file containing the list of whats missing? Also note that one can use autoplist in tandem with pkg-plist (for those missing files, if we cant identify the issue)

With the manual plist, there also appears to be some potentially problematic entries, specifically those that need to be installed with python-version suffixed filenames:

+bin/f2py
+bin/f2py3
Comment 29 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-01 09:54:01 UTC
(In reply to Kubilay Kocak from comment #28)

Apologies, forgot to explicit mention that the problematic aspect is with the use of 'concurrent'. The only package version that should install a 'suffix-less'  filename is the 'default (python) version'. 

I'm not sure how the semantics of this change with 'allflavors' usage, which also begs the question, what are we using allflavors for?
Comment 31 Thierry Thomas freebsd_committer freebsd_triage 2021-06-01 16:24:36 UTC
(In reply to Kubilay Kocak from comment #28)

Adding autoplist to USE_PYTHON and removing pkg-plist produces this error:

$ make check-plist
====> Checking for pkg-plist issues (check-plist)
===> Parsing plist
===> Checking for items in STAGEDIR missing from pkg-plist
===> Checking for items in pkg-plist which are not in STAGEDIR
Error: Missing: %%PYTHON_SITELIBDIR%%/numpy/random/_examples/
===> Error: Plist issues found.
*** Error code 1

Stop.
make: stopped in /usr/ports/math/py-numpy

Does it work for you?
Comment 32 Thierry Thomas freebsd_committer freebsd_triage 2021-06-01 16:45:58 UTC
(In reply to Antoine Brodin from comment #30)

1) science/py-geometer
This port does not use PYNUMPY but a custom line restricted to numpy>=1.15,1<1.20,1

I guess that this restriction is somewhat arbitrary…
See the attached patch science_py-geometer.diff

2) devel/py-qutip
The restriction is not in the Makefile of the port, but in the setup.cfg file.
The attached patch devel_py-qutip.diff fixed the problem for me.
Comment 33 Thierry Thomas freebsd_committer freebsd_triage 2021-06-01 16:47:19 UTC
Created attachment 225466 [details]
Fix science/py-geometer with a recent Numpy

Fix science/py-geometer with a recent Numpy
Comment 34 Thierry Thomas freebsd_committer freebsd_triage 2021-06-01 16:48:19 UTC
Created attachment 225467 [details]
Fix devel/py-qutip with a recent Numpy

Fix devel/py-qutip with a recent Numpy.
Comment 35 Thierry Thomas freebsd_committer freebsd_triage 2021-06-01 16:56:42 UTC
(In reply to Antoine Brodin from comment #30)
py37-numpy-1.20.3,1 builds OK, but the staging fails: all optimized .pyc are missing.
Maybe someone with more Python clue could explain that and fix it?
Comment 36 Kubilay Kocak freebsd_committer freebsd_triage 2021-06-01 23:49:29 UTC
(In reply to Thierry Thomas from comment #31)

Interesting, thanks for that. This appears to be a common bug in upstreams where a data_dir is added to the --record output. The best way to address this is to remove the entry (modify the .pymodtmp pkg-plist before its written), or fix the offending setup.py numpy/distutils issue
Comment 37 Thierry Thomas freebsd_committer freebsd_triage 2021-06-02 08:10:21 UTC
(In reply to Kubilay Kocak from comment #36)

Kubilay,

As much as I can admit the usage of autoplist for lazy porting of a minor port which installs 5 or 6 files, for a port as important as Numpy, I don't understand it , and IMHO it would be a mistake, especially if you have to patch it in order to fix autoplist!

- a static plist allows you to simply answer the question "which port installs this particular file?" with a grep even if not installed;

- during an upgrade, this is easy to see if some important files are missing due to an unoticed error.

For example, the case reported by Antoine in Comment 35 (optimized files missing with Python-3.7) would not even have been noticed with an autoplist!
Comment 38 Antoine Brodin freebsd_committer freebsd_triage 2021-06-02 08:59:16 UTC
(In reply to Thierry Thomas from comment #35)
I think that the plist issue is due to %%PYTHON_EXT_SUFFIX%% being used instead of cpython-%%PYTHON_SUFFIX%% in pyc lines

( %%PYTHON_EXT_SUFFIX%% should only be used in .so lines )
Comment 39 Thierry Thomas freebsd_committer freebsd_triage 2021-06-02 20:14:53 UTC
Created attachment 225511 [details]
math/py-numpy: upgrade to 1.20.3

Thanks Antoine! This new patch adopted your idea.

Note: the previous plist had been generated from `make makeplist', so makeplist must be fixed too!
Comment 40 Kubilay Kocak freebsd_committer freebsd_triage 2021-07-21 08:08:48 UTC
(In reply to Thierry Thomas from comment #39)


Can you clarify where this is at wrt QA?

The exp-run has not been ack'd, the latest patch (and we have three), does it pass QA? Does it need another exp-run? 

With no open questions, python has no issue accepting an explicitly QA'd changeset.

Other open questions:

With the other port fixes:

+-	numpy>=1.16.6,<1.20
++	numpy>=1.16.6,<1.21

Does the software pass tests with 1.21 ? Are there refs upstream that <1.20 is unnecessarily limiting, or has it been bumped upstream?


-RUN_DEPENDS=	${PYTHON_PKGNAMEPREFIX}numpy>=1.15,1<1.20,1:math/py-numpy@${PY_FLAVOR} \
+RUN_DEPENDS=	${PYNUMPY}	\

Shared macros don't work with Python package dependency declarations, and we're removing instances of these.

autoplist is preferred. I agree there are some cases where autoplists can cause lack of observability, but there are plenty of cases where autoplists assist immensely.

We should have a discussion about improving the situation (possibly to have our cake and eat it to) offline. In the meantime autoplist is preferred, especially for complex ports, and particularly for C extension based python ports. It picks up enough upstream packaging issues that cause other issues downstream for us that its beneficial overall.
Comment 41 Thierry Thomas freebsd_committer freebsd_triage 2021-07-21 08:56:33 UTC
(In reply to Kubilay Kocak from comment #40)

There is many points in your comment.

For what I'm concerned with:

- the proposed patch (the latest, of course) is OK for the ports I tested, but of course I have not tested all of them, and an exp-run is necessary (you asked Antoine to stop it)

- for what I'm seeing, most ports are limiting to a version unnecessarily, just because they have not been tested with a higher version. If we remove shared macros for ports like numpy, it means that we must support several versions of these ports, like py-numpy-legacy and py-numpy, without conflicts.

- I do not see any value in auto-plist, excepted lazy-porting, but if you want it, just do it, you are the maintainer. I have not been able to make it work with autoplist, this is why I have generated a static one.
Comment 42 Kubilay Kocak freebsd_committer freebsd_triage 2021-07-21 09:21:21 UTC
(In reply to Thierry Thomas from comment #41)

It is an issue, but it is the price we pay for wanting to package Python programs

We can address the issue in several ways, none resolving it completely. This is a conversation for the Python team that we'd love interested people to participate in.

Shared macros though, don't fix the problem, they make the problem worse. Ports build because the 'version spec' is apparently 'satisfied', but then are broken at rutime, because they're actually not. There is no such thing as a shared version spec in the Python world. Every upstream package declares their dependencies individually and arbitrarily. We can pretend they're 'shared' to make builds work, but this doesn't mean 'usable' software for ports and package users.

I'll respond to the exp-run issue offline
Comment 44 Po-Chuan Hsieh freebsd_committer freebsd_triage 2021-07-24 17:53:45 UTC
(In reply to Thierry Thomas from comment #33)

PYGAME and PYNUMPY should be removed some day. Unlike other PY_foo, they do not bring any benefit but make things more complex. For example, it's getting much harder to check version requirements of dependent ports.

Please do not revert the change for removing PYNUMPY. The correct way is to patch the source (setup.py, setup.cfg, requestments.txt or other files) and update RUN_DEPENDS. In this case, it's setup.py.

(In reply to Thierry Thomas from comment #39)

Note that numpy 1.20.* requires Python 3.7.
For a complete patch, you forgot to change USES=python:3.6+ to USES=python:3.7+ and update all dependent ports as well.
Comment 45 Wen Heping freebsd_committer freebsd_triage 2021-08-09 03:03:20 UTC
Created attachment 227029 [details]
Update to 1.20.3 and add missing TEST_DEPENDS

The test of numpy-1.20.3 require devel/py-hypothesis, I upload the revised patch
with this change. There are 7 fails in all 12000+ tests on my amd64-head system.

Shall we ask for another exp-run for numpy ?
Comment 46 Kubilay Kocak freebsd_committer freebsd_triage 2021-08-09 03:35:00 UTC
(In reply to Wen Heping from comment #45)

Thanks for adding test support Wen. Very surprised but relieved at only 7 failing tests. exp-run call is up to the person intending to commit and address possible fallout.

Still -1 on not using autoplist, we still have frequent QA failures due to this and a lack of comprehensive testing.

@Sunpoet/Wen Let's get a separate tracking issue up/created for broken shared macro removal. I also have a few other framework level tracking issues to create
Comment 47 Thierry Thomas freebsd_committer freebsd_triage 2021-10-26 16:17:18 UTC
Committed by wen@ in 7e9bec828e31369d9b6bab30ac86218bf3def70d .
Comment 48 Kubilay Kocak freebsd_committer freebsd_triage 2021-10-26 23:56:45 UTC
^Triage Assign to committer that resolves and correct resolution (FIXED, by way of a change; commit)