Created attachment 253530 [details] Update numpy to 2.1.1 and scipy to 1.14.1 numpy2 released for a long time, there are 2 choice for us to update numpy: 1) create a new port: math/py-numpy2 2) update math/py-numpy to 2.x and create math/py-numpy1 if necessary I tried both ways now I would like select 2, althought it would break many ports. We can fix it, actually many ports hold the update waiting for numpy2. When update numpy, scipy should be updated together, and pandas, and pythran. With this patches, numpy-2.1.1 and scipy-1.14.1 build well and run well, numpy-2.1.1 pass all tests, scipy-1.14.1 has only 6 fails in all tests. Any suggestions welcomed !
(In reply to wen from comment #0) I would not even break many ports, but also people's stuff having 1.x unter that port/package name.
At first I'm inclined to agree with option 1 due to the types of consumers (ahem, academic-type software) involved, but a quick look of numpy's development trajectory seems to suggest that only 2.x is to have serious development support going forward so option 2 seems more reasonable. Changing consumers that rely on 1.x to a new port directory/name, given that both the referenced software remains the same and can co-exist, does not constitute a POLA trigger.
(In reply to Charlie Li from comment #2) I am not an academic users. We are heavy industry and use numpy serious computations. If something like this changes I need to trigger several engineers to review/validate/release a lot of Python software. They give the development pace, not the IT (me) department.
Reminder: a NumPy 2.0 migration guide is available at https://numpy.org/doc/stable/numpy_2_0_migration_guide.html
(In reply to Michael Osipov from comment #3) I understand where you're coming from, but at some point, your processes cannot become ports' problem in terms of medium-/long-term maintenance, especially when reflecting upstreams' current (stable) development and support focuses. There are other examples in ports where an upstream project released a new major breaking version, the port updated to such due to upstream's focus going forward, but the previous (unchanged) version copied to a different directory name with those consumers reflecting the change.
1. Will make things backward compatible on existing setups / scripts and will clearly distinguish new version. Other ports that require version 2 can just update dependencies to py-numpy2 right? 2. Would work like gnupg where security/gnupg is the current version 2 while security/gnupg1 is the old version 1. But it is a bit confusing. 3. There is also third option to use py-numpy1 for version 1, py-numpy2 for version 2, and py-numpy (metaport) for the most recent version. This seems both versatile and provides best compatibility no matter what version shows in future. It will break current setups though. This approach is used for instance with drm-kmod packages (drm-510-kmod for version 5.10, drm-515-kmod for version 5.15, drm-61-kmod for version 6.1, and just drm-kmod metaport for the latest one whatever it is and matches the system requirements). I like this approach most. Only option 1 does not break existing setups. For option 2 or 3 change is only '1` added to the package name to maintain compatibilty.
(In reply to Charlie Li from comment #5) The process in this case is quite simple: Create a new port, let it sift in, mark the old one as deprecated for a good amount of time, wait for reactions, remove.
(In reply to Tomasz "CeDeROM" CEDRO from comment #6) I would only go for option three if the ports tree requires it and dependent ports need it. Otherwise it cases too much work.
py-numpy1 seems to be in use and might be still in use in future for various reasons so it may be good to provide that package as well?
(In reply to Michael Osipov from comment #7) I think this is a reasonable choice. Add math/py-numpy2 for now allowing new users to start migrating. And mark a future cutover date where math/py-numpy becomes based on 2.x (maybe a month or two). At that time repo copy math/py-numpy to math/py-numpy1. And ports that are still dependent on only numpy1 can update their dependency to start using that until they support numpy2.
(In reply to John Hein from comment #10) Reasonable, but one or two is way too short. How will you notify people? There is only UPDATING, but that is for those who use the ports tree. You need to have at least three to six months.
If I have understood correctly, NumPy versions 1.x and 2.x have to coexist in the ports tree for a while. Has anyone tried whether they can be installed at the same time (side by side) without conflicts?
(In reply to Michael Osipov from comment #11) "... you need to have at least three to six months" Whether that is an appropriate guideline depends on lots of factors that will likely be different in different organizations (among FreeBSD ports tree maintainers and the set of all downstream users). One or two months gives some time, whereas comment 5 advocates zero. I don't know if six months would be acceptable to ports tree needs - it's hard to juggle all the inter-dependencies of all the python ports, and things are moving fast away from distutils, for instance. There's not a perfect answer, but I agree the FreeBSD ports tree updates should be balanced with consideration of ports tree juggling and for downstream out-of-ports-tree users. And numpy1 would still be available after the cutover, so those downstream ports tree users that did not muster the resources to migrate to numpy2 "in time" would be able to switch to the numpy1 package until their in-house uses could be updated. As far as out-of-band (out-of-ports-tree) notice goes, I would recommend using python@freebsd.org to communicate the plan that is settled on. A wiki entry to capture the plan, and guide users through the migration would be useful as well. I don't think a longer time window than a couple months would help much in many cases. Take for example, the py2->py3 update - not a perfect comparison, but lots of time passed to allow people to prepare for 2->3 update, and frequently it prompted procrastination. I still run into things (e.g., in-house scripts & python libraries and other third party python-based dependencies not from the FreeBSD ports tree) that need 2to3 treatment.
(In reply to Rainer Hurling from comment #12) "Has anyone tried whether they can be installed at the same time (side by side) without conflicts?" The current math/py-numpy and the one based on 2.x proposed here do conflict. Some options to support having two math/py-numpy* ports include the following: (a) Allow math/py-numpy1 (or math/py-numpy based on 1.x) and math/py-numpy2 (or math/py-numpy based on 2.x) to both exist as package options, but they conflict. So either one can be installed individually but both cannot be installed together. When you have multiple dependent ports clamoring for different versions (especially all needing run time installation), this becomes intractable to install in the same environment. But some packages use this strategy (like lang/cython and lang/cython3, for instance, which is usually only needed at build time). (b) Allow one of the flavors to install to an alternate namespace (e.g., site-packages/numpy1). (c) Install to a separate virtual environment. This option is not currently supported by the ports tree python infrastructure. So supporting this would be a big change.
(In reply to John Hein from comment #14) Hi John, Thank you for the detailed answer. I had feared such conflicts. Since both NumPy versions are needed at the same time, at least for a probably longer transition phase in more complex installations, e.g. for GIS software such as with QGIS, SAGA GIS etc., variant (b) seems the safest to me.
(In reply to John Hein from comment #14) (b) is technically not possible and technically incorrect. By renaming the subdirector(y/ies) of site-packages, you are actually messing up both the Python package metadata and the module import logic, such that *every* consumer has to be patched to reflect. Upstream made it clear that numpy 1 is no longer supported. (c) is somewhat out of scope for the ports tree itself, as jail(8) is our analogue.
(In reply to Rainer Hurling from comment #15) "... variant (b) seems the safest to me." I generally agree, although it may not be that simple. But variant (a) isn't bad just to get numpy2 into the tree for now - for people to start using / investigating migration of existing code that uses numpy (along with the nice migration guide mentioned in comment 4). This assumes we don't get a rush to update dependencies to point to numpy2 and quickly start hitting dependency hell.
(In reply to Charlie Li from comment #16) "Upstream made it clear that numpy 1 is no longer supported." Thanks, Charlie. There are degrees of "not supported", of course. Do you have a reference to the upstream discussion / notes on that topic? It looks like 2.0 was only tagged in June 2024, so for downstream consumers, that's not a particularly long time, especially if they weren't paying attention in June (or before).
(In reply to John Hein from comment #18) The 1.26.4 release notes state that "this is the last planned release in the 1.26 series." 1.26.5 was released in May only for a few bug fixes. Meanwhile upstream have been tracking consumers' compatibility with 2.x: https://github.com/numpy/numpy/issues/26191
More and more Python packages require numpy-2.x I've hit 3 such ports only today. The latest one is math/py-numpoly. Cython fails with errors like: > ./numpoly/cfunctions/cdispach.c:3108:13: error: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion] > 3108 | __pyx_t_1 = PyDataType_FIELDS(__pyx_v_self); > | ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Such ports can't be updated until math/py-numpy is updated to 2.x
Created attachment 255297 [details] Update numpy to 2.1.3 Update numpy to 2.1.3 the result of `make test`: 46842 passed, 1700 skipped, 2800 deselected, 33 xfailed, 6 xpassed, 21 warnings in 453.90s (0:07:33)
If nobody object, I shall import numpy2 as math/py-numpy2 in this week.
(In reply to wen from comment #22) This port should be updated to 2.x with 1.x moved to a new port, especially considering upstream's efforts to move consumers away from 1.x. We also should not be encouraging continued use of 1.x either, so the port name should reflect such. Note that some consumers currently in the tree already work with 2.x; they probably just need cataloguing.
I suggest another way to update numpy in our portstree: i) copy math/py-numpy to math/py-numpy1 ii) update math/py-numpy to 2.1.3 iii) modify Mk/Uses/python.mk, define PYNUMPY to depends on math/py-numpy1 and add a define of PYNUMPY2 I believe this way shall result in less breakage than other ways. Any suggestions ?
(In reply to wen from comment #24) First two points are exactly what I was thinking. Third point, I think we should gradually deprecate ${PYNUMPY} in favour of the directly specifying the dependency line, but for now keep ${PYNUMPY} to point to math/py-numpy1. Those ports that are already safe for numpy 2.x can already switch to the direct dependency line.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=0c610c4b35941436172bfede63e11a62454581b2 commit 0c610c4b35941436172bfede63e11a62454581b2 Author: Charlie Li <vishwin@FreeBSD.org> AuthorDate: 2024-11-21 20:33:25 +0000 Commit: Charlie Li <vishwin@FreeBSD.org> CommitDate: 2024-11-21 20:33:25 +0000 devel/boost-python-libs: mark as safe for numpy 2.x Upstream commit included in devel/boost-libs/files. Switch away from ${PYNUMPY} to both catalogue and allow seamless transition when math/py-numpy 2.x is committed. While here, remove USES=python upper bound. With hat: python PR: 281470 devel/boost-python-libs/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Created attachment 255373 [details] update math/py-numpy to 2.1.3 Important: Before apply this patch, please copy math/py-numpy to math/py-numpy1 and add py-numpy1 to math/Makefile.
(In reply to Charlie Li from comment #25) Thank you! Now I submitted a new patch and ask for a exp-run.
The patch is wrong or incomplete, please provide a complete patch and verify that math/py-numpy and math/py-numpy1 have a different PKGBASE
Created attachment 255399 [details] numpy-2.1.3p1.diff.txt I regenerate the patch into one file, it include: 1) new port math/py-numpy1 2) add py-numpy1 to math/Makefile 3) modify Mk/Uses/python, make PYNUMPY depends on math/py-numpy1 4) Update math/py-numpy to 2.1.3 5) fix math/py-pandas, math/py-numexpr, math/py-bottleneck
(In reply to Antoine Brodin from comment #29) I regenerate the patch again, now update math/py-numpy to 2.1.3 and set the PKGNAME to py3.11-numpy2.
The patch is still wrong: science/py-HofstadterTools dependency on math/py-numpy@py311 has wrong PKGNAME of 'py311-numpy' but should be 'py311-numpy2' science/py-access dependency on math/py-numpy@py311 has wrong PKGNAME of 'py311-numpy' but should be 'py311-numpy2' science/py-cfgrib dependency on math/py-numpy@py311 has wrong PKGNAME of 'py311-numpy' but should be 'py311-numpy2' etc etc.
(In reply to Wen Heping from comment #31) We're not ready for an exp-run yet. PKGNAME on 2.x should remain the same without the 2 suffix, the only suffixed port should be of 1.x. Phab to follow. Still need to mark which ports are already compatible with 2.x by changing *_DEPENDS. Some ports can still build and run under 1.x despite what pyproject.toml (but not their respective documentation) says; we would just temporarily run ${REINPLACE_CMD} until 2.x is committed, at which point they would need a rebuild for the ABI change anyway.
After multiple attempts, I understand that whether setting PKGBASE to 1 or 2, it requires manually modifying hundreds of ports. Given our current situation, I don't know when we will be able to complete this, next year or next next year. (I shall mention PR/270510, where 20 months have passed and we still haven't been able to commit it. Charlie, I really appreciate and am grateful for your nice work, but I am an impatient person, sorry.) I believe the simplest and fastest way to use numpy2 is to add py-numpy2 port and set PKGBASE to 2, and then define PYNUMPY2 in python.mk. This approach will not break other ports.
I would close this PR and switch to: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279814
No. This is the correct approach. There are enough consumers that are compatible with both 1.x and 2.x, and again, 1.x should not be encouraged. I know there is some impatience but this has to be done correctly the first time or not at all.
(In reply to Charlie Li from comment #36) In your correct way, how long do you think it will take for us to use numpy2 ? in 2024? or 2025? or 2026? or after numpy3 release ? or never ? I shall do my best to help if anyone submit the correct patch in correct way.
(In reply to wen from comment #37) You need to put also company-internal code into consideration, it is not just about the ports which depend on numpy 1.x.