Bug 247542
| Summary: | math/lapack: upgrade to 3.9.1 with math/blas + math/xlapack + math/cblas + math/lapacke | ||
|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Thierry Thomas <thierry> |
| Component: | Individual Port(s) | Assignee: | Thierry Thomas <thierry> |
| Status: | Closed FIXED | ||
| Severity: | Affects Many People | CC: | diizzy, dimpase+freebsd, fortran, j.david.lists, jcfyecrayz, phd_kimberlite, rhurlin, thierry |
| Priority: | --- | Flags: | antoine:
exp-run+
|
| Version: | Latest | ||
| Hardware: | Any | ||
| OS: | Any | ||
| URL: | http://www.netlib.org/lapack/lapack-3.9.1.html | ||
| See Also: | https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244296 | ||
| Bug Depends on: | 247939 | ||
| Bug Blocks: | 247485 | ||
| Attachments: | |||
|
Description
Thierry Thomas
2020-06-25 16:43:07 UTC
portmgr, please schedule an experimental build. Created attachment 215975 [details]
Remove the CONFLICTS line in math/openblas
Add a forgotten patch: remove the CONFLICTS line in math/openblas.
New failures logs on 12.1 amd64: http://package18.nyi.freebsd.org/data/121amd64-default-PR244494/2020-07-06_14h21m06s/logs/treekin-0.5.1_2.log http://package18.nyi.freebsd.org/data/121amd64-default-PR244494/2020-07-06_14h21m06s/logs/maxima-5.44.0_1.log http://package18.nyi.freebsd.org/data/121amd64-default-PR244494/2020-07-06_14h21m06s/logs/scilab-6.1.0_2.log http://package18.nyi.freebsd.org/data/121amd64-default-PR244494/2020-07-06_14h21m06s/logs/py37-phono3py-1.13.3.27_8.log http://package18.nyi.freebsd.org/data/121amd64-default-PR244494/2020-07-06_14h21m06s/logs/py27-phono3py-1.13.3.27_8.log The problem with math/maxima is not related to this upgrade, and it has been fixed by zeising@ in r541427. Created attachment 216409 [details]
2nd version of the patch
New version of the patch:
- science/scilab
unfortunately Scilab still uses deprecated functions
=> buld lapack with deprecated funtions to fix it.
- science/py-phono3py
patch lapacke.h -> lapacke/lapacke.h
The remaining problem is biology/treekin.
(In reply to Thierry Thomas from comment #5) Did you perhaps intend to delete the '.include <bsd.port.options.mk>' line? The latest patch adds a second one. =============== 'make patch' in math/lapack is failing. I think you just forgot some files/patch* deletions that were in the first patch. Some additions were omitted in the new patch, too (e.g., static.mk). =================== The manpages.tgz file now no longer has a version tag. If one keeps old distfiles around (I do), that can cause a conflict on an update. Maybe consider DIST_SUBDIR=lapack/${PORTVERSION} (qemu does that, for example, and linux-flashplayer)? ================== In static.mk (in the original patch), it references SRC in two places. The post-build replaces 'cd SRC' with 'cd xxx/SRC', but does not replace SRC in the RANLIB line. So ranlib is failing as follows when, for instance, building math/blas: ranlib: 'SRC/libblas.a': No such file Maybe just remove 'cd ' in the sed lines in post-build so the sed replaces both instances of SRC? I don't understand why the earlier exp-run was not failing here. ================= Created attachment 216611 [details]
Fix the preious version of the patch
It seems that the previous version of the patch was incorrect: regenerate it.
Thanks to John!
Created attachment 216640 [details]
Complete patch, including a fix for Treekin
This patch fixes the problem encountered with biology/treekin (See PR 247939).
With this one, all the problems shown in the first exp-run are fixed: please schedule another one.
(In reply to Thierry Thomas from comment #8) ====== Carriage return line endings in attachment 216640 [details]. Testers should dos2unix it first. ====== I still get 'ranlib' errors (see comment 6). But I see why exp-run did not notice it last time. When ranlib fails, it still exits with 0, so make does not notice the failure (could be considered a ranlib bug I suppose). In any case, the fix is still the same I think - get rid of 'cd ' in the sed substitutions in post-build. For example, s|SRC|BLAS/SRC| (In reply to John Hein from comment #9) Hello John, Could you please tell me the line number containing a ^M ? I'm sorry, but I cannot find it... I cannot either reproduce the problem with ranlib, neither on poudriere nor on my workstation. Could you please send me a full log, with your platform / version? Thanks. (In reply to Thierry Thomas from comment #10) Every line in attachment 216640 [details] has a '\r' (viewable with hd or vis -tl or sed -n l or basic vi (nvi); vim will "hide" them for you, but you will see "[dos]" at the bottom). I downloaded the attachment with fetch(1) (wget gives the same result). I just tested 'save link as' from firefox - that preserves the '\r' endings that are in the file as well. Sure, I can attach a build log of math/blas showing the ranlib error. But if you inspect work/.build/BLAS/SRC/CMakeFiles/blas.dir/build.make - specifically the build-static target at the bottom - you will see the problem: build-static: cd BLAS/SRC && $(AR) $(ARFLAGS) libblas.a $(blas_OBJECTS) $(RANLIB) SRC/libblas.a The cd && ar works fine, but ranlib is using the wrong path to the lib file. This is because post-build in math/lapack/Makefile does a substitution for the path in math/lapack/files/static.mk that is specific to the slave port (which then is appended to the build.make file). But it only looks for 'cd SRC' to replace. This misses the ranlib line which has no 'cd ' before SRC, so sed does no substitution on that line. (In reply to John Hein from comment #11) I'm really sorry, but I cannot reproduce the problem with the patch format. I could not even imagine how I could produce such a thing: I do not use Windows machines, my patches are produced directly on a FreeBSD workstation by `svn diff', and they are uploaded to bugzilla from the same machine, using Firefox. About ranlib, I have understood! These ports are quiet complex, and I tried to simplify them, but re-using the same parts between the different ports. And you are right, ranlib emit a warning for math/blas, but this is just a warning, and the static library is built anyways during post-build. Does'nt it produce a $PREFIX/lib/libblas.a when you make it? (In reply to Thierry Thomas from comment #12) re: CR line endings... I don't know either why I get the patch with CR line endings and it seems that you don't (I get this on multiple linux, freebsd platforms - I have not tried windows yet). What md5 do you have for the downloaded attachment? Here's what I did: # make sure I don't have an old copy of the attachment file in my current directory, then... % wget 'https://bugs.freebsd.org/bugzilla/attachment.cgi?id=216640' --2020-07-28 13:01:26-- https://bugs.freebsd.org/bugzilla/attachment.cgi?id=216640 Resolving bugs.freebsd.org (bugs.freebsd.org)... 96.47.72.84, 2610:1c1:1:606c::50:15 Connecting to bugs.freebsd.org (bugs.freebsd.org)|96.47.72.84|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://bz-attachments.freebsd.org/attachment.cgi?id=216640 [following] --2020-07-28 13:01:27-- https://bz-attachments.freebsd.org/attachment.cgi?id=216640 Resolving bz-attachments.freebsd.org (bz-attachments.freebsd.org)... 96.47.72.84, 2610:1c1:1:606c::50:15 Connecting to bz-attachments.freebsd.org (bz-attachments.freebsd.org)|96.47.72.84|:443... connected. HTTP request sent, awaiting response... md200 OK Cookie coming from bz-attachments.freebsd.org attempted to set domain to bugs.freebsd.org Length: 41864 (41K) [text/plain] Saving to: ‘attachment.cgi?id=216640’ attachment.cgi?id=216640 100%[=================================================>] 40.88K --.-KB/s in 0.05s 2020-07-28 13:01:28 (840 KB/s) - ‘attachment.cgi?id=216640’ saved [41864/41864] % sed -n l sed -n l attachment.cgi\?id=216640 | head Index: math/blas/Makefile\r$ ===================================================================\r$ --- math/blas/Makefile\t(revision 542588)\r$ +++ math/blas/Makefile\t(working copy)\r$ @@ -2,7 +2,6 @@\r$ # $FreeBSD$\r$ \r$ PORTNAME=\tblas\r$ -PORTREVISION=\t6\r$ \r$ % md5 attachment-216640 MD5 (attachment.cgi?id=216640) = 5d7018e016449aac293b4bb01ed31219 What md5 do you have? ================================================================ Re: ranlib issue... You asked if I get a libblas.a. I do. But that's because ranlib(1) does not produce the file, ar(1) (on the line above the ranlib command) does. ranlib(1) adds/updates a symbol table which is important for linking. Some versions of ar(1) will do this by default (GNU binutils does, for instance). Some versions of ar(1) do not (libarchive's ar in base). Running ranlib(1) assures that a symbol table is added so static linking with the .a archive will "work". Anyway, again... just remove 'cd ' from your sed(1) expressions in post-build and things will work. Namely, you will provide the right path to the .a file for ranlib(1). And a few less characters in post-build - bonus! You seem hesitant to make that change, however. Do you have concerns? There is a problem: ===> Patching for blas-3.9.0 ===> Applying FreeBSD patches for blas-3.9.0 from /usr/ports/math/blas/../lapack/files 2 out of 2 hunks failed--saving rejects to LAPACKE/include/lapack.h.rej ===> FAILED Applying FreeBSD patch-LAPACKE_include_lapack.h ===> FAILED to apply cleanly FreeBSD patch(es) patch-LAPACKE_include_lapack.h *** Error code 1 (In reply to Antoine Brodin from comment #14) Antoine, the patch has CR line endings. Does it work if you dos2unix attachment 216640 [details]? Created attachment 216966 [details]
math/lapack + blas + cblas + xblas + some consumers diff
John: you did not provide a patch, and I'm not sure to understand what you meant, but anyway I tried to fix the problem with ranlib.
Antoine: I do not understand anything about this patch formats, dos2unix and so on, but I have regenerated my patch, and tested it against a fresh ports tree, and it seems OK for me, and the resulting ports build in poudriere.
SHA256 (math_lapack.diff) = 07533618cb74a48d8aeac9ca1fbb473660fc9fc1a45aac990c2ea29d3f68bdf5
Created attachment 216986 [details] [patch] update lapack & slave ports to 3.9.0 [v7] (In reply to Thierry Thomas from comment #16) Thierry, Whatever you are doing during upload is adding CR line endings - attachment 216966 [details] has them as well. You can check yourself by downloading the attachment. If Antoinne runs an exp-run, he's going to hit the same problem again. Here's the sha256 for the downloaded attachment. SHA256 (attachment-216966) = 0c34dd36e60690217213091a55b2d9df2ce17657a02289edac8f2aa89102e6cf After dos2unix: SHA256 (attachment-216966.nocr) = 07533618cb74a48d8aeac9ca1fbb473660fc9fc1a45aac990c2ea29d3f68bdf5 If you don't have dos2unix (converters/unix2dos), you can use tr: tr -d '\r' < attachment-216966 > attachment-216966.nocr The first 4 (obsolete now) versions of you patch did not have CR line endings. Consider what you may have done differently. How do you upload the attachments? ================= Re: the ranlib issue. I didn't add a patch because I thought my suggestion was worded clearly enough twice that you could make the change yourself (that is, to remove the 'cd ' from your sed expressions). But here now is a new attachment that does not have CR line endings. It also has the fix for ranlib that I was suggesting. Note that in your patch, you forgot to fix the case for LAPACKE_SLAVEPORT. This attachment fixes that as well. This was build tested in poudriere. If you are okay with the new patch attached, mark your other patch obsolete and ask for the exp-run with the new one. Since I'm not even able to submit a patch, let's forget about that. I'm watching this in disbelief. Looks like about time to switch to git... (even github) Is it possible to revisit this? The ports tree has moved to git and I still can't build the combination of numeric packages my users need without applying this patch (which I note I have never had a problem applying, even under the old svn trees). (In reply to j.david.lists from comment #20) Sure. Thierry had some sort of mechanical troubles uploading a patch for unknown causes (unknown at least to me - Thierry did not respond with answers to the troubleshooting questions), and it seems he just gave up. Feel free to open a new bug with my patch as a starting point. At the time, I build tested it and did some light run testing. For day to day usage, I mostly use scipy which depends on lapack & blas. I have been using 3.9.0 since last summer. Since last year, lapack 3.9.1 is out, so I would probably bump to 3.9.1 instead of 3.9.0. new distinfo: +TIMESTAMP = 1622566308 +SHA256 (lapack/v3.9.1.tar.gz) = d0085d2caf997ff39299c05d4bacb6f3d27001d25a4cc613d48c1f352b73e7e0 +SIZE (lapack/v3.9.1.tar.gz) = 7543209 +SHA256 (lapack/manpages.tgz) = 65181ed22a87cde3cfee3bc9e4135434e3ee9f5fc5651a37c0ca4e4b737254de +SIZE (lapack/manpages.tgz) = 1378304 (In reply to John Hein from comment #21) Besides this PR, I submitted many PR, and nobody else had problems with my patches, you are the only one. (In reply to Thierry Thomas from comment #22) It wasn't just me. Antoine had problems - see comment 14. It may be a local problem on my end, but it doesn't seem so to me. I tried a number of different tools to grab the patch (fetch, curl, web browser, linux, freebsd). Every time I downloaded the patches in question, they had CR line endings. Did you ever try the steps in comment 13? It seems to work just fine now. Perhaps it was a server-side probem?
% curl -L "https://bz-attachments.freebsd.org/attachment.cgi? id=216640&action=diff&format=raw&headers=1" >x
% file x
x: unified diff output text, ASCII text
Perhaps we could table the issue of whether there's some issue in the bug tracker that causes problems for some people when they download a patch through it. That's not really an issue with the patch. Thierry, the patch appears to work. You have commit privileges in the ports tree. Does your proposed patch pass FreeBSD's ports CI requirements (whatever those may be) when you apply it? If so, what are the next steps required for you to commit the patch? I am not familiar with the process (neither technology nor policy). The existing issue seems to be adequately complicated already. I would suggest updating to 3.9.0 with the existing patch, then doing an update to 3.9.1 in the future when the dust settles. That should be a straightforward update on its own, and incremental progress would be better than no progress. (In reply to j.david.lists from comment #25) Anyway, be it 3.9.0 or 3.9.1, meanwhile many ports have been updated, and everything must be tested again. Let me some days to check it again, and when everything will appear OK on my side, I'll ask Antoine to relaunch an exp-run. Created attachment 225662 [details] Upgrade lapack & Co to 3.9.1 I decided to upgrade directly to 3.9.1: compared to 3.9.0, this is a minor release but it fixes some points (e.g. one patch can be removed). Release notes at <http://www.netlib.org/lapack/lapack-3.9.1.html>. Also: - Add a test target - set -fallow-argument-mismatch for math/xlapack - science/py-phono3py switched to openblas => no patch needed anymore. Antoine, can you please schedule an exp-run? A file seems to be missing: http://package18.nyi.freebsd.org/data/122amd64-default-foo/2021-06-09_15h39m55s/logs/errors/blas-3.9.1.log Created attachment 225664 [details] Upgrade lapack & Co to 3.9.1 (In reply to Antoine Brodin from comment #28) Strange… Sorry, I thought that `git commit -a' would take everything, but a previous `git add' seems required. Exp-run looks fine Committed - finally! A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=c2488a6020830af2cf09699b887adec7af806bf3 commit c2488a6020830af2cf09699b887adec7af806bf3 Author: Thierry Thomas <thierry@FreeBSD.org> AuthorDate: 2021-06-13 17:35:55 +0000 Commit: Thierry Thomas <thierry@FreeBSD.org> CommitDate: 2021-06-13 17:48:12 +0000 math/lapack: +math/blas et al., upgrade to 3.9.1 - Upgrade math/blas, math/cblas, math/lapack, math/lapacke and math/xlapack to 3.9.1; Latest release notes at <http://www.netlib.org/lapack/lapack-3.9.1.html> - Chase this upgrade in biology/treekin; - Add a test target; - Remove a conflict with math/openblas (PR 244296); - Fix the build with Gcc10 (PR 247485). PR: 247542 Approved by: expr-run by antoine@ biology/treekin/Makefile | 2 +- biology/treekin/files/patch-src_calcpp.h (new) | 12 + math/blas/Makefile | 1 - math/cblas/Makefile | 133 +----- math/cblas/distinfo (gone) | 4 - math/lapack/Makefile | 298 ++++++-------- math/lapack/distinfo | 9 +- math/lapack/files/manpages | 458 ++++++++++++++++++--- math/lapack/files/patch-Makefile (gone) | 11 - math/lapack/files/patch-SRC+Makefile (gone) | 98 ----- .../lapack/files/patch-TESTING+LIN+Makefile (gone) | 82 ---- math/lapack/files/patch-TESTING+Makefile (gone) | 14 - math/lapack/files/patch-lapacke+Makefile (gone) | 30 -- .../lapack/files/patch-lapacke+src+Makefile (gone) | 29 -- math/lapack/files/static.mk (new) | 3 + math/lapack/pkg-descr | 1 + math/lapack/pkg-plist | 92 +++-- math/openblas/Makefile | 2 - math/xlapack/Makefile | 1 - 19 files changed, 599 insertions(+), 681 deletions(-) Hi Thierry, Looking at ports-fallout this seems to have broken biology/treekin as _3 doesn't to build on any platform. https://portsfallout.com/port/1558/ Best regards, Daniel |