Bug 247542 - math/lapack: upgrade to 3.9.0 with math/blas + math/xlapack + math/cblas
Summary: math/lapack: upgrade to 3.9.0 with math/blas + math/xlapack + math/cblas
Status: Closed Overcome By Events
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Port Management Team
Depends on: 247939
  Show dependency treegraph
Reported: 2020-06-25 16:43 UTC by Thierry Thomas
Modified: 2020-08-03 18:55 UTC (History)
6 users (show)

See Also:
antoine: exp-run?

Upgrade blas + lapack & Cie (36.19 KB, patch)
2020-06-25 16:43 UTC, Thierry Thomas
no flags Details | Diff
Remove the CONFLICTS line in math/openblas (340 bytes, patch)
2020-06-27 08:55 UTC, Thierry Thomas
no flags Details | Diff
2nd version of the patch (24.01 KB, patch)
2020-07-12 20:46 UTC, Thierry Thomas
no flags Details | Diff
Fix the preious version of the patch (37.14 KB, patch)
2020-07-20 20:42 UTC, Thierry Thomas
no flags Details | Diff
Complete patch, including a fix for Treekin (40.88 KB, patch)
2020-07-21 20:05 UTC, Thierry Thomas
no flags Details | Diff
math/lapack + blas + cblas + xblas + some consumers diff (40.96 KB, patch)
2020-08-02 20:15 UTC, Thierry Thomas
no flags Details | Diff
[patch] update lapack & slave ports to 3.9.0 [v7] (39.66 KB, patch)
2020-08-03 11:08 UTC, John Hein
jcfyecrayz: maintainer-approval? (thierry)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Thierry Thomas freebsd_committer 2020-06-25 16:43:07 UTC
Created attachment 215939 [details]
Upgrade blas + lapack & Cie

- Upgrade to 3.9.0
  Release notes at http://www.netlib.org/lapack/lapack-3.9.0.html
- Upstream suggest to use an optimized BLAS to build LAPACK, but keep
  this BLAS from Netlib to avoid conflict among dependencies.
- Remove options: these ports have a lot of consumers, and it is
  impossible to check every combinations:
  + always build static and shared libraries when possible;
  + remove the profiled binaries, which seem unused.
- Notify phd_kimberlite@yahoo.co.jp (lapacke´s maintainer)
- set the maintainer of math/blacs to fortran@ (because of the MASTER_PORT)
- Avoid conflicts with OpenBLAS (See PR 244296)
Comment 1 Thierry Thomas freebsd_committer 2020-06-25 16:45:02 UTC
portmgr, please schedule an experimental build.
Comment 2 Thierry Thomas freebsd_committer 2020-06-27 08:55:55 UTC
Created attachment 215975 [details]
Remove the CONFLICTS line in math/openblas

Add a forgotten patch: remove the CONFLICTS line in math/openblas.
Comment 4 Thierry Thomas freebsd_committer 2020-07-07 19:34:54 UTC
The problem with math/maxima is not related to this upgrade, and it has been fixed by zeising@ in r541427.
Comment 5 Thierry Thomas freebsd_committer 2020-07-12 20:46:43 UTC
Created attachment 216409 [details]
2nd version of the patch

New version of the patch:

- science/scilab
  unfortunately Scilab still uses deprecated functions
  => buld lapack with deprecated funtions to fix it.

- science/py-phono3py
	patch lapacke.h -> lapacke/lapacke.h

The remaining problem is biology/treekin.
Comment 6 John Hein 2020-07-17 19:36:27 UTC
(In reply to Thierry Thomas from comment #5)
Did you perhaps intend to delete the '.include <bsd.port.options.mk>' line?  The latest patch adds a second one.


'make patch' in math/lapack is failing.  I think you just forgot some files/patch* deletions that were in the first patch.

Some additions were omitted in the new patch, too (e.g., static.mk).


The manpages.tgz file now no longer has a version tag.  If one keeps old distfiles around (I do), that can cause a conflict on an update.  Maybe consider DIST_SUBDIR=lapack/${PORTVERSION} (qemu does that, for example, and linux-flashplayer)?


In static.mk (in the original patch), it references SRC in two places.  The post-build replaces 'cd SRC' with 'cd xxx/SRC', but does not replace SRC in the RANLIB line.  So ranlib is failing as follows when, for instance, building math/blas:

ranlib: 'SRC/libblas.a': No such file

Maybe just remove 'cd ' in the sed lines in post-build so the sed replaces both instances of SRC?

I don't understand why the earlier exp-run was not failing here.

Comment 7 Thierry Thomas freebsd_committer 2020-07-20 20:42:41 UTC
Created attachment 216611 [details]
Fix the preious version of the patch

It seems that the previous version of the patch was incorrect: regenerate it.
Thanks to John!
Comment 8 Thierry Thomas freebsd_committer 2020-07-21 20:05:08 UTC
Created attachment 216640 [details]
Complete patch, including a fix for Treekin

This patch fixes the problem encountered with biology/treekin (See PR 247939).

With this one, all the problems shown in the first exp-run are fixed: please schedule another one.
Comment 9 John Hein 2020-07-22 22:39:40 UTC
(In reply to Thierry Thomas from comment #8)


Carriage return line endings in attachment 216640 [details].  Testers should dos2unix it first.


I still get 'ranlib' errors (see comment 6).  But I see why exp-run did not notice it last time.  When ranlib fails, it still exits with 0, so make does not notice the failure (could be considered a ranlib bug I suppose).

In any case, the fix is still the same I think - get rid of 'cd ' in the sed substitutions in post-build.  For example,

Comment 10 Thierry Thomas freebsd_committer 2020-07-23 16:13:10 UTC
(In reply to John Hein from comment #9)

Hello John,

Could you please tell me the line number containing a ^M ?
I'm sorry, but I cannot find it...

I cannot either reproduce the problem with ranlib, neither on poudriere nor on my workstation. Could you please send me a full log, with your platform / version?

Comment 11 John Hein 2020-07-24 22:30:54 UTC
(In reply to Thierry Thomas from comment #10)
Every line in attachment 216640 [details] has a '\r' (viewable with hd or vis -tl or sed -n l or basic vi (nvi); vim will "hide" them for you, but you will see "[dos]" at the bottom).  I downloaded the attachment with fetch(1) (wget gives the same result).  I just tested 'save link as' from firefox - that preserves the '\r' endings that are in the file as well.

Sure, I can attach a build log of math/blas showing the ranlib error.  But if you inspect work/.build/BLAS/SRC/CMakeFiles/blas.dir/build.make - specifically the build-static target at the bottom - you will see the problem:

    cd BLAS/SRC && $(AR) $(ARFLAGS) libblas.a $(blas_OBJECTS)
    $(RANLIB) SRC/libblas.a

The cd && ar works fine, but ranlib is using the wrong path to the lib file.  This is because post-build in math/lapack/Makefile does a substitution for the path in math/lapack/files/static.mk that is specific to the slave port (which then is appended to the build.make file).  But it only looks for 'cd SRC' to replace.  This misses the ranlib line which has no 'cd ' before SRC, so sed does no substitution on that line.
Comment 12 Thierry Thomas freebsd_committer 2020-07-26 20:41:20 UTC
(In reply to John Hein from comment #11)

I'm really sorry, but I cannot reproduce the problem with the patch format. I could not even imagine how I could produce such a thing: I do not use Windows machines, my patches are produced directly on a FreeBSD workstation by `svn diff', and they are uploaded to bugzilla from the same machine, using Firefox.

About ranlib, I have understood! These ports are quiet complex, and I tried to simplify them, but re-using the same parts between the different ports. And you are right, ranlib emit a warning for math/blas, but this is just a warning, and the static library is built anyways during post-build. Does'nt it produce a $PREFIX/lib/libblas.a when you make it?
Comment 13 John Hein 2020-07-28 19:35:08 UTC
(In reply to Thierry Thomas from comment #12)

re: CR line endings...

I don't know either why I get the patch with CR line endings and it seems that you don't (I get this on multiple linux, freebsd platforms - I have not tried windows yet).

What md5 do you have for the downloaded attachment?

Here's what I did:

# make sure I don't have an old copy of the attachment file in my current directory, then...

% wget 'https://bugs.freebsd.org/bugzilla/attachment.cgi?id=216640'
--2020-07-28 13:01:26--  https://bugs.freebsd.org/bugzilla/attachment.cgi?id=216640
Resolving bugs.freebsd.org (bugs.freebsd.org)..., 2610:1c1:1:606c::50:15
Connecting to bugs.freebsd.org (bugs.freebsd.org)||:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://bz-attachments.freebsd.org/attachment.cgi?id=216640 [following]
--2020-07-28 13:01:27--  https://bz-attachments.freebsd.org/attachment.cgi?id=216640
Resolving bz-attachments.freebsd.org (bz-attachments.freebsd.org)..., 2610:1c1:1:606c::50:15
Connecting to bz-attachments.freebsd.org (bz-attachments.freebsd.org)||:443... connected.
HTTP request sent, awaiting response... md200 OK
Cookie coming from bz-attachments.freebsd.org attempted to set domain to bugs.freebsd.org
Length: 41864 (41K) [text/plain]
Saving to: ‘attachment.cgi?id=216640’

attachment.cgi?id=216640      100%[=================================================>]  40.88K  --.-KB/s    in 0.05s   

2020-07-28 13:01:28 (840 KB/s) - ‘attachment.cgi?id=216640’ saved [41864/41864]

% sed -n l 
sed -n l attachment.cgi\?id=216640 | head
Index: math/blas/Makefile\r$
--- math/blas/Makefile\t(revision 542588)\r$
+++ math/blas/Makefile\t(working copy)\r$
@@ -2,7 +2,6 @@\r$
 # $FreeBSD$\r$

% md5 attachment-216640
MD5 (attachment.cgi?id=216640) = 5d7018e016449aac293b4bb01ed31219

What md5 do you have?


Re: ranlib issue...

You asked if I get a libblas.a.  I do.  But that's because ranlib(1) does not produce the file, ar(1) (on the line above the ranlib command) does.  ranlib(1) adds/updates a symbol table which is important for linking.  Some versions of ar(1) will do this by default (GNU binutils does, for instance).  Some versions of ar(1) do not (libarchive's ar in base).  Running ranlib(1) assures that a symbol table is added so static linking with the .a archive will "work".

Anyway, again... just remove 'cd ' from your sed(1) expressions in post-build and things will work.  Namely, you will provide the right path to the .a file for ranlib(1).  And a few less characters in post-build - bonus!

You seem hesitant to make that change, however.  Do you have concerns?
Comment 14 Antoine Brodin freebsd_committer 2020-07-30 06:04:28 UTC
There is a problem:

===>  Patching for blas-3.9.0
===>  Applying FreeBSD patches for blas-3.9.0 from /usr/ports/math/blas/../lapack/files
2 out of 2 hunks failed--saving rejects to LAPACKE/include/lapack.h.rej
===>  FAILED Applying FreeBSD patch-LAPACKE_include_lapack.h
===> FAILED to apply cleanly FreeBSD patch(es)  patch-LAPACKE_include_lapack.h
*** Error code 1
Comment 15 John Hein 2020-07-30 15:29:16 UTC
(In reply to Antoine Brodin from comment #14)
Antoine, the patch has CR line endings.  Does it work if you dos2unix attachment 216640 [details]?
Comment 16 Thierry Thomas freebsd_committer 2020-08-02 20:15:11 UTC
Created attachment 216966 [details]
math/lapack + blas + cblas + xblas + some consumers diff

John: you did not provide a patch, and I'm not sure to understand what you meant, but anyway I tried to fix the problem with ranlib.

Antoine: I do not understand anything about this patch formats, dos2unix and so on, but I have regenerated my patch, and tested it against a fresh ports tree, and it seems OK for me, and the resulting ports build in poudriere.

SHA256 (math_lapack.diff) = 07533618cb74a48d8aeac9ca1fbb473660fc9fc1a45aac990c2ea29d3f68bdf5
Comment 17 John Hein 2020-08-03 11:08:12 UTC
Created attachment 216986 [details]
[patch] update lapack & slave ports to 3.9.0 [v7]

(In reply to Thierry Thomas from comment #16)

Whatever you are doing during upload is adding CR line endings - attachment 216966 [details] has them as well.  You can check yourself by downloading the attachment.  If Antoinne runs an exp-run, he's going to hit the same problem again.

Here's the sha256 for the downloaded attachment.

SHA256 (attachment-216966) = 0c34dd36e60690217213091a55b2d9df2ce17657a02289edac8f2aa89102e6cf

After dos2unix:

SHA256 (attachment-216966.nocr) = 07533618cb74a48d8aeac9ca1fbb473660fc9fc1a45aac990c2ea29d3f68bdf5

If you don't have dos2unix (converters/unix2dos), you can use tr:

tr -d '\r' < attachment-216966 > attachment-216966.nocr

The first 4 (obsolete now) versions of you patch did not have CR line endings.  Consider what you may have done differently.  How do you upload the attachments? 


Re: the ranlib issue.  I didn't add a patch because I thought my suggestion was worded clearly enough twice that you could make the change yourself (that is, to remove the 'cd ' from your sed expressions).

But here now is a new attachment that does not have CR line endings.  It also has the fix for ranlib that I was suggesting.  Note that in your patch, you forgot to fix the case for LAPACKE_SLAVEPORT.  This attachment fixes that as well.  This was build tested in poudriere.

If you are okay with the new patch attached, mark your other patch obsolete and ask for the exp-run with the new one.
Comment 18 Thierry Thomas freebsd_committer 2020-08-03 18:41:33 UTC
Since I'm not even able to submit a patch, let's forget about that.
Comment 19 Dima Pasechnik 2020-08-03 18:55:39 UTC
I'm watching this in disbelief. Looks like about time to switch to git...  (even github)