Bug 247542 - math/lapack: upgrade to 3.9.1 with math/blas + math/xlapack + math/cblas + math/lapacke
Summary: math/lapack: upgrade to 3.9.1 with math/blas + math/xlapack + math/cblas + ma...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Thierry Thomas
URL: http://www.netlib.org/lapack/lapack-3...
Keywords:
Depends on: 247939
Blocks: 247485
  Show dependency treegraph
 
Reported: 2020-06-25 16:43 UTC by Thierry Thomas
Modified: 2021-09-11 12:40 UTC (History)
8 users (show)

See Also:
antoine: exp-run+


Attachments
Upgrade blas + lapack & Cie (36.19 KB, patch)
2020-06-25 16:43 UTC, Thierry Thomas
no flags Details | Diff
Remove the CONFLICTS line in math/openblas (340 bytes, patch)
2020-06-27 08:55 UTC, Thierry Thomas
no flags Details | Diff
2nd version of the patch (24.01 KB, patch)
2020-07-12 20:46 UTC, Thierry Thomas
no flags Details | Diff
Fix the preious version of the patch (37.14 KB, patch)
2020-07-20 20:42 UTC, Thierry Thomas
no flags Details | Diff
Complete patch, including a fix for Treekin (40.88 KB, patch)
2020-07-21 20:05 UTC, Thierry Thomas
no flags Details | Diff
math/lapack + blas + cblas + xblas + some consumers diff (40.96 KB, patch)
2020-08-02 20:15 UTC, Thierry Thomas
no flags Details | Diff
[patch] update lapack & slave ports to 3.9.0 [v7] (39.66 KB, patch)
2020-08-03 11:08 UTC, John Hein
no flags Details | Diff
Upgrade lapack & Co to 3.9.1 (53.44 KB, patch)
2021-06-09 15:28 UTC, Thierry Thomas
no flags Details | Diff
Upgrade lapack & Co to 3.9.1 (54.31 KB, patch)
2021-06-09 16:54 UTC, Thierry Thomas
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Thierry Thomas freebsd_committer 2020-06-25 16:43:07 UTC
Created attachment 215939 [details]
Upgrade blas + lapack & Cie

- Upgrade to 3.9.0
  Release notes at http://www.netlib.org/lapack/lapack-3.9.0.html
- Upstream suggest to use an optimized BLAS to build LAPACK, but keep
  this BLAS from Netlib to avoid conflict among dependencies.
- Remove options: these ports have a lot of consumers, and it is
  impossible to check every combinations:
  + always build static and shared libraries when possible;
  + remove the profiled binaries, which seem unused.
- Notify phd_kimberlite@yahoo.co.jp (lapacke´s maintainer)
- set the maintainer of math/blacs to fortran@ (because of the MASTER_PORT)
- Avoid conflicts with OpenBLAS (See PR 244296)
Comment 1 Thierry Thomas freebsd_committer 2020-06-25 16:45:02 UTC
portmgr, please schedule an experimental build.
Comment 2 Thierry Thomas freebsd_committer 2020-06-27 08:55:55 UTC
Created attachment 215975 [details]
Remove the CONFLICTS line in math/openblas

Add a forgotten patch: remove the CONFLICTS line in math/openblas.
Comment 4 Thierry Thomas freebsd_committer 2020-07-07 19:34:54 UTC
The problem with math/maxima is not related to this upgrade, and it has been fixed by zeising@ in r541427.
Comment 5 Thierry Thomas freebsd_committer 2020-07-12 20:46:43 UTC
Created attachment 216409 [details]
2nd version of the patch

New version of the patch:

- science/scilab
  unfortunately Scilab still uses deprecated functions
  => buld lapack with deprecated funtions to fix it.

- science/py-phono3py
	patch lapacke.h -> lapacke/lapacke.h

The remaining problem is biology/treekin.
Comment 6 John Hein 2020-07-17 19:36:27 UTC
(In reply to Thierry Thomas from comment #5)
Did you perhaps intend to delete the '.include <bsd.port.options.mk>' line?  The latest patch adds a second one.


===============

'make patch' in math/lapack is failing.  I think you just forgot some files/patch* deletions that were in the first patch.

Some additions were omitted in the new patch, too (e.g., static.mk).

===================

The manpages.tgz file now no longer has a version tag.  If one keeps old distfiles around (I do), that can cause a conflict on an update.  Maybe consider DIST_SUBDIR=lapack/${PORTVERSION} (qemu does that, for example, and linux-flashplayer)?

==================

In static.mk (in the original patch), it references SRC in two places.  The post-build replaces 'cd SRC' with 'cd xxx/SRC', but does not replace SRC in the RANLIB line.  So ranlib is failing as follows when, for instance, building math/blas:

ranlib: 'SRC/libblas.a': No such file

Maybe just remove 'cd ' in the sed lines in post-build so the sed replaces both instances of SRC?

I don't understand why the earlier exp-run was not failing here.

=================
Comment 7 Thierry Thomas freebsd_committer 2020-07-20 20:42:41 UTC
Created attachment 216611 [details]
Fix the preious version of the patch

It seems that the previous version of the patch was incorrect: regenerate it.
Thanks to John!
Comment 8 Thierry Thomas freebsd_committer 2020-07-21 20:05:08 UTC
Created attachment 216640 [details]
Complete patch, including a fix for Treekin

This patch fixes the problem encountered with biology/treekin (See PR 247939).

With this one, all the problems shown in the first exp-run are fixed: please schedule another one.
Comment 9 John Hein 2020-07-22 22:39:40 UTC
(In reply to Thierry Thomas from comment #8)

======

Carriage return line endings in attachment 216640 [details].  Testers should dos2unix it first.

======

I still get 'ranlib' errors (see comment 6).  But I see why exp-run did not notice it last time.  When ranlib fails, it still exits with 0, so make does not notice the failure (could be considered a ranlib bug I suppose).

In any case, the fix is still the same I think - get rid of 'cd ' in the sed substitutions in post-build.  For example,

s|SRC|BLAS/SRC|
Comment 10 Thierry Thomas freebsd_committer 2020-07-23 16:13:10 UTC
(In reply to John Hein from comment #9)

Hello John,

Could you please tell me the line number containing a ^M ?
I'm sorry, but I cannot find it...

I cannot either reproduce the problem with ranlib, neither on poudriere nor on my workstation. Could you please send me a full log, with your platform / version?

Thanks.
Comment 11 John Hein 2020-07-24 22:30:54 UTC
(In reply to Thierry Thomas from comment #10)
Every line in attachment 216640 [details] has a '\r' (viewable with hd or vis -tl or sed -n l or basic vi (nvi); vim will "hide" them for you, but you will see "[dos]" at the bottom).  I downloaded the attachment with fetch(1) (wget gives the same result).  I just tested 'save link as' from firefox - that preserves the '\r' endings that are in the file as well.

Sure, I can attach a build log of math/blas showing the ranlib error.  But if you inspect work/.build/BLAS/SRC/CMakeFiles/blas.dir/build.make - specifically the build-static target at the bottom - you will see the problem:

build-static:
    cd BLAS/SRC && $(AR) $(ARFLAGS) libblas.a $(blas_OBJECTS)
    $(RANLIB) SRC/libblas.a

The cd && ar works fine, but ranlib is using the wrong path to the lib file.  This is because post-build in math/lapack/Makefile does a substitution for the path in math/lapack/files/static.mk that is specific to the slave port (which then is appended to the build.make file).  But it only looks for 'cd SRC' to replace.  This misses the ranlib line which has no 'cd ' before SRC, so sed does no substitution on that line.
Comment 12 Thierry Thomas freebsd_committer 2020-07-26 20:41:20 UTC
(In reply to John Hein from comment #11)

I'm really sorry, but I cannot reproduce the problem with the patch format. I could not even imagine how I could produce such a thing: I do not use Windows machines, my patches are produced directly on a FreeBSD workstation by `svn diff', and they are uploaded to bugzilla from the same machine, using Firefox.

About ranlib, I have understood! These ports are quiet complex, and I tried to simplify them, but re-using the same parts between the different ports. And you are right, ranlib emit a warning for math/blas, but this is just a warning, and the static library is built anyways during post-build. Does'nt it produce a $PREFIX/lib/libblas.a when you make it?
Comment 13 John Hein 2020-07-28 19:35:08 UTC
(In reply to Thierry Thomas from comment #12)


re: CR line endings...

I don't know either why I get the patch with CR line endings and it seems that you don't (I get this on multiple linux, freebsd platforms - I have not tried windows yet).

What md5 do you have for the downloaded attachment?

Here's what I did:

# make sure I don't have an old copy of the attachment file in my current directory, then...

% wget 'https://bugs.freebsd.org/bugzilla/attachment.cgi?id=216640'
--2020-07-28 13:01:26--  https://bugs.freebsd.org/bugzilla/attachment.cgi?id=216640
Resolving bugs.freebsd.org (bugs.freebsd.org)... 96.47.72.84, 2610:1c1:1:606c::50:15
Connecting to bugs.freebsd.org (bugs.freebsd.org)|96.47.72.84|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://bz-attachments.freebsd.org/attachment.cgi?id=216640 [following]
--2020-07-28 13:01:27--  https://bz-attachments.freebsd.org/attachment.cgi?id=216640
Resolving bz-attachments.freebsd.org (bz-attachments.freebsd.org)... 96.47.72.84, 2610:1c1:1:606c::50:15
Connecting to bz-attachments.freebsd.org (bz-attachments.freebsd.org)|96.47.72.84|:443... connected.
HTTP request sent, awaiting response... md200 OK
Cookie coming from bz-attachments.freebsd.org attempted to set domain to bugs.freebsd.org
Length: 41864 (41K) [text/plain]
Saving to: ‘attachment.cgi?id=216640’

attachment.cgi?id=216640      100%[=================================================>]  40.88K  --.-KB/s    in 0.05s   

2020-07-28 13:01:28 (840 KB/s) - ‘attachment.cgi?id=216640’ saved [41864/41864]

% sed -n l 
sed -n l attachment.cgi\?id=216640 | head
Index: math/blas/Makefile\r$
===================================================================\r$
--- math/blas/Makefile\t(revision 542588)\r$
+++ math/blas/Makefile\t(working copy)\r$
@@ -2,7 +2,6 @@\r$
 # $FreeBSD$\r$
 \r$
 PORTNAME=\tblas\r$
-PORTREVISION=\t6\r$
 \r$

% md5 attachment-216640
MD5 (attachment.cgi?id=216640) = 5d7018e016449aac293b4bb01ed31219


What md5 do you have?


================================================================

Re: ranlib issue...

You asked if I get a libblas.a.  I do.  But that's because ranlib(1) does not produce the file, ar(1) (on the line above the ranlib command) does.  ranlib(1) adds/updates a symbol table which is important for linking.  Some versions of ar(1) will do this by default (GNU binutils does, for instance).  Some versions of ar(1) do not (libarchive's ar in base).  Running ranlib(1) assures that a symbol table is added so static linking with the .a archive will "work".

Anyway, again... just remove 'cd ' from your sed(1) expressions in post-build and things will work.  Namely, you will provide the right path to the .a file for ranlib(1).  And a few less characters in post-build - bonus!

You seem hesitant to make that change, however.  Do you have concerns?
Comment 14 Antoine Brodin freebsd_committer 2020-07-30 06:04:28 UTC
There is a problem:

===>  Patching for blas-3.9.0
===>  Applying FreeBSD patches for blas-3.9.0 from /usr/ports/math/blas/../lapack/files
2 out of 2 hunks failed--saving rejects to LAPACKE/include/lapack.h.rej
===>  FAILED Applying FreeBSD patch-LAPACKE_include_lapack.h
===> FAILED to apply cleanly FreeBSD patch(es)  patch-LAPACKE_include_lapack.h
*** Error code 1
Comment 15 John Hein 2020-07-30 15:29:16 UTC
(In reply to Antoine Brodin from comment #14)
Antoine, the patch has CR line endings.  Does it work if you dos2unix attachment 216640 [details]?
Comment 16 Thierry Thomas freebsd_committer 2020-08-02 20:15:11 UTC
Created attachment 216966 [details]
math/lapack + blas + cblas + xblas + some consumers diff

John: you did not provide a patch, and I'm not sure to understand what you meant, but anyway I tried to fix the problem with ranlib.

Antoine: I do not understand anything about this patch formats, dos2unix and so on, but I have regenerated my patch, and tested it against a fresh ports tree, and it seems OK for me, and the resulting ports build in poudriere.

SHA256 (math_lapack.diff) = 07533618cb74a48d8aeac9ca1fbb473660fc9fc1a45aac990c2ea29d3f68bdf5
Comment 17 John Hein 2020-08-03 11:08:12 UTC
Created attachment 216986 [details]
[patch] update lapack & slave ports to 3.9.0 [v7]

(In reply to Thierry Thomas from comment #16)

Thierry,
Whatever you are doing during upload is adding CR line endings - attachment 216966 [details] has them as well.  You can check yourself by downloading the attachment.  If Antoinne runs an exp-run, he's going to hit the same problem again.

Here's the sha256 for the downloaded attachment.

SHA256 (attachment-216966) = 0c34dd36e60690217213091a55b2d9df2ce17657a02289edac8f2aa89102e6cf

After dos2unix:

SHA256 (attachment-216966.nocr) = 07533618cb74a48d8aeac9ca1fbb473660fc9fc1a45aac990c2ea29d3f68bdf5

If you don't have dos2unix (converters/unix2dos), you can use tr:

tr -d '\r' < attachment-216966 > attachment-216966.nocr


The first 4 (obsolete now) versions of you patch did not have CR line endings.  Consider what you may have done differently.  How do you upload the attachments? 

=================

Re: the ranlib issue.  I didn't add a patch because I thought my suggestion was worded clearly enough twice that you could make the change yourself (that is, to remove the 'cd ' from your sed expressions).

But here now is a new attachment that does not have CR line endings.  It also has the fix for ranlib that I was suggesting.  Note that in your patch, you forgot to fix the case for LAPACKE_SLAVEPORT.  This attachment fixes that as well.  This was build tested in poudriere.

If you are okay with the new patch attached, mark your other patch obsolete and ask for the exp-run with the new one.
Comment 18 Thierry Thomas freebsd_committer 2020-08-03 18:41:33 UTC
Since I'm not even able to submit a patch, let's forget about that.
Comment 19 Dima Pasechnik 2020-08-03 18:55:39 UTC
I'm watching this in disbelief. Looks like about time to switch to git...  (even github)
Comment 20 j.david.lists 2021-05-28 22:34:07 UTC
Is it possible to revisit this?

The ports tree has moved to git and I still can't build the combination of numeric packages my users need without applying this patch (which I note I have never had a problem applying, even under the old svn trees).
Comment 21 John Hein 2021-06-07 16:04:31 UTC
(In reply to j.david.lists from comment #20)
Sure.  Thierry had some sort of mechanical troubles uploading a patch for unknown causes (unknown at least to me - Thierry did not respond with answers to the troubleshooting questions), and it seems he just gave up.

Feel free to open a new bug with my patch as a starting point.  At the time, I build tested it and did some light run testing.  For day to day usage, I mostly use scipy which depends on lapack & blas.  I have been using 3.9.0 since last summer.

Since last year, lapack 3.9.1 is out, so I would probably bump to 3.9.1 instead of 3.9.0.  new distinfo:

+TIMESTAMP = 1622566308
+SHA256 (lapack/v3.9.1.tar.gz) = d0085d2caf997ff39299c05d4bacb6f3d27001d25a4cc613d48c1f352b73e7e0
+SIZE (lapack/v3.9.1.tar.gz) = 7543209
+SHA256 (lapack/manpages.tgz) = 65181ed22a87cde3cfee3bc9e4135434e3ee9f5fc5651a37c0ca4e4b737254de
+SIZE (lapack/manpages.tgz) = 1378304
Comment 22 Thierry Thomas freebsd_committer 2021-06-07 16:50:37 UTC
(In reply to John Hein from comment #21)

Besides this PR, I submitted many PR, and nobody else had problems with my patches, you are the only one.
Comment 23 John Hein 2021-06-08 18:05:16 UTC
(In reply to Thierry Thomas from comment #22)
It wasn't just me.  Antoine had problems - see comment 14.

It may be a local problem on my end, but it doesn't seem so to me.  I tried a number of different tools to grab the patch (fetch, curl, web browser, linux, freebsd).  Every time I downloaded the patches in question, they had CR line endings.  Did you ever try the steps in comment 13?
Comment 24 Dima Pasechnik 2021-06-08 18:37:59 UTC
It seems to work just fine now. Perhaps it was a server-side probem?


    % curl -L "https://bz-attachments.freebsd.org/attachment.cgi?      id=216640&action=diff&format=raw&headers=1" >x
    % file x                                                                                                   
    x: unified diff output text, ASCII text
Comment 25 j.david.lists 2021-06-08 18:49:07 UTC
Perhaps we could table the issue of whether there's some issue in the bug tracker that causes problems for some people when they download a patch through it.  That's not really an issue with the patch.

Thierry, the patch appears to work.  You have commit privileges in the ports tree.

Does your proposed patch pass FreeBSD's ports CI requirements (whatever those may be) when you apply it?

If so, what are the next steps required for you to commit the patch?  I am not familiar with the process (neither technology nor policy).

The existing issue seems to be adequately complicated already.  I would suggest updating to 3.9.0 with the existing patch, then doing an update to 3.9.1 in the future when the dust settles.  That should be a straightforward update on its own, and incremental progress would be better than no progress.
Comment 26 Thierry Thomas freebsd_committer 2021-06-08 19:39:19 UTC
(In reply to j.david.lists from comment #25)

Anyway, be it 3.9.0 or 3.9.1, meanwhile many ports have been updated, and everything must be tested again.

Let me some days to check it again, and when everything will appear OK on my side, I'll ask Antoine to relaunch an exp-run.
Comment 27 Thierry Thomas freebsd_committer 2021-06-09 15:28:14 UTC
Created attachment 225662 [details]
Upgrade lapack & Co to 3.9.1

I decided to upgrade directly to 3.9.1: compared to 3.9.0, this is a minor release but it fixes some points (e.g. one patch can be removed).

Release notes at <http://www.netlib.org/lapack/lapack-3.9.1.html>.

Also:

- Add a test target
- set -fallow-argument-mismatch for math/xlapack
- science/py-phono3py switched to openblas => no patch needed anymore.

Antoine, can you please schedule an exp-run?
Comment 29 Thierry Thomas freebsd_committer 2021-06-09 16:54:45 UTC
Created attachment 225664 [details]
Upgrade lapack & Co to 3.9.1

(In reply to Antoine Brodin from comment #28)
Strange…
Sorry, I thought that `git commit -a' would take everything, but a previous `git add' seems required.
Comment 30 Antoine Brodin freebsd_committer 2021-06-13 16:09:22 UTC
Exp-run looks fine
Comment 31 Thierry Thomas freebsd_committer 2021-06-13 17:49:00 UTC
Committed - finally!
Comment 32 commit-hook freebsd_committer 2021-06-13 17:49:10 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=c2488a6020830af2cf09699b887adec7af806bf3

commit c2488a6020830af2cf09699b887adec7af806bf3
Author:     Thierry Thomas <thierry@FreeBSD.org>
AuthorDate: 2021-06-13 17:35:55 +0000
Commit:     Thierry Thomas <thierry@FreeBSD.org>
CommitDate: 2021-06-13 17:48:12 +0000

    math/lapack: +math/blas et al., upgrade to 3.9.1

    - Upgrade math/blas, math/cblas, math/lapack, math/lapacke and math/xlapack
      to 3.9.1;
      Latest release notes at <http://www.netlib.org/lapack/lapack-3.9.1.html>

    - Chase this upgrade in biology/treekin;

    - Add a test target;

    - Remove a conflict with math/openblas (PR 244296);

    - Fix the build with Gcc10 (PR 247485).

    PR:             247542
    Approved by:    expr-run by antoine@

 biology/treekin/Makefile                           |   2 +-
 biology/treekin/files/patch-src_calcpp.h (new)     |  12 +
 math/blas/Makefile                                 |   1 -
 math/cblas/Makefile                                | 133 +-----
 math/cblas/distinfo (gone)                         |   4 -
 math/lapack/Makefile                               | 298 ++++++--------
 math/lapack/distinfo                               |   9 +-
 math/lapack/files/manpages                         | 458 ++++++++++++++++++---
 math/lapack/files/patch-Makefile (gone)            |  11 -
 math/lapack/files/patch-SRC+Makefile (gone)        |  98 -----
 .../lapack/files/patch-TESTING+LIN+Makefile (gone) |  82 ----
 math/lapack/files/patch-TESTING+Makefile (gone)    |  14 -
 math/lapack/files/patch-lapacke+Makefile (gone)    |  30 --
 .../lapack/files/patch-lapacke+src+Makefile (gone) |  29 --
 math/lapack/files/static.mk (new)                  |   3 +
 math/lapack/pkg-descr                              |   1 +
 math/lapack/pkg-plist                              |  92 +++--
 math/openblas/Makefile                             |   2 -
 math/xlapack/Makefile                              |   1 -
 19 files changed, 599 insertions(+), 681 deletions(-)
Comment 33 Daniel Engberg freebsd_committer 2021-09-11 12:40:31 UTC
Hi Thierry,

Looking at ports-fallout this seems to have broken biology/treekin as _3 doesn't to build on any platform.

https://portsfallout.com/port/1558/

Best regards,
Daniel