Bug 257122

Summary: math/atlas fails with lapack 3.10.0 lapacklib not valid make target during LAPACK build phase
Product: Ports & Packages Reporter: alt2600
Component: Individual Port(s)Assignee: Thierry Thomas <thierry>
Status: Closed FIXED    
Severity: Affects Only Me CC: thierry
Priority: --- Keywords: needs-qa
Version: LatestFlags: thierry: merge-quarterly+
Hardware: Any   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247485
Attachments:
Description Flags
Full Atlas Build log none

Description alt2600 2021-07-11 23:30:39 UTC
Original PR 247485 marked closed, fixed by updating to LAPACK 3.9.1 but atlas still doesn't build. This is a new error to the original issue on atlas of having mismatched parameters in function calls (strict type checking in gcc fortran 10+). This says it doesn't know how to make lapacklib, I didn't see anything in lapack that seems to define this target in history of its main Makefile. I see this lapacklib target is within the Atlas do-build: .for opt loop. Not sure this implies lapack Makefile needs to have an ATLAS_SLAVEPORT defined, if so I can look into. Might be beyond my skills though.

I did look over updating Atlas itself, but the issue is it has so many architecture patches for things like Sparc and PowerPC that was more then a little overwhelmingly daunting. Not to mention the custom build system defined inside atlas's Makefile seemingly to get lapack staged for optimizations, unless the above mentioned ATLAS_SLAVEPORT could make this become not necessary. It may just be checking our patches against what is in upstream now, but I'd have no way to check anything but amd64. Not sure anyone is working on the task, or if help is needed there.

===>  Building for atlas-math-3.8.4_18,1
Building the LAPACK archives with shared ATLAS flags:
F77FLAGS="`make -f /usr/ports/math/atlas/work/ATLAS/saved_flags -V F77FLAGS` " ;  LW=`make -C /usr/ports/math/lapack -V WRKSRC_SHARED` ;  make -C /usr/ports/math/lapack WITH=SHARED clean patch ;  cd ${LW} ; /usr/bin/env F77="gfortran10" FC="gfortran10" FCFLAGS="-Wl,-rpath=/usr/local/lib/gcc10" XDG_DATA_HOME=/usr/ports/math/atlas/work XDG_CONFIG_HOME=/usr/ports/math/atlas/work HOME=/usr/ports/math/atlas/work PATH=/usr/ports/math/atlas/work/.bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/backyard/bin:/usr/local/tigcc/bin:/usr/local/tigcc/bin NO_PIE=yes MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES ADDR2LINE="/usr/local/bin/addr2line" AR="/usr/local/bin/ar" AS="/usr/local/bin/as" CPPFILT="/usr/local/bin/c++filt" GPROF="/usr/local/bin/gprof" LD="/usr/local/bin/ld" NM="/usr/local/bin/nm" OBJCOPY="/usr/local/bin/objcopy" OBJDUMP="/usr/local/bin/objdump" RANLIB="/usr/local/bin/ranlib" READELF="/usr/local/bin/readelf" SIZE="/usr/local/bin/size" STRINGS="/usr/local/bin/strings" PREFIX=/usr/local LOCALBASE=/usr/local CC="gcc10" CFLAGS="-O2 -pipe -march=westmere  -fstack-protector-strong -Wl,-rpath=/usr/local/lib/gcc10 -fno-strict-aliasing " CPP="cpp10" CPPFLAGS="" LDFLAGS=" -Wl,-rpath=/usr/local/lib/gcc10  -L/usr/local/lib/gcc10 -B/usr/local/bin -fstack-protector-strong -Wl,-rpath=/usr/local/lib/gcc10 -L/usr/local/lib/gcc10 " LIBS="" CXX="g++10" CXXFLAGS="-O2 -pipe -march=westmere  -fstack-protector-strong -Wl,-rpath=/usr/local/lib/gcc10  -Wl,-rpath=/usr/local/lib/gcc10 " MANPREFIX="/usr/local" BSD_INSTALL_PROGRAM="install  -s -m 555" BSD_INSTALL_LIB="install  -s -m 0644" BSD_INSTALL_SCRIPT="install  -m 555" BSD_INSTALL_DATA="install  -m 0644" BSD_INSTALL_MAN="install  -m 444" EXTRAFLAGS="-fpic -DPIC"  FFLAGS="-pipe ${F77FLAGS}" make -j10 DESTDIR=/usr/ports/math/atlas/work/stage  ARCH="/usr/local/bin/ar" lapacklib ;  TIMEFLAG=`make -f /usr/ports/math/atlas/work/ATLAS/saved_flags -V TIMEFLAG` ;  /bin/mkdir -p /usr/ports/math/atlas/work/ATLAS/shared ; cd /usr/ports/math/atlas/work/ATLAS/shared ;  /bin/mv ${LW}/liblapack.a liblapack_shared.a ;  make -C /usr/ports/math/lapack clean ;  echo "Configuring shared ATLAS:" ;  ../configure --cc="gcc10" --cflags="-O2 -pipe -march=westmere  -fstack-protector-strong -Wl,-rpath=/usr/local/lib/gcc10 -fno-strict-aliasing "  --with-netlib-lapack="/usr/ports/math/atlas/work/ATLAS/shared/liblapack_shared.a"  --prefix="/usr/local" -v 2 ${TIMEFLAG}   -Ss f77lib " " -Ss pmake "make -j10"  -Fa alg "-fpic -DPIC " -b 64 ;  if [ "x" != "x" ] ; then  if [ "x" != "xNONE" ] ; then  if [ -f  ] ; then  cp -f -f  /usr/ports/math/atlas/work/ATLAS/CONFIG/ARCHS/ ;  elif [ ! -f /usr/ports/math/atlas/work/ATLAS/CONFIG/ARCHS/ ] ; then  echo  "==> atlas-math-3.8.4_18,1 : ARCHDEF \"\" does not exist" ;  exit 1 ;  fi ;  /usr/bin/sed -i.bak -E -e  "\|[[:blank:]]*ARCH =|s|=.*|= |"  /usr/ports/math/atlas/work/ATLAS/shared/Make.inc ;  fi ;  fi ;  echo "Building shared ATLAS:" ; cd /usr/ports/math/atlas/work/ATLAS/shared ;  /usr/bin/env F77="gfortran10" FC="gfortran10" FFLAGS="-O -Wl,-rpath=/usr/local/lib/gcc10" FCFLAGS="-Wl,-rpath=/usr/local/lib/gcc10" XDG_DATA_HOME=/usr/ports/math/atlas/work  XDG_CONFIG_HOME=/usr/ports/math/atlas/work  HOME=/usr/ports/math/atlas/work PATH=/usr/ports/math/atlas/work/.bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/backyard/bin:/usr/local/tigcc/bin:/usr/local/tigcc/bin NO_PIE=yes MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES ADDR2LINE="/usr/local/bin/addr2line" AR="/usr/local/bin/ar" AS="/usr/local/bin/as" CPPFILT="/usr/local/bin/c++filt" GPROF="/usr/local/bin/gprof" LD="/usr/local/bin/ld" NM="/usr/local/bin/nm" OBJCOPY="/usr/local/bin/objcopy" OBJDUMP="/usr/local/bin/objdump" RANLIB="/usr/local/bin/ranlib" READELF="/usr/local/bin/readelf" SIZE="/usr/local/bin/size" STRINGS="/usr/local/bin/strings" PREFIX=/usr/local  LOCALBASE=/usr/local  CC="gcc10" CFLAGS="-O2 -pipe -march=westmere  -fstack-protector-strong -Wl,-rpath=/usr/local/lib/gcc10 -fno-strict-aliasing "  CPP="cpp10" CPPFLAGS=""  LDFLAGS=" -Wl,-rpath=/usr/local/lib/gcc10  -L/usr/local/lib/gcc10 -B/usr/local/bin -fstack-protector-strong -Wl,-rpath=/usr/local/lib/gcc10 -L/usr/local/lib/gcc10 " LIBS=""  CXX="g++10" CXXFLAGS="-O2 -pipe -march=westmere  -fstack-protector-strong -Wl,-rpath=/usr/local/lib/gcc10  -Wl,-rpath=/usr/local/lib/gcc10 "  MANPREFIX="/usr/local" BSD_INSTALL_PROGRAM="install  -s -m 555"  BSD_INSTALL_LIB="install  -s -m 0644"  BSD_INSTALL_SCRIPT="install  -m 555"  BSD_INSTALL_DATA="install  -m 0644"  BSD_INSTALL_MAN="install  -m 444" /usr/bin/make DESTDIR=/usr/ports/math/atlas/work/stage all
===>  Cleaning for lapack-3.10.0
===>  License BSD3CLAUSE accepted by the user
===>   lapack-3.10.0 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by lapack-3.10.0 for building
===>  Extracting for lapack-3.10.0
=> SHA256 Checksum OK for lapack-3.10.0/v3.10.0.tar.gz.
=> SHA256 Checksum OK for lapack-3.10.0/manpages.tgz.
===>  Patching for lapack-3.10.0
make[1]: make[1]: don't know how to make lapacklib. Stop

make[1]: stopped in /root
*** Error code 2

Stop.
make: stopped in /usr/ports/math/atlas
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2021-07-12 02:29:02 UTC
^Triage: Request support from Fortran team. Can you help with this Thierry?

@Reporter Can you include a full build log (as an attachment, compressed if necessary) please.
Comment 2 alt2600 2021-07-12 19:05:46 UTC
Created attachment 226407 [details]
Full Atlas Build log

See attached
Comment 3 alt2600 2021-07-14 16:08:32 UTC
(In reply to alt2600 from comment #2)
I just noticed this when I was checking status. I'm not sure why it is saying make failed in /root, I didn't start make from /root, I was in the port directory.
Comment 4 commit-hook freebsd_committer 2021-07-21 20:59:41 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=75752cd50be8de2ff11026bf3a536bcad6245a8f

commit 75752cd50be8de2ff11026bf3a536bcad6245a8f
Author:     Thierry Thomas <thierry@FreeBSD.org>
AuthorDate: 2021-07-21 20:48:14 +0000
Commit:     Thierry Thomas <thierry@FreeBSD.org>
CommitDate: 2021-07-21 20:48:14 +0000

    math/atlas: unbreak after math/lapack + various fixes

    The latest upgrade of math/lapack broke ATLAS, due to the lack of the
    post-target, but anyways it was broken in many others ways:

    - gmake is required (the fixes ifdef / #ifdef are not sufficient)

    - -fPIC is needed everywhere

    - fix EXTRA_FFLAGS

    - remove a conflict with atlas-devel.

    PR:             257122
    Reported by:    alt2600 (at) icloud.com
    MFH:            2021Q3

 math/atlas/Makefile | 29 ++++++++++++++---------------
 1 file changed, 14 insertions(+), 15 deletions(-)
Comment 5 commit-hook freebsd_committer 2021-07-21 20:59:42 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=b7eec409d5664a73725c392b83e4f2741c5ad33f

commit b7eec409d5664a73725c392b83e4f2741c5ad33f
Author:     Thierry Thomas <thierry@FreeBSD.org>
AuthorDate: 2021-07-21 20:38:05 +0000
Commit:     Thierry Thomas <thierry@FreeBSD.org>
CommitDate: 2021-07-21 20:44:31 +0000

    math/lapack: fix post-patch target for math/atlas

    When upgrading this port, I previously merged the post-patch with the
    pre-configure target, and it was a mistake.
    The patch target is used from math/atlas, and make.inc is needed.

    PR:             257122
    Reported by:    alt2600 (at) icloud.com
    MFH:            2021Q3

 math/lapack/Makefile | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Comment 6 Thierry Thomas freebsd_committer 2021-07-21 21:06:27 UTC
Indeed, my previous upgrade of math/lapack broke ATLAS: the post-patch target of lapack is required, because atlas uses the patch target of lapack and nothing more.

But this was not sufficient, and atlas was still broken in several other ways…
I hope that everything is now fixed, but let me know if you encounter other problem.

Next step: upgrade to the latest release!

Note: I have not used this port for many years, since I need OpenBLAS.
Just out of curiosity, what is your use case for this beast?
Comment 7 alt2600 2021-07-21 23:50:22 UTC
(In reply to Thierry Thomas from comment #6)


It is still considered marginally faster then OpenBLAS, and in the past the dev notes for some packages such as SciPy or maybe Numpy seem to suggest it doesn't build properly when openblas is configured to use openmpi, which is our default due to some threading issue in python, but i seem to have deleted that bookmark.

but form a site.cfg.example for SciPy I have this which seems to suggest this was an issue with Python 2.7 and no longer a problem perhaps.

# OpenBLAS
# --------
# OpenBLAS is another open source optimized implementation of BLAS and Lapack
# and can be seen as an alternative to Atlas. To build Scipy against OpenBLAS
# instead of Atlas, use this section instead of the above, adjusting as needed
# for your configuration (in the following example we installed OpenBLAS with
# ``make install PREFIX=/opt/OpenBLAS``.
# OpenBLAS is generically installed as a shared library, to force the OpenBLAS
# library linked to also be used at runtime you can utilize the
# runtime_library_dirs variable.
#
# **Warning**: OpenBLAS, by default, is built in multithreaded mode. Due to the
# way Python's multiprocessing is implemented, a multithreaded OpenBLAS can
# cause programs using both to hang as soon as a worker process is forked on
# POSIX systems (Linux, Mac).
# This is fixed in Openblas 0.2.9 for the pthread build, the OpenMP build using
# GNU openmp is as of gcc-4.9 not fixed yet.
# Python 3.4 will introduce a new feature in multiprocessing, called the
# "forkserver", which solves this problem. For older versions, make sure
# OpenBLAS is built using pthreads or use Python threads instead of
# multiprocessing.
# (This problem does not exist with multithreaded ATLAS.)
#
# https://docs.python.org/3.4/library/multiprocessing.html#contexts-and-start-methods
# https://github.com/xianyi/OpenBLAS/issues/294

 Not that our SciPy supports atlas right now.  I do not like that many ports ignore BLAS selection and force OpenBLAS for instance when the default is netlib. In reality there doesn't seem to be any reason why we cannot use all of them other then they prefix into conflicting install prefixes, whereas in other systems they are setup to use be non-conflicting. I've been using this for near 15 years now, I don't see a compelling reason to change right now. Given openmpi is broken in tree when libfabric is installed, and qmake needs to have AVX fixes to fix non-AVX architectures due to a change made in qt5-3d rebasing my system is currently impossible anyway.

I'll give the new updates a try and open new issue if it fails to install.
Comment 8 commit-hook freebsd_committer 2021-07-25 16:03:55 UTC
A commit in branch 2021Q3 references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=7e1d6c0f6c1b935413eb5ecc689a60a5d236af80

commit 7e1d6c0f6c1b935413eb5ecc689a60a5d236af80
Author:     Thierry Thomas <thierry@FreeBSD.org>
AuthorDate: 2021-07-21 20:48:14 +0000
Commit:     Thierry Thomas <thierry@FreeBSD.org>
CommitDate: 2021-07-25 16:02:35 +0000

    math/atlas: unbreak after math/lapack + various fixes

    The latest upgrade of math/lapack broke ATLAS, due to the lack of the
    post-target, but anyways it was broken in many others ways:

    - gmake is required (the fixes ifdef / #ifdef are not sufficient)

    - -fPIC is needed everywhere

    - fix EXTRA_FFLAGS

    - remove a conflict with atlas-devel.

    PR:             257122
    Reported by:    alt2600 (at) icloud.com
    MFH:            2021Q3

    (cherry picked from commit 75752cd50be8de2ff11026bf3a536bcad6245a8f)

 math/atlas/Makefile | 29 ++++++++++++++---------------
 1 file changed, 14 insertions(+), 15 deletions(-)
Comment 9 commit-hook freebsd_committer 2021-07-25 16:03:56 UTC
A commit in branch 2021Q3 references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=a36e5a3440e59bd0e2a42f16d0ca5b820be234d9

commit a36e5a3440e59bd0e2a42f16d0ca5b820be234d9
Author:     Thierry Thomas <thierry@FreeBSD.org>
AuthorDate: 2021-07-21 20:38:05 +0000
Commit:     Thierry Thomas <thierry@FreeBSD.org>
CommitDate: 2021-07-25 16:01:48 +0000

    math/lapack: fix post-patch target for math/atlas

    When upgrading this port, I previously merged the post-patch with the
    pre-configure target, and it was a mistake.
    The patch target is used from math/atlas, and make.inc is needed.

    PR:             257122
    Reported by:    alt2600 (at) icloud.com
    MFH:            2021Q3

    (cherry picked from commit b7eec409d5664a73725c392b83e4f2741c5ad33f)

 math/lapack/Makefile | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)