Bug 243497 - math/py-numpy: Fails to build with OpenBlas
Summary: math/py-numpy: Fails to build with OpenBlas
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-python (Nobody)
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2020-01-21 23:40 UTC by rsmith
Modified: 2020-06-13 22:52 UTC (History)
7 users (show)

See Also:
bugzilla: maintainer-feedback? (python)


Attachments
Updated files/patch-numpy-distutils-system_info.py (1.56 KB, text/plain)
2020-01-21 23:40 UTC, rsmith
no flags Details
numpy configure log (6.22 KB, text/plain)
2020-01-22 15:54 UTC, Robert Kruus
no flags Details
build log (11.88 KB, text/plain)
2020-01-22 18:05 UTC, rsmith
no flags Details
[patch] add openblas to list of cblas compatible libs to check (3.21 KB, patch)
2020-01-23 21:21 UTC, John Hein
no flags Details | Diff
[patch] add openblas to list of cblas compatible libs to check [v2] (2.79 KB, patch)
2020-01-23 22:18 UTC, John Hein
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description rsmith 2020-01-21 23:40:33 UTC
Created attachment 210941 [details]
Updated files/patch-numpy-distutils-system_info.py

TL;DR

* Problem: Compiling py37-numpy-1.16.5_3,1 with OpenBlas fails in numpy/distutils/system_info.py with an uncaught distutils.errors.LinkError exception.
* Solution: also catch distutils.errors.LinkError on line 1742 of numpy/distutils/system_info.py.

This is the error (only the last part shown for clarity).

    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 1740, in has_cblas
        extra_postargs=info.get('extra_link_args', []))
    File "/usr/local/lib/python3.7/distutils/ccompiler.py", line 734, in link_executable
        debug, extra_preargs, extra_postargs, None, target_lang)
    File "/usr/local/lib/python3.7/distutils/unixccompiler.py", line 206, in link
        raise LinkError(msg)
    distutils.errors.LinkError: Command "cc /tmp/tmpm3gnb0ev/tmp/tmpm3gnb0ev/source.o -L/usr/local/lib -lblas -o /tmp/tmpm3gnb0ev/a.out" failed with exit status 1
    *** Error code 1

    Stop.
    make[1]: stopped in /usr/ports/math/py-numpy
    *** Error code 1

    Stop.
    make: stopped in /usr/ports/math/py-numpy

This is the relevant code:

        try:
            with open(src, 'wt') as f:
                f.write(s)

            try:
                # check we can compile (find headers)
                obj = c.compile([src], output_dir=tmpdir,
                                include_dirs=self.get_include_dirs())

                # check we can link (find library)
                # some systems have separate cblas and blas libs. First
                # check for cblas lib, and if not present check for blas lib.
                try:
                    c.link_executable(obj, os.path.join(tmpdir, "a.out"),
                                      libraries=["cblas"],
                                      library_dirs=info['library_dirs'],
                                      extra_postargs=info.get('extra_link_args', []))
                    res = "cblas"
                except distutils.ccompiler.LinkError:
                    c.link_executable(obj, os.path.join(tmpdir, "a.out"),
                                      libraries=["blas"],
                                      library_dirs=info['library_dirs'],
                                      extra_postargs=info.get('extra_link_args', []))
                    res = "blas"
            except distutils.ccompiler.CompileError:
                res = None
        finally:
            shutil.rmtree(tmpdir)
        return res

If linking with “cblas” fails, it tries again by linking with “blas”.
However, the last “except” only handles a compilation error but not a linking error.

When we change:

            except distutils.ccompiler.CompileError:

to

            except (distutils.ccompiler.CompileError, distutils.ccompiler.LinkError):

the problem is fixed.
Comment 1 Steve Wills freebsd_committer 2020-01-22 02:03:14 UTC
Can you provide the full build log? I'm trying to understand the issue better.
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2020-01-22 03:08:41 UTC
Thank you for the report and proposed patch. Could you create an issue/pr upstream as well for this please
Comment 3 Steve Wills freebsd_committer 2020-01-22 12:15:52 UTC
I wasn't able to reproduce this, so I'm trying to understand what's different between your build and mine. The full build log would help with that. Are you using the latest version from ports with the updated openblas 0.3.7?
Comment 4 Robert Kruus 2020-01-22 15:54:17 UTC
Created attachment 210969 [details]
numpy configure log
Comment 5 Robert Kruus 2020-01-22 15:56:56 UTC
My build fails at configure -- seems to not find the libopenblas for me, see attached make configure log.
FreeBSD flash.gos.ca 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC  amd64
Comment 6 Steve Wills freebsd_committer 2020-01-22 16:20:34 UTC
(In reply to Robert Kruus from comment #5)
What version of openblas do you have installed? Do you have cblas installed, and if so what version? (If you have cblas, remove it.) A full `pkg info` output might be helpful.
Comment 7 Robert Kruus 2020-01-22 16:48:36 UTC
openblas-0.3.7,1
Just update today.  I notice there is no libopenblasp.so in this version, my numpy is now linked to:
/usr/local/lib/compat/pkg/libopenblasp.so.0

No other blas or lapack installed.

pkg info openblas
openblas-0.3.7,1
Name           : openblas
Version        : 0.3.7,1
Installed on   : Wed Jan 22 09:02:24 2020 CST
Origin         : math/openblas
Architecture   : FreeBSD:12:amd64
Prefix         : /usr/local
Categories     : math
Licenses       : BSD3CLAUSE
Maintainer     : phd_kimberlite@yahoo.co.jp
WWW            : https://github.com/xianyi/OpenBLAS
Comment        : Optimized BLAS library based on GotoBLAS2
Options        :
        AVX            : on
        AVX2           : on
        DYNAMIC_ARCH   : off
        INTERFACE64    : off
        OPENMP         : off
Shared Libs required:
        libquadmath.so.0
        libgfortran.so.5
Annotations    :
        FreeBSD_version: 1201000
Flat size      : 45.1MiB
Description    :
OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version.
OpenBLAS is an open source project supported by
Lab of Parallel Software and Computational Science, ISCAS.

NOTE: If you want to specify your CPU microarchitecture manually,
please use TARGET_CPU_ARCH knob, e.g., "make TARGET_CPU_ARCH=NEHALEM".
This value is set TARGET build flag.

WWW: https://github.com/xianyi/OpenBLAS
Comment 8 Steve Wills freebsd_committer 2020-01-22 17:51:40 UTC
(In reply to Robert Kruus from comment #7)
Can you show my the initial build failure message?
Comment 9 rsmith 2020-01-22 18:05:23 UTC
Created attachment 210971 [details]
build log

Please find attached the build log.

Below is the info about the OpenBlas I'm building against:

> pkg info openblas-0.3.7,1
openblas-0.3.7,1
Name           : openblas
Version        : 0.3.7,1
Installed on   : Tue Jan 21 23:29:48 2020 CET
Origin         : math/openblas
Architecture   : FreeBSD:12:amd64
Prefix         : /usr/local
Categories     : math
Licenses       : BSD3CLAUSE
Maintainer     : phd_kimberlite@yahoo.co.jp
WWW            : https://github.com/xianyi/OpenBLAS
Comment        : Optimized BLAS library based on GotoBLAS2
Options        :
	AVX            : on
	AVX2           : off
	DYNAMIC_ARCH   : off
	INTERFACE64    : off
	OPENMP         : on
Shared Libs required:
	libquadmath.so.0
	libgfortran.so.5
Annotations    :
	FreeBSD_version: 1201500
Comment 10 rsmith 2020-01-22 18:06:48 UTC
(In reply to Steve Wills from comment #3)

I've added my build log as an attachment. I'm building against OpenBlas 0.3.7.
Comment 11 Steve Wills freebsd_committer 2020-01-22 19:10:35 UTC
(In reply to rsmith from comment #10)
What is in your /usr/local/lib/compat/pkg/? Does it change things if you were to say temporarily rename that directory during the rebuild?
Comment 12 rsmith 2020-01-22 19:15:10 UTC
(In reply to Steve Wills from comment #11)

I don't have a /usr/local/lib/compat directory.
Comment 13 Steve Wills freebsd_committer 2020-01-22 19:17:45 UTC
(In reply to rsmith from comment #12)
Sorry, I think I was confused since earlier you said:

"my numpy is now linked to:
/usr/local/lib/compat/pkg/libopenblasp.so.0"
Comment 14 rsmith 2020-01-22 19:39:04 UTC
(In reply to Steve Wills from comment #13)

That was Robert Kruus in comment #7, not me. :-)
Comment 15 Steve Wills freebsd_committer 2020-01-22 19:48:41 UTC
(In reply to rsmith from comment #14)
Ah, ok. Can you tell me if you have cblas installed or give me a full pkg listing? Apologies if you did already and I missed it.
Comment 16 rsmith 2020-01-22 20:13:37 UTC
(In reply to Steve Wills from comment #15)

Here are all my blas packages:

> pkg info -x blas
blas-3.5.0_6
openblas-0.3.7,1

There is no cblas, which explains why the error occurs; the system_info.py script tries to link with cblas (under two names) and doesn't succeed. This raises the aforementioned link exception which isn't caught -> boom!
Comment 17 rsmith 2020-01-22 20:28:23 UTC
(In reply to Kubilay Kocak from comment #2)

Upstream issue: https://github.com/numpy/numpy/issues/15389
Comment 18 Robert Kruus 2020-01-22 20:29:57 UTC
So py-numpy installed/upgraded fine if I did pkg delete suitesparse and reinstalled it.
(see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243524 as to why I did that).

It rebuilt the following (even after I had installed suitesparse)
        Re-installation of gcc9-9.2.0_1
        Re-installation of python37-3.7.6
        Re-installation of openblas-0.3.7,1
        Re-installation of suitesparse-5.4.0_4
        Installation of math/py-numpy (py37-numpy-1.16.5_3,1)
Comment 19 Robert Kruus 2020-01-22 20:30:58 UTC
BTW:
pkg info -x blas
openblas-0.3.7,1
Comment 20 rsmith 2020-01-22 20:59:50 UTC
(In reply to Steve Wills from comment #1)

The fact that I also have blas-3.5.0_6 installed (for Calculix) has a bearing on this issue.

> pkg list blas-3.5.0_6
/usr/local/lib/libblas.a
/usr/local/lib/libblas.so
/usr/local/lib/libblas.so.2
/usr/local/lib/libblas_pic.a
/usr/local/share/licenses/blas-3.5.0_6/BSD3CLAUSE
/usr/local/share/licenses/blas-3.5.0_6/LICENSE
/usr/local/share/licenses/blas-3.5.0_6/catalog.mk

So when system_info.py is looking for cblas, it tries to compile and link a program with both libcblas and libblas, see my original description.

Since I don't have cblas installed, the first compile fails.
The second compile *finds* libblas, (from blas-3.5.0_6) but it doesn't contain the symbol (cblas_ddot) referenced in the test program, this produces a linker error rather than a compilation error.
Comment 21 Steve Wills freebsd_committer 2020-01-22 21:30:45 UTC
(In reply to rsmith from comment #20)
I see now. I was able to reproduce with math/blas (blas-3.5.0_6) installed. The patch does seem to avoid the issue. I'll commit it in a bit.
Comment 22 Steve Wills freebsd_committer 2020-01-22 21:32:40 UTC
(In reply to Robert Kruus from comment #19)
Robert, is your issue solved now that you've reinstalled suitesparse?
Comment 23 Robert Kruus 2020-01-22 22:03:03 UTC
Yes, but the key was to delete suitesparse first, and any libs in /usr/local/compat... from the old openblas.  Not deleting suitesparse kept references to libopenblasp.so.0 (from the older version of openblas, as the newest version only installs libopenblas.so.0).  And since my py-numpy has:
Options        :
        ATLAS          : off
        DOCS           : off
        NETLIB         : off
        OPENBLAS       : on
        SUITESPARSE    : on
(default I believe) I am guessing that cascaded to the configure stage of py-numpy.
A different issue for sure.
Comment 24 lumiwa 2020-01-23 00:21:26 UTC
I have the same problem as rsmith@xs4all.nl but before I had math/cblas installed and I didn't have any problem but the last update I cannot install cblas again because:
Installing for cblas-1.0_12
===>  Checking if cblas is already installed
===>   Registering installation for cblas-1.0_12
Installing cblas-1.0_12...
pkg-static: cblas-1.0_12 conflicts with openblas-0.3.7,1 (installs files into the same place).  Problematic file: /usr/local/include/cblas.h
*** Error code 70

Stop.
make[1]: stopped in /usr/ports/math/cblas
*** Error code 1

Thank you.
Comment 25 rsmith 2020-01-23 20:32:23 UTC
(In reply to Kubilay Kocak from comment #2)

Upstream pull request created: https://github.com/numpy/numpy/pull/15406

BTW, the numpy master branch (which will become numpy 1.19) doesn't have this bug. ATM I couldn't build the master branch because our Cython is slightly too old; numpy master requires 0.29.14 and we have 0.29.13.

Is there any particular reason we're still on numpy 1.16? Otherwise I might try my hand at updating cython and numpy.
Comment 26 John Hein 2020-01-23 20:39:58 UTC
(In reply to Steve Wills from comment #21)
Wouldn't it be better to update the check to look for -lopenblas (along with the existing check for -lblas & -lcblas).  Catching the linker exception is good, but isn't openblas supposed to be compatible(ish) with the reference cblas implementation?
Comment 27 John Hein 2020-01-23 21:21:38 UTC
Created attachment 210997 [details]
[patch] add openblas to list of cblas compatible libs to check

(In reply to John Hein from comment #26)

Here's a patch to add openblas to the list of cblas compatible libs (also includes Roland's patch to catch the linker error in the outer catch).

It looks like newer versions of upstream don't have the same 'have_cblas' test, and does search all the configured libraries (and will catch the linker error).  So we probably won't need to carry on either patch after updating numpy.

Note: numpy 1.16 is the last to support python2.7, I think.  If we need the python2.7 version of numpy around a bit longer (friendly to the many users still working on updating to python3), maybe a repocopy to math/py-numpy-1.16 is warranted.  If so, I wouldn't spend any time allowing them (old and new numpy) to be concurrently installed since it would just be a stop gap until python2.7 is officially out of the tree.
Comment 28 John Hein 2020-01-23 21:31:09 UTC
(In reply to John Hein from comment #27)
This patch (the patch with the openblas cblas_ddot test vs the patch without) does have a real effect on the build.  _multiarray_umath.so is built differently (I only looked at ldd output).
Comment 29 commit-hook freebsd_committer 2020-01-23 21:32:19 UTC
A commit references this bug:

Author: swills
Date: Thu Jan 23 21:31:23 UTC 2020
New revision: 523944
URL: https://svnweb.freebsd.org/changeset/ports/523944

Log:
  math/py-numpy: fix build when math/blas is installed

  PR:		243497
  Submitted by:	rsmith@xs4all.nl

Changes:
  head/math/py-numpy/files/patch-numpy-distutils-system_info.py
Comment 30 Steve Wills freebsd_committer 2020-01-23 21:36:44 UTC
(In reply to John Hein from comment #28)
Ah, sorry, didn't see your comments until after I committed the patch. If you think it's more correct, update your patch and I'll test and commit.
Comment 31 Steve Wills freebsd_committer 2020-01-23 22:03:18 UTC
(In reply to rsmith from comment #25)
I think the only reason is no one has done it, please feel free to have a go at updating cython and numpy, that would be great.
Comment 32 John Hein 2020-01-23 22:18:40 UTC
Created attachment 211000 [details]
[patch] add openblas to list of cblas compatible libs to check [v2]

(In reply to Steve Wills from comment #30)

Here's a refreshed patch with the openblas test after your commit.
I also reverted Roland's part of the patch since the linker error will now never get to the outer exception (so there's no need to try to catch it).
Comment 33 rsmith 2020-01-23 23:28:02 UTC
(In reply to Steve Wills from comment #31)

The point that John Hein raises is a valid one; Numpy 1.16 is indeed the last release to support Python 2.7.

Looking at the numpy release notes, I'm not sure if there are significant enough improvements between 1.16 and 1.18 to warrant leaving Python 2 users without numpy. Although the support for OpenBlas with 8-byte integers might appeal to those with huge datasets.

Additionally, there doesn't seem to be a huge uproar about the ports tree not being on the latest stable numpy version. I only brought it up because in principle it is good to stay current with important libraries like numpy.

So, maybe we should keep numpy at 1.16 until Python 2.7 expires at the end of the year?
Comment 34 lumiwa 2020-01-23 23:39:22 UTC
I did install py-numpy with the last update without problem on FreeBSD 12.1-RELEASE-p1 and reinstall openblas and suitesparse but when I run:
 
pkg check -d -n -a
Checking all packages: 100%
py27-numpy is missing a required shared library: libopenblas.so
py37-numpy is missing a required shared library: libopenblas.so
py37-scipy is missing a required shared library: libopenblas.so.0
py37-scipy is missing a required shared library: libcblas.so.2
suitesparse is missing a required shared library: libopenblas.so

pkg list oenblas show:
...
/usr/local/lib/libopenblas.a
/usr/local/lib/libopenblas.so
/usr/local/lib/libopenblas.so.0
...

Thank you.
Comment 35 commit-hook freebsd_committer 2020-01-31 14:09:11 UTC
A commit references this bug:

Author: swills
Date: Fri Jan 31 14:08:51 UTC 2020
New revision: 524711
URL: https://svnweb.freebsd.org/changeset/ports/524711

Log:
  math/py-numpy: improve patch to fix build with OpenBlas

  This patch adds openblas to the list of cblas compatible libs instead

  PR:		243497
  Submitted by:	John Hein <jcfyecrayz@liamekaens.com>

Changes:
  head/math/py-numpy/files/patch-numpy-distutils-system_info.py
Comment 36 Steve Wills freebsd_committer 2020-01-31 23:33:56 UTC
Updated patch committed, thanks!
Comment 37 Harry NEWTON 2020-06-02 07:23:39 UTC
I'm not sure this is closed: I cannot build numpy (and therefore neither scipy) as it now depends on openblas and cblas which install files in the same place

pkg-static: openblas-0.3.9_2,1 conflicts with cblas-1.0_12 (installs files into the same place).  Problematic file: /usr/local/include/cblas.h

There's a related bug: 244296.
Comment 38 dave 2020-06-13 22:52:54 UTC
I have the same issue:

pkg-static: openblas-0.3.7_2,1 conflicts with cblas-1.0_12 (installs files into the same place).  Problematic file: /usr/local/include/cblas.h