Bug 203638
| Summary: | math/py-numpy: Undefined symbol "cblas_sgemm" | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Philip Homburg <pch-freebsd-bugs-1> | ||||||
| Component: | Individual Port(s) | Assignee: | freebsd-python (Nobody) <python> | ||||||
| Status: | Closed FIXED | ||||||||
| Severity: | Affects Only Me | CC: | igor.arabesc.pavlov, jcfyecrayz, olivierd, python, russo, thierry, xmj | ||||||
| Priority: | --- | Keywords: | needs-patch, needs-qa, patch | ||||||
| Version: | Latest | Flags: | bugzilla:
maintainer-feedback?
(python) |
||||||
| Hardware: | Any | ||||||||
| OS: | Any | ||||||||
| Attachments: |
|
||||||||
I have the same bug. (In reply to Philip Homburg from comment #0) I can't reproduce this error (tested with NumPy 1.10.0 and 1.10.1). This problem appears when NumPy is launched through another application, or in "standalone"? Python 2.7.9 (default, Mar 27 2015, 09:40:11) [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> import numpy.core.multiarray >>> I use default option (NETLIB), and I get no error. make showconfig ===> The following configuration options are available for py27-numpy-1.10.1,1: DOCS=off: Build and/or install documentation SUITESPARSE=on: Use AMD and UMFPACK in SuiteSparse TESTS=off: Install test suite requirements ====> Options available for the single BLASLIB: you have to select exactly one of them ATLAS=off: ATLAS blas implementation NETLIB=on: Blas implementation from Netlib OPENBLAS=off: OpenBLAS blas implementation ===> Use 'make config' to modify these settings I add also thierry@ because he's maintainer of Mk/Uses/blaslapack.mk, perhaps he knows difference between math/blas and math/cblas. Created attachment 161975 [details]
typescript of numpy build and resulting error when importing in python
Hi,
It fails already just importing in python. I added a typescript of the build and the resulting errors. Hopefully you can spot what is different.
I can reproduce this by building math/suitesparse with NETLIB and math/py-numpy with NETLIB. Then "python -c 'import numpy'" gives the same undefined symbol error as the OP.
/usr/local/lib/python2.7/site-packages/numpy/core/multiarray.so:
libblas.so.2 => /usr/local/lib/libblas.so.2 (0x801781000)
libm.so.5 => /lib/libm.so.5 (0x8019d3000)
libpython2.7.so.1 => /usr/local/lib/libpython2.7.so.1 (0x801bfc000)
libc.so.7 => /lib/libc.so.7 (0x800821000)
libgfortran.so.3 => /usr/local/lib/gcc48/libgfortran.so.3 (0x801fb7000)
libgcc_s.so.1 => /usr/local/lib/gcc48/libgcc_s.so.1 (0x8022ce000)
libquadmath.so.0 => /usr/local/lib/gcc48/libquadmath.so.0 (0x8024e4000)
libthr.so.3 => /lib/libthr.so.3 (0x80271f000)
libintl.so.8 => /usr/local/lib/libintl.so.8 (0x802943000)
libutil.so.9 => /lib/libutil.so.9 (0x802b4e000)
numpy options in this case are:
OPTIONS_FILE_SET+=DOCS
OPTIONS_FILE_SET+=SUITESPARSE
OPTIONS_FILE_UNSET+=TESTS
OPTIONS_FILE_UNSET+=ATLAS
OPTIONS_FILE_SET+=NETLIB
OPTIONS_FILE_UNSET+=OPENBLAS
I also removed the openblas package before building both suitesparse and numpy - I believe openblas is detected by numpy configure if it just exists. And that is enough to avoid the problem because the numpy configure explicitly turns off HAVE_CBLAS if it detects openblas.
If I install openblas again (default options - all unset) and rebuild numpy with the same options (NETLIB set, OPENBLAS unset) and no other changes from the above failure case, I do NOT get the undefined symbol.
Here's the ldd output in that case:
% ldd /usr/local/lib/python2.7/site-packages/numpy/core/multiarray.so
/usr/local/lib/python2.7/site-packages/numpy/core/multiarray.so:
libopenblas.so => /usr/local/lib/libopenblas.so (0x801800000)
libm.so.5 => /lib/libm.so.5 (0x802575000)
libpython2.7.so.1 => /usr/local/lib/libpython2.7.so.1 (0x80279e000)
libc.so.7 => /lib/libc.so.7 (0x800821000)
libgfortran.so.3 => /usr/local/lib/gcc48/libgfortran.so.3 (0x802b59000)
libquadmath.so.0 => /usr/local/lib/gcc48/libquadmath.so.0 (0x802e70000)
libthr.so.3 => /lib/libthr.so.3 (0x8030ab000)
libintl.so.8 => /usr/local/lib/libintl.so.8 (0x8032cf000)
libutil.so.9 => /lib/libutil.so.9 (0x8034da000)
libgcc_s.so.1 => /usr/local/lib/gcc48/libgcc_s.so.1 (0x8036ec000)
From this, it seems math/py-numpy + blaslapack:netlib (without openblas installed at numpy build time) do not currently play well together.
I don't have a suggested fix for this at the moment.
Seems to me that the HAVE_CBLAS detection is not working for numpy. I know very little about configuring numpy, but perhaps there's an appropriate fix to get the numpy setup machinery to more reliably be able to determine if HAVE_CBLAS should be on or not. Seems like an upstream problem to me right now. Workarounds for the port are perhaps: - force HAVE_CBLAS off when building with NETLIB option (much like numpy forces it off when it detects openblas) - depend on a library that has the cblas* functions (math/cblas?) and encourage HAVE_CBLAS to be on If I make this change to the numpy Makefile:
Index: Makefile
===================================================================
--- Makefile (revision 399233)
+++ Makefile (working copy)
@@ -71,7 +71,7 @@
pre-configure-NETLIB-on:
@${REINPLACE_CMD} -e "s|%%BLASNAME%%|atlas|" \
-e "s|%%LIBRARIES%%|atlas_libs|" \
- -e "s|%%BLASLIBS%%|blas|" \
+ -e "s|%%BLASLIBS%%|blas, cblas|" \
-e "s|%%LAPACKLIBS%%|lapack|" \
${WRKSRC}/site.cfg
(pardon the inline rather than attached patch, but it's simple enough)
And then I install math/cblas before building math/py-numpy with NETLIB, then the import does not trigger the undefined symbol problem (and ldd(1) shows multiarray.so linked with libcblas).
Also, if I leave the same change in the Makefile and uninstall cblas, then the multiarray.so that is built is not link with either libblas or libcblas. How useful it is, I don't know (maybe just doesn't perform as well???). That is a totally untested guess.
But it seems to me that if we specify that it should use blas & cblas libs and can't find one or both, it should fail to build. But it seems numpy quietly builds itself without support for the libs specified in the configuration.
Hmm... well, maybe not completely quiet (but certainly not fatal to the build):
/usr/ports/math/py-numpy/work/numpy-1.10.0/numpy/distutils/system_info.py:1651: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.__doc__)
If the above Makefile change is used, it should also probably add math/cblas to LIB_DEPENDS.
I have had the same problem, and do have suitesparse and numpy both set to use netlib. The suggestion in comment 6 to install cblas and edit Makefile did work for me for the time being. Upstream has clearly introduced a problem, though, since HAVE_CBLAS isn't actually used to prevent "cblasfuncs" from being linked in, and that's where the cblas_sgemm call is being made. Looks like they're actually requiring cblas, not letting it be an optional linkage. In the meantime, perhaps forcing cblas install and linkage in the FreeBSD package is the right thing to do? The patch in comment 6 fixes it for at least me and the tester in comment 7. It'd be good to hear from the reporter... Philip? (In reply to John Hein from comment #8) Patches in attachments please John :) Created attachment 163456 [details] [patch] fix build when openblas installed but NETLIB option selected Patch attached. I don't have time to poke upstream about the issues raised in comment 5, comment 6 & comment 7. But this should fix the build in ports. It'd be great if someone could shepherd this issue upstream. portlint: 1 warning Q/A tests: pass Yes, adding cblas solves the problem. (In reply to Olivier Duchateau from comment #2) Reproduced with py27-numpy-1.10.1,1, in case that helps. A commit references this bug: Author: olivierd Date: Fri Jan 1 17:34:41 UTC 2016 New revision: 405020 URL: https://svnweb.freebsd.org/changeset/ports/405020 Log: - Update to 1.10.2 - Reintroduce DOCVERSION variable, PDF documentation is not synchronized with release - Add missing dependency [1] for NETLIB option PR: 203638 [1] Submitted by: John Hein [1] Changes: head/math/py-numpy/Makefile head/math/py-numpy/distinfo |
numpy version py27-numpy-1.10.0,1 on FreeBSD 10.2. Configured with NETLIB as the BLAS library >>> import numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/site-packages/numpy/__init__.py", line 180, in <module> from . import add_newdocs File "/usr/local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module> from numpy.lib import add_newdoc File "/usr/local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module> from .type_check import * File "/usr/local/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module> import numpy.core.numeric as _nx File "/usr/local/lib/python2.7/site-packages/numpy/core/__init__.py", line 14, in <module> from . import multiarray ImportError: /usr/local/lib/python2.7/site-packages/numpy/core/multiarray.so: Undefined symbol "cblas_sgemm" The cause seems to be that numpy links with blas, but uses symbols from cblas. Getting numpy to link with cblas solves the problem.