Bug 257400 - science/py-scipy Add OPTION for Atlas blas
Summary: science/py-scipy Add OPTION for Atlas blas
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-python (Nobody)
URL:
Keywords: feature, needs-qa
Depends on:
Blocks:
 
Reported: 2021-07-25 06:13 UTC by alt2600
Modified: 2021-08-02 01:16 UTC (History)
2 users (show)

See Also:
koobs: maintainer-feedback+
koobs: maintainer-feedback? (python)


Attachments
git-diff-scipy-add-atlas (2.46 KB, patch)
2021-07-25 06:13 UTC, alt2600
no flags Details | Diff
files/site.cfg (309 bytes, text/plain)
2021-07-25 06:14 UTC, alt2600
no flags Details
Makefile (2.17 KB, text/plain)
2021-07-25 06:15 UTC, alt2600
no flags Details
patch-Makefile-scipy-add-atlas (1.79 KB, patch)
2021-07-25 06:19 UTC, alt2600
no flags Details | Diff
git-diff-scipy-add-atlas (2.46 KB, patch)
2021-07-26 14:13 UTC, alt2600
no flags Details | Diff
git-diff-scipy-add-atlas-rev1 (2.50 KB, patch)
2021-07-29 22:20 UTC, alt2600
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description alt2600 2021-07-25 06:13:24 UTC
Created attachment 226666 [details]
git-diff-scipy-add-atlas

Not sure I made the git diff correctly, getting used to this, and hacking as I go, so I will also submit all the individual files. I took the basic Makefile framework from py-numpy and tweaked it so it would be appropriate for py-scipy. I tried to do netlib as well, but while it seemed to configure to use the right libraries the final package had a dependency on atlas-math. I didn't dig to hard to find why, but I suspect it is because the build notes for scipy say they will default built against Atlas if found as presumably the preferred blas. As I doubt anyone would be using netlib with this package, I opted to limit things to Atlas and OpenBLAS. The issue is I have no way to test the OpenBLAS is still working ok, but I can confirm the Atlas build is fine, and am using it to build towards qgis ultimately. I believe the OpenBLAS case should work as it is the same mechanism of numpy, which seems to be the master library to scipy if I'm reading things right. Would be great to get this patch to allow Atlas as well.


### ATLAS Configure abbreviated tail

creating /tmp/tmpcc9xcnvz/tmp
creating /tmp/tmpcc9xcnvz/tmp/tmpcc9xcnvz
compile options: '-MMD -MF /tmp/tmpcc9xcnvz/file.c.d -c'
cc: /tmp/tmpcc9xcnvz/file.c
  libraries tatlas not found in ['/usr/lib', '/usr/local/lib', '/usr/local/lib/gcc10/gcc/x86_64-portbld-freebsd12.2/10.3.0/../../../']
Library tatlas was not found. Ignoring
  FOUND:
    include_dirs = ['/usr/local/include']
    language = f77
    libraries = ['alapack', 'ptf77blas', 'ptcblas']
    library_dirs = ['/usr/local/lib']
    define_macros = [('ATLAS_INFO', '"None"')]

accelerate_info:
  NOT AVAILABLE

  FOUND:
    include_dirs = ['/usr/local/include']
    language = f77
    libraries = ['alapack', 'ptf77blas', 'ptcblas']
    library_dirs = ['/usr/local/lib']
    define_macros = [('ATLAS_INFO', '"None"')]

blas_opt_info:
blas_mkl_info:
customize UnixCCompiler
C compiler: cc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -march=westmere -fstack-protector-strong -fno-strict-aliasing -O2 -pipe -march=westmere -fstack-protector-strong -fno-strict-aliasing -fPIC

creating /tmp/tmpqzo50prz/tmp
creating /tmp/tmpqzo50prz/tmp/tmpqzo50prz
compile options: '-MMD -MF /tmp/tmpqzo50prz/file.c.d -c'
cc: /tmp/tmpqzo50prz/file.c
  libraries mkl_rt not found in ['/usr/lib', '/usr/local/lib', '/usr/local/lib/gcc10/gcc/x86_64-portbld-freebsd12.2/10.3.0/../../../']
  NOT AVAILABLE

blis_info:
customize UnixCCompiler
C compiler: cc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -march=westmere -fstack-protector-strong -fno-strict-aliasing -O2 -pipe -march=westmere -fstack-protector-strong -fno-strict-aliasing -fPIC

creating /tmp/tmp32nyh_rz/tmp
creating /tmp/tmp32nyh_rz/tmp/tmp32nyh_rz
compile options: '-MMD -MF /tmp/tmp32nyh_rz/file.c.d -c'
cc: /tmp/tmp32nyh_rz/file.c
  libraries blis not found in ['/usr/lib', '/usr/local/lib', '/usr/local/lib/gcc10/gcc/x86_64-portbld-freebsd12.2/10.3.0/../../../']
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
customize UnixCCompiler
C compiler: cc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -march=westmere -fstack-protector-strong -fno-strict-aliasing -O2 -pipe -march=westmere -fstack-protector-strong -fno-strict-aliasing -fPIC

creating /tmp/tmp1llo3grv/tmp
creating /tmp/tmp1llo3grv/tmp/tmp1llo3grv
compile options: '-MMD -MF /tmp/tmp1llo3grv/file.c.d -c'
cc: /tmp/tmp1llo3grv/file.c
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
customize UnixCCompiler
C compiler: cc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -march=westmere -fstack-protector-strong -fno-strict-aliasing -O2 -pipe -march=westmere -fstack-protector-strong -fno-strict-aliasing -fPIC

creating /tmp/tmp3uzvn3cw/tmp
creating /tmp/tmp3uzvn3cw/tmp/tmp3uzvn3cw
compile options: '-MMD -MF /tmp/tmp3uzvn3cw/file.c.d -c'
cc: /tmp/tmp3uzvn3cw/file.c
  libraries tatlas not found in ['/usr/lib', '/usr/local/lib', '/usr/local/lib/gcc10/gcc/x86_64-portbld-freebsd12.2/10.3.0/../../../']
Library tatlas was not found. Ignoring
  FOUND:
    include_dirs = ['/usr/local/include']
    language = c
    define_macros = [('HAVE_CBLAS', None), ('ATLAS_INFO', '"None"')]
    libraries = ['ptf77blas', 'ptcblas']
    library_dirs = ['/usr/local/lib']

  FOUND:
    include_dirs = ['/usr/local/include']
    language = c
    define_macros = [('HAVE_CBLAS', None), ('ATLAS_INFO', '"None"')]
    libraries = ['ptf77blas', 'ptcblas']
    library_dirs = ['/usr/local/lib']

non-existing path in 'scipy/signal/windows': 'tests'
non-existing path in 'scipy/sparse': 'sparsetools/sparsetools_impl.h'
non-existing path in 'scipy/sparse': 'sparsetools/bsr_impl.h'
non-existing path in 'scipy/sparse': 'sparsetools/csc_impl.h'
non-existing path in 'scipy/sparse': 'sparsetools/csr_impl.h'
non-existing path in 'scipy/sparse': 'sparsetools/other_impl.h'
[makenpz] scipy/special/tests/data/boost.npz not rebuilt
[makenpz] scipy/special/tests/data/gsl.npz not rebuilt
[makenpz] scipy/special/tests/data/local.npz not rebuilt
running config

## pkg info -dr py38-scipy
/usr/ports/science/py-scipy|$} pkg info -dr py38-scipy
py38-scipy-1.6.0
Depends on     :
        py38-numpy-1.16.6_2,1
        atlas-math-3.8.4_18,1
        python38-3.8.11
        gcc10-10.3.0
        py38-setuptools-57.0.0
Comment 1 alt2600 2021-07-25 06:14:33 UTC
Created attachment 226667 [details]
files/site.cfg

new files/site.cfg adapted from numpy
Comment 2 alt2600 2021-07-25 06:15:29 UTC
Created attachment 226668 [details]
Makefile

complete py-scipy Makefile for Atlas/OpenBLAS selection I left OpenBLAS the default
Comment 3 alt2600 2021-07-25 06:19:59 UTC
Created attachment 226669 [details]
patch-Makefile-scipy-add-atlas

standalone Makefile patch too add atlas and hopefully not break openblas. Just like files/site.cfg in case I didn't prepare the git diff correctly or if its easier to use these instead of that
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2021-07-26 01:23:32 UTC
Thank you for your patch. Can you provide your changes as a single diff against the port? You can use `git diff > patch`.

For updating/patching WRKSRC (upstream sources) files, you can use the `make makepatch method` outlined here:

  https://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/slow-patch.html
Comment 5 alt2600 2021-07-26 05:05:12 UTC
(In reply to Kubilay Kocak from comment #4)

that is the first attachment.
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2021-07-26 06:47:18 UTC
(In reply to alt2600 from comment #5)

If the other patches duplicate the one you refer to, just separated, please obsolete them, otherwise include their changes into a single diff

Thanks!
Comment 7 alt2600 2021-07-26 14:13:25 UTC
Created attachment 226716 [details]
git-diff-scipy-add-atlas

obsoleting redundant attachments
Comment 8 alt2600 2021-07-26 22:19:02 UTC
(In reply to Kubilay Kocak from comment #6)

sorry, if this notified already, not sure if it only sends email when its a reply or if just posting does it. not trying to be bothersome, just didn't want you waiting on me doing this earlier. I plan to stand up a bhyve to make testing easier when options are system breaking for my setup, ie openblas. in case there are issues with the patch and openblas.
Comment 9 alt2600 2021-07-29 22:20:53 UTC
Created attachment 226789 [details]
git-diff-scipy-add-atlas-rev1

confirmed under openblas based bhyve to configure and build correctly. Noticed under PR 251718 and from the configure this wants both openblas and blas so I added back a LIB_DEPENDS for the OPENBLAS option to include it, but otherwise the uses blaslapack:openblas caused no issue. not sure if in the end this port wants cblas or how applicable that is with openblas but I just wanted to confirm no breakage, not extend features. This patch will allow ATLAS or OpenBLAS use, maintaining OpenBLAS as the default. I consider this patch to now be tested and complete.

it should work for Netlib case, but if Atlas is installed it will bring in an Atlas dependency, despite possibly not linking to it in reality. If someone wants to get netlib I'm sure they can explore the syntax further. I can confirm it appears to configure for it, build, and install, but still shows an Atlas pkg dependency once installed so it remains excluded. see math/py-numpy if someone really wanted to take the time for netlib. 


### make configure tail excerpt
creating /tmp/tmpqmtazvj7/tmp
creating /tmp/tmpqmtazvj7/tmp/tmpqmtazvj7
compile options: '-MMD -MF /tmp/tmpqmtazvj7/file.c.d -c'
cc: /tmp/tmpqmtazvj7/file.c
  FOUND:
    libraries = ['openblas', 'blas']
    library_dirs = ['/usr/local/lib']
    include_dirs = ['/usr/include', '/usr/local/include']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

  FOUND:
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    libraries = ['openblas', 'blas']
    library_dirs = ['/usr/local/lib']
    include_dirs = ['/usr/include', '/usr/local/include']
    language = c

non-existing path in 'scipy/signal/windows': 'tests'
non-existing path in 'scipy/sparse': 'sparsetools/sparsetools_impl.h'
non-existing path in 'scipy/sparse': 'sparsetools/bsr_impl.h'
non-existing path in 'scipy/sparse': 'sparsetools/csc_impl.h'
non-existing path in 'scipy/sparse': 'sparsetools/csr_impl.h'
non-existing path in 'scipy/sparse': 'sparsetools/other_impl.h'
[makenpz] scipy/special/tests/data/boost.npz not rebuilt
[makenpz] scipy/special/tests/data/gsl.npz not rebuilt
[makenpz] scipy/special/tests/data/local.npz not rebuilt
running config



/usr/ports/science/py-scipy $ sudo make install clean
Password:
===>  Installing for py38-scipy-1.6.0
===>  Checking if py38-scipy is already installed
===>   Registering installation for py38-scipy-1.6.0
Installing py38-scipy-1.6.0...


/usr/ports/science/py-scipy $ pkg info -dr py38-scipy
py38-scipy-1.6.0
Depends on     :
        py38-numpy-1.16.6_2,1
        openblas-0.3.16,1
        blas-3.10.0
        python38-3.8.11
        gcc10-10.3.0
        py38-setuptools-57.0.0