Created attachment 173018 [details] svn diff to improve math/R Proposed changes: - Always build the shared library, libR. I can't think of a good reason not to always turn this on. Unless I'm missing a use-case, this eliminates the need from math/libR. - Memory profiling can cause problems with external BLAS, so only turn on memory profiling with R's internal BLAS. - Use R's internal BLAS by default. The authors recommend the internal BLAS for most use cases, because it is more widely tested. I've also encountered occasional segmentation faults with OpenBLAS. Any potential speed gains with machine-customized OpenBLAS, if any at all, shouldn't trump stability. - Add ONLY_FOR_ARCHS=i386 amd64. Building on ARMv6 fails (even with the updated patches that were relevant for ARMv6) and, I'm told, there are other run-time problems related to calling fortran libraries from C on ARMv6. I don't have any powerpc hardware (yet) to test on. Anyone interested in testing there? - Add a workaround for broken autoconf JPEG detection. - Don't force GCC as the C compiler. This has implications for dependent ports and wen@ should provide feedback before this is committed. [1] - Present the options organized by major dependencies. For example, group LTO and OPENMP together because they, for now, both require GCC. Newer versions of clang support both, so we can look into turning these options on by default soon. - LIBMISSING is no longer required because support for ARMv6 and powerpc have been removed. My guess is that it wouldn't be required (without LONGDOUBLE) in any case with the removed workarounds for quadmath. - Remove the THREADS options. The configure flags --enable-threads=posix versus --disable-threads doesn't seem to have any effect on the build. - Remove patches that are no longer required, because they have been fixed upstream. The workaround in patch-src_extra_tre_tre-internal.h has been included upstream, but an #include <stdint.h> is missing. This is the only line in the updated patch. [1] I see cran.mk has .if ${cran_ARGS:Mcompiles} .include "${PORTSDIR}/math/R/compiler.mk" .endif A quick search shows ./R-cran-lme4 ./R-cran-memisc ./R-cran-MCMCpack ./R-cran-influenceR ./R-cran-mcmc ./R-cran-RcppArmadillo ./R-cran-Rcpp ./R-cran-tibble would be affected. Wen, can you suggest how you would like to handle this? It would be nice to not force GCC if it's a reasonable change for these dependent ports. portlint: WARN: Makefile: unless this is a master port, PORTNAME has to be set by "=", not by "?=". WARN: Makefile: do not call install-info directly. Use the INFO macro instead. (spurious) testport: OK (poudriere: 9.3-RELEASE-p44, i386, default options) testport: OK (poudriere: 9.3-RELEASE-p44, amd64, default options) testport: OK (poudriere: 10.3-RELEASE-p5, i386, default options) testport: OK (poudriere: 10.3-RELEASE-p5, amd64, default options)
Just a note to the need for GCC: GCC is not necessarily needed to use OpenMP. These days, OpenMP can also be done by the CLANG compiler (basically using -lomp instead of -lgomp). A working example is found in the port graphics/darktable. But, GCC _is needed_ for the Fortran functionalities. As far, as I know, there is no alternate in the LLVM/Clang world until today. And many of R's packages do need Fortran!
But since OpenMP support ins't in base clang and GCC will be pulled in anyway for USES=fortran, I thought, for now, the reasonable solution was to pull in GCC when OpenMP was chosen.
##### Hmm, I tried with GCC options (LTO and OpenMP): R is now configured for amd64-portbld-freebsd12.0 Source directory: . Installation directory: /usr/local C compiler: gcc48 -std=gnu99 -O2 -pipe -DLIBICONV_PLUG -isystem /usr/local/include -fstack-protector -Wl,-rpath=/usr/local/lib/gcc48 -fno-strict-aliasing Fortran 77 compiler: gfortran48 -O -Wl,-rpath=/usr/local/lib/gcc48 C++ compiler: g++48 -O2 -pipe -DLIBICONV_PLUG -isystem /usr/local/include -fstack-protector -Wl,-rpath=/usr/local/lib/gcc48 -fno-strict-aliasing -DLIBICONV_PLUG -isystem /usr/local/include -Wl,-rpath=/usr/local/lib/gcc48 C++11 compiler: g++48 -std=c++11 -O2 -pipe -DLIBICONV_PLUG -isystem /usr/local/include -fstack-protector -Wl,-rpath=/usr/local/lib/gcc48 -fno-strict-aliasing -DLIBICONV_PLUG -isystem /usr/local/include -Wl,-rpath=/usr/local/lib/gcc48 C++14 compiler: Fortran 90/95 compiler: gfortran48 -Wl,-rpath=/usr/local/lib/gcc48 Obj-C compiler: gcc48 -O2 -pipe -DLIBICONV_PLUG -isystem /usr/local/include -fstack-protector -Wl,-rpath=/usr/local/lib/gcc48 -fno-strict-aliasing -Wno-import -DLIBICONV_PLUG -fobjc-exceptions Interfaces supported: X11, tcltk External libraries: readline, BLAS(generic), curl Additional capabilities: PNG, JPEG, TIFF, NLS, cairo, ICU Options enabled: shared R library, shared BLAS, R profiling, memory profiling Capabilities skipped: Options not enabled: Recommended packages: yes ##### and without GCC options: R is now configured for amd64-portbld-freebsd12.0 Source directory: . Installation directory: /usr/local C compiler: cc -O2 -pipe -DLIBICONV_PLUG -isystem /usr/local/include -fstack-protector -fno-strict-aliasing Fortran 77 compiler: gfortran48 -O -Wl,-rpath=/usr/local/lib/gcc48 C++ compiler: c++ -O2 -pipe -DLIBICONV_PLUG -isystem /usr/local/include -fstack-protector -fno-strict-aliasing -DLIBICONV_PLUG -isystem /usr/local/include C++11 compiler: c++ -std=c++11 -O2 -pipe -DLIBICONV_PLUG -isystem /usr/local/include -fstack-protector -fno-strict-aliasing -DLIBICONV_PLUG -isystem /usr/local/include C++14 compiler: c++ -std=c++14 -O2 -pipe -DLIBICONV_PLUG -isystem /usr/local/include -fstack-protector -fno-strict-aliasing -DLIBICONV_PLUG -isystem /usr/local/include Fortran 90/95 compiler: gfortran48 -Wl,-rpath=/usr/local/lib/gcc48 Obj-C compiler: cc -O2 -pipe -DLIBICONV_PLUG -isystem /usr/local/include -fstack-protector -fno-strict-aliasing -Wno-import -DLIBICONV_PLUG -fobjc-exceptions Interfaces supported: X11, tcltk External libraries: readline, curl Additional capabilities: PNG, JPEG, TIFF, NLS, cairo, ICU Options enabled: shared R library, shared BLAS, R profiling Capabilities skipped: Options not enabled: memory profiling Recommended packages: yes In the second case (GCC options disabled), you get a mixture of both compiler suites (clang for most, gcc for Fortran). I don't know, if this could be a problem with R packages, depending on them. In the past, I always tried to avoid this kind of mixture. Maybe, that it was not necessary ...
This is intended. Can you describe a specific problem when clang is the C compiler? I haven't had any problems and can't find any reported problems. If there were a specific package that had problems with clang, users could always customize the compiler used for package builds. https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Customizing-package-compilation P.S., Memory profiling was turned off in the second output, not because you chose clang, but because you changed the BLAS option, right?
Yes, you are right. As far as I can see, memory profiling in my second output was disabled, because I choose RBLAS instead of ATLAS. BTW, if I enable LTO and disable OpenMP at the same time, there seems to be a problem with choosing the right compiler: BLAS | LTO | OPENMP | clang usage | memory profiling ========================================================================== RBLAS | disabled | disabled | yes | not enabled RBLAS | enabled | disabled | ERROR: R Unknown version of GCC specified (USE_GCC=GCC). RBLAS | enabled | enabled | no | not enabled -------------------------------------------------------------------------- ATLAS | disabled | disabled | yes | enabled ATLAS | enabled | disabled | ERROR: R Unknown version of GCC specified (USE_GCC=GCC). ATLAS | enabled | enabled | no | enabled I prefer ATLAS over RBLAS, because it is up to 6 times faster than RBLAS. Of course, ATLAS has to be build with the right cpu settings.
Created attachment 173086 [details] second svn diff to improve math/R Thanks for testing. The last diff had LTO_USE=GCC, but it should have been LTO_USE= gcc=yes. Other new changes: - Return the option for the shared R library [1]. This performance hit described in the reference is one reason why the math/libR port should stick around, i.e., for the few ports that require an embedded R. - Create an option for memory profiling (like upstream, off by default). - Create an option for R profiling (like upstream, on by default). - --enable-BLAS-shlib when R's internal BLAS is chosen. P.S., Rainer, I've seen some old blog posts that show ATLAS can be faster. Do you have any specific tests I can reproduce to gauge the current speedups using ATLAS versus R's internal BLAS? ---------------------------------------- [1] From the R Installation and Administration Manual: "Flag --enable-R-shlib causes the make process to build R as a dynamic (shared) library, typically called libR.so, and link the main R executable R.bin against that library. This can only be done if all the code (including system libraries) can be compiled into a dynamic library, and there may be a performance47 penalty*. So you probably only want this if you will be using an application which embeds R." "* We have measured 15-20% on i686 Linux and around 10% on x86_64 Linux."
(In reply to Joseph Mingrone from comment #6) I used [1] to get an old, but quick test about differences between several BLAS versions. It can be run via cat R-benchmark-25.R | time R --slave as described for example in [2]. [1] http://r.research.att.com/benchmarks/R-benchmark-25.R [2] http://edustatistics.org/nathanvan/2013/07/09/for-faster-r-use-openblas-instead-better-than-atlas-trivial-to-switch-to-on-ubuntu/
Thanks. ATLAS and OpenBLAS compiled locally from ports with the default options were indeed faster. |---------------+------------------------------------------+----------| | Implemenation | Overall Results | [1] | |---------------+------------------------------------------+----------| | Internal | http://ftfl.ca/misc/R-bm-25-internal.out | 5.97 | | NetLib | http://ftfl.ca/misc/R-bm-25-netlib.out | 6.12 | | Atlas | http://ftfl.ca/misc/R-bm-25-atlas.out | 0.87 | | OpenBLAS | http://ftfl.ca/misc/R-bm-25-openblas.out | 0.67 | |---------------+------------------------------------------+----------| [1] time (s) to calculate the determinant of a 2500x2500 random matrix Since both ATLAS and OpenBLAS tune to specific hardware, and OpenBLAS is giving segmentation faults [2], I'll leave the default to RBLAS. [2] address 0x0, cause 'memory not mapped' Possible actions: 1: abort (with core dump, if enabled) 2: normal R exit 3: exit R without saving workspace 4: exit R saving workspace
Created attachment 173182 [details] svn diff #3 Changes since last diff: - add readline to USES= - change the BLAS group of options from OPTIONS_RADIO to OPTIONS_SINGLE - explicitly disable BLAS-shlib when the internal BLAS is not chosen (can cause a problem when building outside of poudriere) - remove a misplaced trailing '\' portlint: same two warnings as above testport: OK (poudriere: 9.3-RELEASE-p44, i386, default options) testport: OK (poudriere: 9.3-RELEASE-p44, amd64, default options) testport: OK (poudriere: 10.3-RELEASE-p5, i386, default options) testport: OK (poudriere: 10.3-RELEASE-p5, amd64, default options)
Created attachment 173265 [details] svn diff #4: redo diff after other commits to math/R There were other commits to math/R, so redo this patch. I've also attempted to reduce disruption with these changes, but still permit clang to build on 10+ by updating compiler.mk with # Compiler settings required for R and the R-cran ports. .if ${OPSYS} == FreeBSD .if ${OSVERSION} < 1000015 USE_GCC= yes .endif .endif I tested all ports that have USES= cran:compiles and they all built successfully, so I think this patch is ready unless anyone (wen@? tota@?) has issues. Thanks to rhurlin@gwdg.de for testing. Build logs using default options: math/R http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_01h23m39s/logs/R-3.3.1_2.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_01h23m40s/logs/R-3.3.1_2.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-04_01h23m42s/logs/R-3.3.1_2.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-04_01h23m44s/logs/R-3.3.1_2.log Build logs for ports with USES=cran:compiles math/R-cran-lme4 http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-03_23h08m45s/logs/R-cran-lme4-1.1.12.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-03_23h08m53s/logs/R-cran-lme4-1.1.12.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-03_23h08m59s/logs/R-cran-lme4-1.1.12.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-03_23h09m05s/logs/R-cran-lme4-1.1.12.log math/R-cran-memisc http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-03_23h23m28s/logs/R-cran-memisc-0.99.7.1.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-03_23h23m37s/logs/R-cran-memisc-0.99.7.1.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-03_23h23m42s/logs/R-cran-memisc-0.99.7.1.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-03_23h23m47s/logs/R-cran-memisc-0.99.7.1.log math/R-cran-spdep http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-03_23h27m19s/logs/R-cran-spdep-0.6.6.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-03_23h27m25s/logs/R-cran-spdep-0.6.6.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-03_23h27m31s/logs/R-cran-spdep-0.6.6.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-03_23h27m36s/logs/R-cran-spdep-0.6.6.log math/R-cran-MCMCpack http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-03_23h37m50s/logs/R-cran-MCMCpack-1.3.6.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-03_23h37m55s/logs/R-cran-MCMCpack-1.3.6.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-03_23h37m59s/logs/R-cran-MCMCpack-1.3.6.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-03_23h38m05s/logs/R-cran-MCMCpack-1.3.6.log math/R-cran-influenceR http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_00h18m40s/logs/R-cran-influenceR-0.1.0.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_00h18m44s/logs/R-cran-influenceR-0.1.0.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-03_23h46m17s/logs/R-cran-influenceR-0.1.0.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-03_23h46m23s/logs/R-cran-influenceR-0.1.0.log math/R-cran-mcmc http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_00h27m07s/logs/R-cran-mcmc-0.9.4.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_00h27m14s/logs/R-cran-mcmc-0.9.4.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-04_00h27m19s/logs/R-cran-mcmc-0.9.4.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-04_00h27m23s/logs/R-cran-mcmc-0.9.4.log math/R-cran-RcppArmadillo http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_00h31m05s/logs/R-cran-RcppArmadillo-0.7.200.2.0.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_00h31m11s/logs/R-cran-RcppArmadillo-0.7.200.2.0.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-04_00h31m16s/logs/R-cran-RcppArmadillo-0.7.200.2.0.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-04_00h31m22s/logs/R-cran-RcppArmadillo-0.7.200.2.0.log science/R-cran-som http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_00h34m46s/logs/R-cran-som-0.3.5.1.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_00h34m54s/logs/R-cran-som-0.3.5.1.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-04_00h35m00s/logs/R-cran-som-0.3.5.1.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-04_00h35m05s/logs/R-cran-som-0.3.5.1.log textproc/R-cran-xml2 http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_00h37m07s/logs/R-cran-xml2-1.0.0.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_00h37m12s/logs/R-cran-xml2-1.0.0.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-04_00h37m17s/logs/R-cran-xml2-1.0.0.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-04_00h37m23s/logs/R-cran-xml2-1.0.0.log ftp/R-cran-curl http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_00h42m08s/logs/R-cran-curl-1.1.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_00h42m12s/logs/R-cran-curl-1.1.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-04_00h42m17s/logs/R-cran-curl-1.1.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-04_00h42m23s/logs/R-cran-curl-1.1.log misc/R-cran-mime http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_00h44m18s/logs/R-cran-mime-0.5.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_00h44m24s/logs/R-cran-mime-0.5.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-04_00h44m28s/logs/R-cran-mime-0.5.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-04_00h44m33s/logs/R-cran-mime-0.5.log devel/R-cran-Rcpp http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_00h48m46s/logs/R-cran-Rcpp-0.12.6.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_00h48m50s/logs/R-cran-Rcpp-0.12.6.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-04_00h48m53s/logs/R-cran-Rcpp-0.12.6.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-04_00h48m58s/logs/R-cran-Rcpp-0.12.6.log devel/R-cran-tibble http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_00h51m42s/logs/R-cran-tibble-1.1.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_00h51m47s/logs/R-cran-tibble-1.1.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-04_00h51m51s/logs/R-cran-tibble-1.1.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-04_00h51m59s/logs/R-cran-tibble-1.1.log finance/R-cran-tseries http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_00h57m37s/logs/R-cran-tseries-0.10.35_1.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_00h57m41s/logs/R-cran-tseries-0.10.35_1.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-04_00h57m45s/logs/R-cran-tseries-0.10.35_1.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-04_00h57m51s/logs/R-cran-tseries-0.10.35_1.log converters/R-cran-jsonlite http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_01h01m52s/logs/R-cran-jsonlite-1.0.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_01h01m56s/logs/R-cran-jsonlite-1.0.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-04_01h02m01s/logs/R-cran-jsonlite-1.0.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-04_01h02m06s/logs/R-cran-jsonlite-1.0.log converters/R-cran-base64enc http://pkg.awarnach.mathstat.dal.ca/data/9i386-default/2016-08-04_01h05m34s/logs/R-cran-base64enc-0.1.3.log http://pkg.awarnach.mathstat.dal.ca/data/9amd64-default/2016-08-04_01h05m38s/logs/R-cran-base64enc-0.1.3.log http://pkg.awarnach.mathstat.dal.ca/data/10i386-default/2016-08-04_01h05m42s/logs/R-cran-base64enc-0.1.3.log http://pkg.awarnach.mathstat.dal.ca/data/10amd64-default/2016-08-04_01h05m47s/logs/R-cran-base64enc-0.1.3.log
A commit references this bug: Author: marino Date: Fri Aug 5 02:30:18 UTC 2016 New revision: 419666 URL: https://svnweb.freebsd.org/changeset/ports/419666 Log: math/R: maintainer rework See PR for signficant changes to math/R port including options, build defaults, and patch updates. PR: 211393 Submitted by: Joseph Mingrone (maintainer) Changes: head/math/R/Makefile head/math/R/compiler.mk head/math/R/distinfo head/math/R/files/patch-configure head/math/R/files/patch-src__nmath__standalone__test.c head/math/R/files/patch-src_extra_tre_tre-internal.h
enjoy!