Bug 252379
| Summary: | Mk/Uses/compiler.mk: remove USE_GCC=yes for compiler:openmp due to libomp.so is part of base since 11.3/12.1 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Vladimir Druzenko <vvd> | ||||
| Component: | Individual Port(s) | Assignee: | Port Management Team <portmgr> | ||||
| Status: | Closed Overcome By Events | ||||||
| Severity: | Affects Many People | CC: | cmt, cyberbotx, danfe, demon, fbsd, freebsd-ppc, gnome, h2+fbsdports, jhale, jrm, jwb, kwm, lantw44, mail, mi, pfg, phd_kimberlite, pi, portmgr, rhurlin, riggs, rodrigo, sevenjp, stephen, sunpoet, tatsuki_makino, thierry, val, yuri | ||||
| Priority: | --- | Flags: | vvd:
maintainer-feedback?
(portmgr) vvd: exp-run? |
||||
| Version: | Latest | ||||||
| Hardware: | Any | ||||||
| OS: | Any | ||||||
| URL: | https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222858#c4 | ||||||
| See Also: | https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254284 | ||||||
| Attachments: |
|
||||||
|
Description
Vladimir Druzenko
2021-01-03 10:07:09 UTC
Can't add maintainer of the biology/pooler to CC: ssb22 [at] cam.ac.uk. Commenting for audio/libsoxr (my only port from the list if I didn't overlook something): The only thing soxr is using from openmp is `#pragma omp parallel for` in two places with non-overlapping data and no dependencies (i.e. simplest possible use case for omp threading). Will test it soonish, but don't expect problems here. (In reply to Thomas Zander from comment #2) audio/libsoxr - build fine (forgot add this status to 1st post). Will this work on all architectures now? It's been feasible on x86 for years (using openmp port), but IIRC the move was held up by gcc 4.2 based platforms (e.g. ppc). There are also other potential issues that have to be addressed: https://lists.freebsd.org/pipermail/freebsd-ports/2020-March/117947.html I'd like to see this happen as well, if we can do it without causing regressions. (In reply to Jason W. Bacon from comment #4) As a 1st step, we can do something like this: .if ${_COMPILER_ARGS:Mopenmp} && ${ARCH} != amd64 .if ${COMPILER_TYPE} == clang USE_GCC= yes CHOSEN_COMPILER_TYPE= gcc .endif .endif Tested compiling multimedia/pHash with OPENMP option enabled and the given patch to remove USE_GCC. Compiles fine, also compiles fine without compiler:openmp given. I found it a bother :), so multimedia/libopenshot stopped supporting non-base openmp at bug 244063 comment #3 commit. Then its fence was taken down in ports/r528082 commit. (In reply to VVD from comment #5) I think we would also need the following to avoid mixing clang and gfortran openmp: .if ${USES:Mfortran} && ${FC} == gfortran USE_GCC= yes CHOSEN_COMPILER_TYPE= gcc .endif I.e. exempt any ports that use gfortran from the change. Likewise for ports that currently use gcc for any reason other than openmp. That should keep most of the worms in the can. Ports that can use purely clang should be easier to deal with. (In reply to Jason W. Bacon from comment #8) Maybe easier to start with removing "compiler:openmp" from ports. Like here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247753 Just found graphics/GraphicsMagick have OPENMP_USES=compiler:gcc-c++11-lib. It build and work fine without this line. MAINTAINER= sunpoet@ math/octave-forge-nurbs builds fine without compiler:openmp. It also seems to work. math/librsb doesn't. (In reply to Stephen Montgomery-Smith from comment #11) Maybe rename "compiler:openmp" to "compiler:gcc-openmp" and remove "compiler:openmp" from tested ports (build and work fine with openmp from base)? graphics/shotwell and graphics/libraw (after patch, with openmp enabled) seem to work ok with what I've on hand (In reply to Jason W. Bacon from comment #4) As I suspected, this will cause breakage on FreeBSD < 13 ppc. Presently a lot of ports have the following hack for ppc because it still uses gcc 4.2: # Force newer GCC on platforms using GCC 4.2 as base .if ${CHOSEN_COMPILER_TYPE} == gcc USE_GCC= yes .endif CHOSEN_COMPILER_TYPE is only defined when the port has USES=compiler:something So simply removing compiler:openmp will break a lot of ports for ppc right now with the following error: make: "/usr/ports/wip/gcta/Makefile" line 29: Malformed conditional (${CHOSEN_COMPILER_TYPE} == gcc && ${COMPILER_VERSION} <= 42) make: Fatal errors encountered -- cannot continue I think this should be easy to fix by defining CHOSEN_COMPILER_TYPE in the absence of USES=compiler:*. Then we just have to make sure that ports using gcc/gfortran and openmp use gcc for everything. Some might try to use clang or clang++ along with gfortran which as someone mentioned is not tested when openmp is in play. (In reply to Jason W. Bacon from comment #14) I should clarify that removing compiler:openmp will break ports referencing CHOSEN_COMPILER_TYPE on all platforms, not just on ppc. (In reply to Jason W. Bacon from comment #15) Those ports should be switched to USES=compiler (In reply to Antoine Brodin from comment #16) That strategy falls short on our goal to stop using gcc where it isn't really needed. We'd be using gcc on amd64 because gcc 4.2 doesn't provide c++-14 on powerpc, for example. The patch below adds a default value for CHOSEN_COMPILER_TYPE based on the architecture, which can be overridden by compiler.mk. Index: /usr/ports/Mk/bsd.port.mk =================================================================== --- /usr/ports/Mk/bsd.port.mk (revision 560811) +++ /usr/ports/Mk/bsd.port.mk (working copy) @@ -1141,6 +1141,15 @@ _EXPORTED_VARS+= PPC_ABI .endif +.if !defined(CHOSEN_COMPILER_TYPE) +# FIXME: Are there other archs that use gcc in base? +.if ${ARCH} == powerpc64 || ${ARCH} == powerpc +CHOSEN_COMPILER_TYPE=gcc +.else +CHOSEN_COMPILER_TYPE=clang +.endif +.endif + # Get operating system versions for a cross build .if defined(CROSS_SYSROOT) .if !exists(${CROSS_SYSROOT}/usr/include/sys/param.h) I think this is a good starting point, but there may be a better place to set this or other issues to resolve. So far biology/clang and biology/gcta both fail to build with clang. I'll try to correct these issues and temporarily set USE_GCC unconditionally as a stop-gap if I can't resolve them quickly. (In reply to Jason W. Bacon from comment #17) Ports testing CHOSEN_COMPILER_TYPE value must have USES=compiler, this must not be provided by bsd.port.mk (In reply to Antoine Brodin from comment #18) Why shouldn't it added somewhere outside compiler.mk? We have 138 ports using this check according to port-grep '.if.*CHOSEN_COMPILER' -l|wc many of which no longer need gcc. So we have two choices if we want to eliminate unnecessary gcc deps: 1. Support CHOSEN_COMPILER_TYPE without USES=compiler 2. Alter 138 ports to use a different strategy, like using ARCH and OSVERSION, which would be a maintenance nightmare (In reply to Jason W. Bacon from comment #19) Any port can use USES=compiler, this won't force gcc or clang but will set COMPILER_TYPE, ALT_COMPILER_TYPE, COMPILER_VERSION and ALT_COMPILER_VERSION Testing for CHOSEN_COMPILER_TYPE without USES=compiler:something doesn't make sense (In reply to Antoine Brodin from comment #20) I see, I didn't realize USES=compiler could be used without any parameters. In any cases where something better than gcc 4.2 is needed, we can simply replace compiler:openmp with something like compiler:c++11-lang. Since two compiler: parameters cannot be specified together and both just set USE_GCC anyway, one had to be picked at random. In most cases, the need for the other is not documented, so that will have to be figured out for each port. This is the case for biology/canu, which also needs USES=localbase to quell a compile error when building with base clang. I've updated all my ports using openmp to eliminate the gcc dep. Below is a summary of the necessary changes to show some examples of what we're up against. I plan to do some runtime testing before committing these changes. === /usr/ports/biology/canu/Makefile:MAINTAINER= jwb@FreeBSD.org Requires both openmp and c++11: -USES= compiler:openmp gmake perl5 +USES= compiler:c++11-lang gmake localbase perl5 No longer needed since replacing compiler:openmp with compiler:c++11-lang: -.include <bsd.port.pre.mk> - -# GCC 4.2.1 (still base compiler on some 2nd tier platforms) cannot build canu -.if ${COMPILER_TYPE} == gcc && ${COMPILER_VERSION} <= 42 -USE_GCC= yes -.endif - === /usr/ports/biology/cd-hit/Makefile:MAINTAINER= jwb@FreeBSD.org Requires openmp only: -USES= compiler:openmp gmake shebangfix +USES= gmake shebangfix /usr/ports/biology/fasttree/Makefile:MAINTAINER= jwb@FreeBSD.org Requires openmp only: -USES= compiler:openmp === /usr/ports/biology/gcta/Makefile:MAINTAINER= jwb@FreeBSD.org Requires both openmp and c++11: -USES= compiler:openmp dos2unix eigen:3 gmake localbase zip +USES= compiler:c++11-lang dos2unix eigen:3 gmake localbase:ldflags zip No longer needed since replacing compiler:openmp with compiler:c++11-lang: -.include <bsd.port.pre.mk> - -.if ${CHOSEN_COMPILER_TYPE} == gcc && ${COMPILER_VERSION} <= 42 -USE_GCC= yes -.endif - === /usr/ports/biology/ncbi-blast+/Makefile:MAINTAINER= jwb@FreeBSD.org Requires both openmp and c++14: -USES= compiler:openmp gmake shebangfix perl5 python +USES= compiler:c++14-lang gmake shebangfix perl5 python -LDFLAGS+= -latomic -latomic required for gcc, does not work with clang: -# Force newer GCC on platforms using GCC 4.2 as base .if ${CHOSEN_COMPILER_TYPE} == gcc -USE_GCC= yes +# In case of problems: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220822 +LDFLAGS+= -latomic # Only exists in gcc libs .endif === /usr/ports/biology/stacks/Makefile:MAINTAINER= jwb@FreeBSD.org -USES= compiler:openmp gmake python shebangfix +USES= compiler:c++11-lang gmake python shebangfix clang is missing some atomic functionality on i386: -# Force newer GCC on platforms using GCC 4.2 as base -.if ${CHOSEN_COMPILER_TYPE} == gcc +# Temp workaround, see PR 230888 +.if ${ARCH} == i386 USE_GCC= yes .endif === /usr/ports/biology/star/Makefile:MAINTAINER= jwb@FreeBSD.org Requires both openmp and c++11: -USES= compiler:openmp gmake +USES= compiler:c++11-lang gmake No longer needed since replacing compiler:openmp with compiler:c++11-lang: -.include <bsd.port.pre.mk> - -# Force newer compiler on platforms using GCC 4.2 as base -.if ${CHOSEN_COMPILER_TYPE} == gcc && ${COMPILER_VERSION} <= 42 -USE_GCC= yes -.endif - === /usr/ports/sysutils/parafly/Makefile:MAINTAINER= jwb@FreeBSD.org Requires openmp only: -USES= compiler:openmp tar:tgz +USES= tar:tgz A commit references this bug: Author: jwb Date: Mon Jan 11 16:51:16 UTC 2021 New revision: 561227 URL: https://svnweb.freebsd.org/changeset/ports/561227 Log: biology/canu: Drop dependency on gcc Replace compiler:openmp with compiler:c++11-lang and localbase PR: port/247753, ports/252379 Changes: head/biology/canu/Makefile A commit references this bug: Author: jwb Date: Mon Jan 11 16:55:16 UTC 2021 New revision: 561228 URL: https://svnweb.freebsd.org/changeset/ports/561228 Log: biology/cd-hit: Drop dependency on gcc Simply remove compiler:openmp Builds with base compiler on amd64, i386, and powerpc64 PR: port/247753, ports/252379 Changes: head/biology/cd-hit/Makefile A commit references this bug: Author: jwb Date: Mon Jan 11 16:58:14 UTC 2021 New revision: 561229 URL: https://svnweb.freebsd.org/changeset/ports/561229 Log: biology/fasttree: Drop dependency on gcc Simply remove compiler:openmp Builds with base compiler on amd64, i386, and powerpc64 12.x-RELEASE PR: port/247753, ports/252379 Changes: head/biology/fasttree/Makefile A commit references this bug: Author: jwb Date: Mon Jan 11 17:09:38 UTC 2021 New revision: 561231 URL: https://svnweb.freebsd.org/changeset/ports/561231 Log: biology/gcta: Drop dependency on gcc Replace compiler:openmp with compiler:c++11-lang Clang build also requires localbase:ldflags Build tested on amd64, i386 and powerpc64 12.x-RELEASE PR: port/247753, ports/252379 Changes: head/biology/gcta/Makefile A commit references this bug: Author: jwb Date: Mon Jan 11 17:17:36 UTC 2021 New revision: 561233 URL: https://svnweb.freebsd.org/changeset/ports/561233 Log: biology/ncbi-blast+: Drop dependency on gcc Replace compiler:openmp with compiler:c++14-lang Clang build also requires localbase:ldflags and a source patch to explicitly set variables to shared GCC-based platforms still require -latomic PR: port/247753, ports/252379 Changes: head/biology/ncbi-blast+/Makefile head/biology/ncbi-blast+/files/patch-src_objtools_blast_seqdb__reader_test_seqdb__perf.cpp A commit references this bug: Author: jwb Date: Mon Jan 11 17:27:30 UTC 2021 New revision: 561234 URL: https://svnweb.freebsd.org/changeset/ports/561234 Log: biology/stacks: Drop dependency on gcc Replace compiler:openmp with compiler:c++11-lang and localbase Set USE_GCC for i386 to work around limitations in clang atomic (PR 230888) PR: port/247753, ports/252379 Changes: head/biology/stacks/Makefile A commit references this bug: Author: jwb Date: Mon Jan 11 17:39:52 UTC 2021 New revision: 561235 URL: https://svnweb.freebsd.org/changeset/ports/561235 Log: biology/star: Drop dependency on gcc Replace compiler:openmp with compiler:c++11-lang and localbase PR: port/247753, ports/252379 Changes: head/biology/star/Makefile head/biology/star/files/patch-Makefile A commit references this bug: Author: jwb Date: Mon Jan 11 17:43:22 UTC 2021 New revision: 561236 URL: https://svnweb.freebsd.org/changeset/ports/561236 Log: sysutils/parafly: Upgrade to 0.1.0 from Github, drop gcc dependency Simply remove compiler:openmp Builds with base compiler on amd64, i386, and powerpc64 12.x-RELEASE PR: port/247753, ports/252379 Changes: head/sysutils/parafly/Makefile head/sysutils/parafly/distinfo head/sysutils/parafly/pkg-descr A commit references this bug: Author: jrm Date: Mon Jan 11 17:54:42 UTC 2021 New revision: 561239 URL: https://svnweb.freebsd.org/changeset/ports/561239 Log: biology/hyphy: Remove USES=compiler:openmp USES=compiler:openmp still forces building with GCC. This is no longer necessary on tier 1 platforms, which have openmp in the base system. PR: 247753, 252379 Changes: head/biology/hyphy/Makefile biology/iqtree gives linker errors with clang. I spent a bit of time trying to figure out what's going on, but no luck yet. http://pkg.awarnach.mathstat.dal.ca/data/12amd64-default/2021-01-11_14h39m54s/logs/errors/iqtree-2.0.6_1.log (In reply to Joseph Mingrone from comment #32) Did you tried compile it without ccache and with clang 10? And openblas-0.3.12_1,1 require gcc, but build fine with system clang 10 from 12.2. (In reply to Joseph Mingrone from comment #32) The problem with iqtree is that is ships with their private libomp (among others) - precompiled on Linux as lib/libomp.a (there'S also Windows and Mac versions of that, and 32bit version in lib32). That's a stunningly horrible idea... Try someting along these lines as files/patch-CMakeLists.txt: --- CMakeLists.txt.orig 2021-01-11 20:24:32.584318000 +0100 +++ CMakeLists.txt 2021-01-11 20:26:49.293185000 +0100 @@ -323,12 +323,12 @@ if(CLANG AND APPLE) link_directories(${PROJECT_SOURCE_DIR}/libmac) - elseif (WIN32 OR UNIX) - if (BINARY32) - link_directories(${PROJECT_SOURCE_DIR}/lib32) - else() - link_directories(${PROJECT_SOURCE_DIR}/lib) - endif() +# elseif (WIN32 OR UNIX) +# if (BINARY32) +# link_directories(${PROJECT_SOURCE_DIR}/lib32) +# else() +# link_directories(${PROJECT_SOURCE_DIR}/lib) +# endif() endif() if (VCC) That makes the port compile with clang for me (but I can't do any runtime tests) (In reply to Christoph Moench-Tegeder from comment #35) Thanks. I'll use a similar patch to fix the amd64 build. This one will probably be acceptable upstream. if(CLANG AND APPLE) link_directories(${PROJECT_SOURCE_DIR}/libmac) - elseif (WIN32 OR UNIX) + elseif (WIN32 OR UNIX AND NOT ${CMAKE_SYSTEM_NAME} STREQUAL "FreeBSD") if (BINARY32) link_directories(${PROJECT_SOURCE_DIR}/lib32) else() Switching to clang++ does break i386 builds because base clang doesn't have 64 bit atomic functions for i386 (see bug 230888). I'll probably just mark i386 broken, but if there is a single user running iqtree on i386, then I'll investigate further. A commit references this bug: Author: jrm Date: Tue Jan 12 18:12:13 UTC 2021 New revision: 561368 URL: https://svnweb.freebsd.org/changeset/ports/561368 Log: biology/iqtree: Remove USES=compiler:openmp, fix amd64 clang++ build - USES=compiler:openmp still forces building with GCC. This is no longer necessary on tier 1 platforms, which have openmp in base. - Mark broken on i386, because base clang doesn't have 64 bit atomic functions for i386 (see bug 230888). PR: 247753, 252379 Submitted by: cmt (based on) Changes: head/biology/iqtree/Makefile head/biology/iqtree/files/ head/biology/iqtree/files/patch-CMakeLists.txt In math/ceres-solver and graphics/colmap removing USES=compiler:openmp should be fine because it has no effect until bug 199603 is fixed. However, some Clang architectures like armv6 and armv7 still lack libomp thus use GCC via USES=compiler:gcc-c++11-lib. I've also dropped maintainership in ports r563450. Is there anything blocking left? Manual fix Mk/Uses/compiler.mk every days is annoying already… :-D All my ports are ready. biology/hyphy and biology/iqtree are all set. 30 ports left with "USES=compiler:openmp": archivers/rpm4 archivers/pxz audio/libsoxr audio/soundtouch biology/seqan-apps biology/pooler cad/calculix-ccx graphics/aaphoto graphics/ImageMagick6 graphics/colmap graphics/ImageMagick7 graphics/libmypaint graphics/libraw graphics/mypaint graphics/pfstools graphics/inkscape lang/J math/metis math/hmat-oss math/ceres-solver math/librsb math/octave-forge-nurbs math/openblas math/fftw3 math/tmv math/viennacl math/py-theano multimedia/pHash science/afni sysutils/b2sum A commit references this bug: Author: stephen Date: Wed Feb 24 18:12:58 UTC 2021 New revision: 566495 URL: https://svnweb.freebsd.org/changeset/ports/566495 Log: - Remove uses=compiler:openmp PR: 252379 Changes: head/math/octave-forge-nurbs/Makefile Is there a still a version of FreeBSD out there, which is both "supported" by ports, and requires a special compiler to use OpenMP? Maybe, it is time to just make this flag into a no-op? As things stand, ImageMagick7, for one, is outright broken, when OpenMP is enabled -- though its C-code is Ok, all of the C++-code self-tests segfault, because they are compiled with g++10, whereas the stock c++ is used for most of the dependencies. (See also Bug 247753.) Closing as this was addressed in 07fb2d5e9d001934f1670d01aec1f536f14ebde2. |