Bug 252379 - Mk/Uses/compiler.mk: remove USE_GCC=yes for compiler:openmp due to libomp.so is part of base since 11.3/12.1
Summary: Mk/Uses/compiler.mk: remove USE_GCC=yes for compiler:openmp due to libomp.so ...
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Port Management Team
URL: https://bugs.freebsd.org/bugzilla/sho...
Keywords:
Depends on:
Blocks:
 
Reported: 2021-01-03 10:07 UTC by VVD
Modified: 2021-01-12 18:12 UTC (History)
26 users (show)

See Also:
vvd: maintainer-feedback? (portmgr)
vvd: exp-run?


Attachments
remove USE_GCC=yes for compiler:openmp (318 bytes, patch)
2021-01-03 10:07 UTC, VVD
vvd: maintainer-approval? (portmgr)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description VVD 2021-01-03 10:07:09 UTC
Created attachment 221237 [details]
remove USE_GCC=yes for compiler:openmp

The source for this PR is build error of multimedia/libopenshot if ImageMagick7 option is on and graphics/ImageMagick7 build with OPENMP (USES=compiler:openmp):
ld: error: src/libopenshot.so.0.2.5: undefined reference to Magick::Image::Image(unsigned long, unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MagickCore::StorageType, void const*)

But multimedia/libopenshot build fine if graphics/ImageMagick7 compiled without OPENMP or with my patch and with OPENMP on.

This changes affected 42 ports:
archivers/rpm4 - build and run fine (not much tested)
archivers/pxz
audio/libsoxr
biology/cd-hit
biology/gcta
biology/seqan-apps
biology/fasttree
biology/hyphy
biology/bolt-lmm
biology/iqtree
biology/ncbi-blast+
biology/stacks
biology/star
biology/canu
biology/pooler
cad/calculix-ccx
graphics/aaphoto
graphics/ImageMagick6 - build and run fine (manual test https://imagemagick.org/script/openmp.php: "convert logo: -resize 200% -bench 40 null:")
graphics/colmap
graphics/ImageMagick7 - build and run fine (manual test https://imagemagick.org/script/openmp.php: "convert logo: -resize 200% -bench 40 null:")
graphics/shotwell
graphics/libmypaint - build fine (not much tested, but gimp run and open image work fine)
graphics/libraw - build fine
graphics/mypaint
graphics/pfstools
graphics/inkscape - build, run open image work fine (not much tested)
lang/J
math/metis - build fine
math/openblas - build fine
math/hmat-oss
math/ceres-solver
math/librsb
math/octave-forge-nurbs
math/fftw3 - build fine
math/tmv
math/viennacl
math/py-theano
multimedia/pHash
science/afni
security/john
sysutils/b2sum
sysutils/parafly

Added to CC maintainers of all those ports and ask for test those ports, because "compiles != works as intended".

Possible is safe to remove compiler:openmp.

P.S. There is other PR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247753.
Comment 1 VVD 2021-01-03 10:10:30 UTC
Can't add maintainer of the biology/pooler to CC: ssb22 [at] cam.ac.uk.
Comment 2 Thomas Zander freebsd_committer 2021-01-03 11:15:54 UTC
Commenting for audio/libsoxr (my only port from the list if I didn't overlook something): The only thing soxr is using from openmp is `#pragma omp parallel for` in two places with non-overlapping data and no dependencies (i.e. simplest possible use case for omp threading). Will test it soonish, but don't expect problems here.
Comment 3 VVD 2021-01-03 11:31:31 UTC
(In reply to Thomas Zander from comment #2)
audio/libsoxr - build fine (forgot add this status to 1st post).
Comment 4 Jason W. Bacon freebsd_committer 2021-01-03 15:10:14 UTC
Will this work on all architectures now?  It's been feasible on x86 for years (using openmp port), but IIRC the move was held up by gcc 4.2 based platforms (e.g. ppc).

There are also other potential issues that have to be addressed:

https://lists.freebsd.org/pipermail/freebsd-ports/2020-March/117947.html

I'd like to see this happen as well, if we can do it without causing regressions.
Comment 5 VVD 2021-01-03 15:41:48 UTC
(In reply to Jason W. Bacon from comment #4)

As a 1st step, we can do something like this:

.if ${_COMPILER_ARGS:Mopenmp} && ${ARCH} != amd64
.if ${COMPILER_TYPE} == clang
USE_GCC=	yes
CHOSEN_COMPILER_TYPE=	gcc
.endif
.endif
Comment 6 Naram Qashat 2021-01-03 17:19:12 UTC
Tested compiling multimedia/pHash with OPENMP option enabled and the given patch to remove USE_GCC. Compiles fine, also compiles fine without compiler:openmp given.
Comment 7 Tatsuki Makino 2021-01-03 21:31:45 UTC
I found it a bother :), so multimedia/libopenshot stopped supporting non-base openmp at bug 244063 comment #3 commit.
Then its fence was taken down in ports/r528082 commit.
Comment 8 Jason W. Bacon freebsd_committer 2021-01-03 23:13:21 UTC

(In reply to VVD from comment #5)
I think we would also need the following to avoid mixing clang and gfortran openmp:

.if ${USES:Mfortran} && ${FC} == gfortran
USE_GCC=	yes
CHOSEN_COMPILER_TYPE=	gcc
.endif

I.e. exempt any ports that use gfortran from the change.

Likewise for ports that currently use gcc for any reason other than openmp.

That should keep most of the worms in the can.  Ports that can use purely clang should be easier to deal with.
Comment 9 VVD 2021-01-04 01:20:15 UTC
(In reply to Jason W. Bacon from comment #8)
Maybe easier to start with removing "compiler:openmp" from ports.
Like here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247753
Comment 10 VVD 2021-01-04 07:05:41 UTC
Just found graphics/GraphicsMagick have OPENMP_USES=compiler:gcc-c++11-lib.
It build and work fine without this line.
MAINTAINER=     sunpoet@
Comment 11 Stephen Montgomery-Smith freebsd_committer 2021-01-04 08:03:44 UTC
math/octave-forge-nurbs builds fine without compiler:openmp.  It also seems to work.

math/librsb doesn't.
Comment 12 VVD 2021-01-04 08:28:26 UTC
(In reply to Stephen Montgomery-Smith from comment #11)
Maybe rename "compiler:openmp" to "compiler:gcc-openmp" and remove "compiler:openmp" from tested ports (build and work fine with openmp from base)?
Comment 13 Christoph Moench-Tegeder freebsd_committer 2021-01-05 11:03:27 UTC
graphics/shotwell and graphics/libraw (after patch, with openmp enabled) seem to work ok with what I've on hand
Comment 14 Jason W. Bacon freebsd_committer 2021-01-08 23:31:19 UTC
(In reply to Jason W. Bacon from comment #4)

As I suspected, this will cause breakage on FreeBSD < 13 ppc.

Presently a lot of ports have the following hack for ppc because it still uses gcc 4.2:

# Force newer GCC on platforms using GCC 4.2 as base
.if ${CHOSEN_COMPILER_TYPE} == gcc
USE_GCC=        yes
.endif

CHOSEN_COMPILER_TYPE is only defined when the port has

USES=compiler:something

So simply removing compiler:openmp will break a lot of ports for ppc right now with the following error:

make: "/usr/ports/wip/gcta/Makefile" line 29: Malformed conditional (${CHOSEN_COMPILER_TYPE} == gcc && ${COMPILER_VERSION} <= 42)
make: Fatal errors encountered -- cannot continue

I think this should be easy to fix by defining CHOSEN_COMPILER_TYPE in the absence of USES=compiler:*.

Then we just have to make sure that ports using gcc/gfortran and openmp use gcc for everything.  Some might try to use clang or clang++ along with gfortran which as someone mentioned is not tested when openmp is in play.
Comment 15 Jason W. Bacon freebsd_committer 2021-01-08 23:33:16 UTC
(In reply to Jason W. Bacon from comment #14)

I should clarify that removing compiler:openmp will break ports referencing CHOSEN_COMPILER_TYPE on all platforms, not just on ppc.
Comment 16 Antoine Brodin freebsd_committer 2021-01-09 08:50:25 UTC
(In reply to Jason W. Bacon from comment #15)
Those ports should be switched to USES=compiler
Comment 17 Jason W. Bacon freebsd_committer 2021-01-09 14:38:51 UTC
(In reply to Antoine Brodin from comment #16)

That strategy falls short on our goal to stop using gcc where it isn't really needed.  We'd be using gcc on amd64 because gcc 4.2 doesn't provide c++-14 on powerpc, for example.

The patch below adds a default value for CHOSEN_COMPILER_TYPE based on the architecture, which can be overridden by compiler.mk.

Index: /usr/ports/Mk/bsd.port.mk
===================================================================
--- /usr/ports/Mk/bsd.port.mk	(revision 560811)
+++ /usr/ports/Mk/bsd.port.mk	(working copy)
@@ -1141,6 +1141,15 @@
 _EXPORTED_VARS+=	PPC_ABI
 .endif
 
+.if !defined(CHOSEN_COMPILER_TYPE)
+# FIXME: Are there other archs that use gcc in base?
+.if ${ARCH} == powerpc64 || ${ARCH} == powerpc
+CHOSEN_COMPILER_TYPE=gcc
+.else
+CHOSEN_COMPILER_TYPE=clang
+.endif
+.endif
+
 # Get operating system versions for a cross build
 .if defined(CROSS_SYSROOT)
 .if !exists(${CROSS_SYSROOT}/usr/include/sys/param.h)

I think this is a good starting point, but there may be a better place to set this or other issues to resolve.

So far biology/clang and biology/gcta both fail to build with clang.

I'll try to correct these issues and temporarily set USE_GCC unconditionally as a stop-gap if I can't resolve them quickly.
Comment 18 Antoine Brodin freebsd_committer 2021-01-09 15:59:17 UTC
(In reply to Jason W. Bacon from comment #17)
Ports testing CHOSEN_COMPILER_TYPE value must have USES=compiler, this must not be provided by bsd.port.mk
Comment 19 Jason W. Bacon freebsd_committer 2021-01-09 20:39:47 UTC
(In reply to Antoine Brodin from comment #18)

Why shouldn't it added somewhere outside compiler.mk?

We have 138 ports using this check according to 

port-grep '.if.*CHOSEN_COMPILER' -l|wc

many of which no longer need gcc.  So we have two choices if we want to eliminate unnecessary gcc deps:

1. Support CHOSEN_COMPILER_TYPE without USES=compiler
2. Alter 138 ports to use a different strategy, like using ARCH and OSVERSION, which would be a maintenance nightmare
Comment 20 Antoine Brodin freebsd_committer 2021-01-09 21:46:21 UTC
(In reply to Jason W. Bacon from comment #19)

Any port can use USES=compiler,  this won't force gcc or clang but will set COMPILER_TYPE, ALT_COMPILER_TYPE, COMPILER_VERSION and ALT_COMPILER_VERSION

Testing for CHOSEN_COMPILER_TYPE without USES=compiler:something doesn't make sense
Comment 21 Jason W. Bacon freebsd_committer 2021-01-09 22:22:00 UTC
(In reply to Antoine Brodin from comment #20)

I see, I didn't realize USES=compiler could be used without any parameters.

In any cases where something better than gcc 4.2 is needed, we can simply replace compiler:openmp with something like compiler:c++11-lang.

Since two compiler: parameters cannot be specified together and both just set USE_GCC anyway, one had to be picked at random.  In most cases, the need for the other is not documented, so that will have to be figured out for each port.

This is the case for biology/canu, which also needs USES=localbase to quell a compile error when building with base clang.
Comment 22 Jason W. Bacon freebsd_committer 2021-01-10 21:52:12 UTC
I've updated all my ports using openmp to eliminate the gcc dep.  Below is a summary of the necessary changes to show some examples of what we're up against.

I plan to do some runtime testing before committing these changes.

===

/usr/ports/biology/canu/Makefile:MAINTAINER=    jwb@FreeBSD.org

Requires both openmp and c++11:

-USES=          compiler:openmp gmake perl5
+USES=          compiler:c++11-lang gmake localbase perl5

No longer needed since replacing compiler:openmp with compiler:c++11-lang:

-.include <bsd.port.pre.mk>
-
-# GCC 4.2.1 (still base compiler on some 2nd tier platforms) cannot build canu
-.if ${COMPILER_TYPE} == gcc && ${COMPILER_VERSION} <= 42
-USE_GCC=       yes
-.endif
-

===

/usr/ports/biology/cd-hit/Makefile:MAINTAINER=  jwb@FreeBSD.org

Requires openmp only:

-USES=          compiler:openmp gmake shebangfix
+USES=          gmake shebangfix

/usr/ports/biology/fasttree/Makefile:MAINTAINER=        jwb@FreeBSD.org

Requires openmp only:

-USES=          compiler:openmp

===

/usr/ports/biology/gcta/Makefile:MAINTAINER=    jwb@FreeBSD.org

Requires both openmp and c++11:

-USES=          compiler:openmp dos2unix eigen:3 gmake localbase zip
+USES=          compiler:c++11-lang dos2unix eigen:3 gmake localbase:ldflags zip

No longer needed since replacing compiler:openmp with compiler:c++11-lang:

-.include <bsd.port.pre.mk>
-
-.if ${CHOSEN_COMPILER_TYPE} == gcc && ${COMPILER_VERSION} <= 42
-USE_GCC=       yes
-.endif
-

===

/usr/ports/biology/ncbi-blast+/Makefile:MAINTAINER=     jwb@FreeBSD.org

Requires both openmp and c++14:

-USES=          compiler:openmp gmake shebangfix perl5 python
+USES=          compiler:c++14-lang gmake shebangfix perl5 python
-LDFLAGS+=      -latomic

-latomic required for gcc, does not work with clang:
 
-# Force newer GCC on platforms using GCC 4.2 as base
 .if ${CHOSEN_COMPILER_TYPE} == gcc
-USE_GCC=       yes
+# In case of problems: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220822
+LDFLAGS+=      -latomic # Only exists in gcc libs
 .endif

===

/usr/ports/biology/stacks/Makefile:MAINTAINER=  jwb@FreeBSD.org

-USES=          compiler:openmp gmake python shebangfix
+USES=          compiler:c++11-lang gmake python shebangfix

clang is missing some atomic functionality on i386:

-# Force newer GCC on platforms using GCC 4.2 as base
-.if ${CHOSEN_COMPILER_TYPE} == gcc
+# Temp workaround, see PR 230888
+.if ${ARCH} == i386
 USE_GCC=       yes
 .endif

===

/usr/ports/biology/star/Makefile:MAINTAINER=    jwb@FreeBSD.org

Requires both openmp and c++11:

-USES=          compiler:openmp gmake
+USES=          compiler:c++11-lang gmake
 
No longer needed since replacing compiler:openmp with compiler:c++11-lang:

-.include <bsd.port.pre.mk>
-
-# Force newer compiler on platforms using GCC 4.2 as base
-.if ${CHOSEN_COMPILER_TYPE} == gcc && ${COMPILER_VERSION} <= 42
-USE_GCC=       yes
-.endif
-

===

/usr/ports/sysutils/parafly/Makefile:MAINTAINER=        jwb@FreeBSD.org

Requires openmp only:

-USES=          compiler:openmp tar:tgz
+USES=          tar:tgz
Comment 23 commit-hook freebsd_committer 2021-01-11 16:51:56 UTC
A commit references this bug:

Author: jwb
Date: Mon Jan 11 16:51:16 UTC 2021
New revision: 561227
URL: https://svnweb.freebsd.org/changeset/ports/561227

Log:
  biology/canu: Drop dependency on gcc

  Replace compiler:openmp with compiler:c++11-lang and localbase

  PR:             port/247753, ports/252379

Changes:
  head/biology/canu/Makefile
Comment 24 commit-hook freebsd_committer 2021-01-11 16:56:04 UTC
A commit references this bug:

Author: jwb
Date: Mon Jan 11 16:55:16 UTC 2021
New revision: 561228
URL: https://svnweb.freebsd.org/changeset/ports/561228

Log:
  biology/cd-hit: Drop dependency on gcc

  Simply remove compiler:openmp
  Builds with base compiler on amd64, i386, and powerpc64

  PR:             port/247753, ports/252379

Changes:
  head/biology/cd-hit/Makefile
Comment 25 commit-hook freebsd_committer 2021-01-11 16:59:10 UTC
A commit references this bug:

Author: jwb
Date: Mon Jan 11 16:58:14 UTC 2021
New revision: 561229
URL: https://svnweb.freebsd.org/changeset/ports/561229

Log:
  biology/fasttree: Drop dependency on gcc

  Simply remove compiler:openmp
  Builds with base compiler on amd64, i386, and powerpc64 12.x-RELEASE

  PR:             port/247753, ports/252379

Changes:
  head/biology/fasttree/Makefile
Comment 26 commit-hook freebsd_committer 2021-01-11 17:10:17 UTC
A commit references this bug:

Author: jwb
Date: Mon Jan 11 17:09:38 UTC 2021
New revision: 561231
URL: https://svnweb.freebsd.org/changeset/ports/561231

Log:
  biology/gcta: Drop dependency on gcc

  Replace compiler:openmp with compiler:c++11-lang
  Clang build also requires localbase:ldflags
  Build tested on amd64, i386 and powerpc64 12.x-RELEASE

  PR:             port/247753, ports/252379

Changes:
  head/biology/gcta/Makefile
Comment 27 commit-hook freebsd_committer 2021-01-11 17:18:24 UTC
A commit references this bug:

Author: jwb
Date: Mon Jan 11 17:17:36 UTC 2021
New revision: 561233
URL: https://svnweb.freebsd.org/changeset/ports/561233

Log:
  biology/ncbi-blast+: Drop dependency on gcc

  Replace compiler:openmp with compiler:c++14-lang
  Clang build also requires localbase:ldflags and a source patch to explicitly
  set variables to shared
  GCC-based platforms still require -latomic

  PR:             port/247753, ports/252379

Changes:
  head/biology/ncbi-blast+/Makefile
  head/biology/ncbi-blast+/files/patch-src_objtools_blast_seqdb__reader_test_seqdb__perf.cpp
Comment 28 commit-hook freebsd_committer 2021-01-11 17:28:31 UTC
A commit references this bug:

Author: jwb
Date: Mon Jan 11 17:27:30 UTC 2021
New revision: 561234
URL: https://svnweb.freebsd.org/changeset/ports/561234

Log:
  biology/stacks: Drop dependency on gcc

  Replace compiler:openmp with compiler:c++11-lang and localbase
  Set USE_GCC for i386 to work around limitations in clang atomic (PR 230888)

  PR:             port/247753, ports/252379

Changes:
  head/biology/stacks/Makefile
Comment 29 commit-hook freebsd_committer 2021-01-11 17:40:39 UTC
A commit references this bug:

Author: jwb
Date: Mon Jan 11 17:39:52 UTC 2021
New revision: 561235
URL: https://svnweb.freebsd.org/changeset/ports/561235

Log:
  biology/star: Drop dependency on gcc

  Replace compiler:openmp with compiler:c++11-lang and localbase

  PR:             port/247753, ports/252379

Changes:
  head/biology/star/Makefile
  head/biology/star/files/patch-Makefile
Comment 30 commit-hook freebsd_committer 2021-01-11 17:43:45 UTC
A commit references this bug:

Author: jwb
Date: Mon Jan 11 17:43:22 UTC 2021
New revision: 561236
URL: https://svnweb.freebsd.org/changeset/ports/561236

Log:
  sysutils/parafly: Upgrade to 0.1.0 from Github, drop gcc dependency

  Simply remove compiler:openmp
  Builds with base compiler on amd64, i386, and powerpc64 12.x-RELEASE

  PR:             port/247753, ports/252379

Changes:
  head/sysutils/parafly/Makefile
  head/sysutils/parafly/distinfo
  head/sysutils/parafly/pkg-descr
Comment 31 commit-hook freebsd_committer 2021-01-11 17:54:50 UTC
A commit references this bug:

Author: jrm
Date: Mon Jan 11 17:54:42 UTC 2021
New revision: 561239
URL: https://svnweb.freebsd.org/changeset/ports/561239

Log:
  biology/hyphy: Remove USES=compiler:openmp

  USES=compiler:openmp still forces building with GCC.  This is no longer
  necessary on tier 1 platforms, which have openmp in the base system.

  PR:		247753, 252379

Changes:
  head/biology/hyphy/Makefile
Comment 32 Joseph Mingrone freebsd_committer 2021-01-11 18:48:12 UTC
biology/iqtree gives linker errors with clang.  I spent a bit of time trying to figure out what's going on, but no luck yet.

http://pkg.awarnach.mathstat.dal.ca/data/12amd64-default/2021-01-11_14h39m54s/logs/errors/iqtree-2.0.6_1.log
Comment 33 VVD 2021-01-11 19:02:46 UTC
(In reply to Joseph Mingrone from comment #32)
Did you tried compile it without ccache and with clang 10?
Comment 34 VVD 2021-01-11 19:32:59 UTC
And openblas-0.3.12_1,1 require gcc, but build fine with system clang 10 from 12.2.
Comment 35 Christoph Moench-Tegeder freebsd_committer 2021-01-11 19:39:18 UTC
(In reply to Joseph Mingrone from comment #32)

The problem with iqtree is that is ships with their private libomp (among others) - precompiled on Linux as lib/libomp.a (there'S also Windows and Mac versions of that, and 32bit version in lib32). That's a stunningly horrible idea...

Try someting along these lines as files/patch-CMakeLists.txt:
--- CMakeLists.txt.orig 2021-01-11 20:24:32.584318000 +0100
+++ CMakeLists.txt      2021-01-11 20:26:49.293185000 +0100
@@ -323,12 +323,12 @@
 
     if(CLANG AND APPLE)
         link_directories(${PROJECT_SOURCE_DIR}/libmac)
-    elseif (WIN32 OR UNIX)
-        if (BINARY32)
-            link_directories(${PROJECT_SOURCE_DIR}/lib32)
-        else()
-            link_directories(${PROJECT_SOURCE_DIR}/lib)
-        endif()
+#    elseif (WIN32 OR UNIX)
+#        if (BINARY32)
+#            link_directories(${PROJECT_SOURCE_DIR}/lib32)
+#        else()
+#            link_directories(${PROJECT_SOURCE_DIR}/lib)
+#        endif()
     endif()
 
     if (VCC)


That makes the port compile with clang for me (but I can't do any runtime tests)
Comment 36 Joseph Mingrone freebsd_committer 2021-01-12 17:58:07 UTC
(In reply to Christoph Moench-Tegeder from comment #35)
Thanks.  I'll use a similar patch to fix the amd64 build.  This one will probably be acceptable upstream.

     if(CLANG AND APPLE)
         link_directories(${PROJECT_SOURCE_DIR}/libmac)
-    elseif (WIN32 OR UNIX)
+    elseif (WIN32 OR UNIX AND NOT ${CMAKE_SYSTEM_NAME} STREQUAL "FreeBSD")
         if (BINARY32)
             link_directories(${PROJECT_SOURCE_DIR}/lib32)
         else()

Switching to clang++ does break i386 builds because base clang doesn't have 64 bit atomic functions for i386 (see bug 230888). I'll probably just mark i386 broken, but if there is a single user running iqtree on i386, then I'll investigate further.
Comment 37 commit-hook freebsd_committer 2021-01-12 18:12:50 UTC
A commit references this bug:

Author: jrm
Date: Tue Jan 12 18:12:13 UTC 2021
New revision: 561368
URL: https://svnweb.freebsd.org/changeset/ports/561368

Log:
  biology/iqtree: Remove USES=compiler:openmp, fix amd64 clang++ build

  - USES=compiler:openmp still forces building with GCC.  This is no longer
    necessary on tier 1 platforms, which have openmp in base.

  - Mark broken on i386, because base clang doesn't have 64 bit atomic
    functions for i386 (see bug 230888).

  PR:		247753, 252379
  Submitted by:	cmt (based on)

Changes:
  head/biology/iqtree/Makefile
  head/biology/iqtree/files/
  head/biology/iqtree/files/patch-CMakeLists.txt