Bug 233495

Summary: devel/llvm: LLVM ERROR: Inconsistency in commandline options with multiple OpenCL vendor libraries installed
Product: Ports & Packages Reporter: O. Hartmann <ohartmann>
Component: Individual Port(s)Assignee: freebsd-ports-bugs (Nobody) <ports-bugs>
Status: New ---    
Severity: Affects Many People CC: jcfyecrayz, ohartmann
Priority: --- Flags: ohartmann: maintainer-feedback+
Version: Latest   
Hardware: Any   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224584
Bug Depends on:    
Bug Blocks: 223879, 232357, 219562, 231226, 241500    

Description O. Hartmann 2018-11-25 09:38:11 UTC
I'd like to refer to this bug which seems to be still open and still present in all versions of LLVM since LLVM 4.0 (my guess, I did not track that down exactly).

Since this severe problem has not been solved even for LLVM 7.0, the issue not being able to use an ICD for multiple platforms available on FreeBSD in cases the OpenCL backends (lang/pocl (See https://github.com/pocl/pocl/issues/474), devel/beignet, lang/clover) are all compiled dynamically against the same LLVM backend. The issue disapperas were the backend is compiled against a different version of LLVM or statically linked. Both is inacceptable.

I'd like to refer to this LLVM bug:



Bug 30587 - Inconsistency in commandline options with multiple OpenCL vendor libraries installed :


How to reproduce:

Compile lang/clover, lang/beignet, lang/pocl against the same LLVM backend (devel/llvm60 or devel/llvm70) and use /devel/clinfo for checking. You should be presented with:

$ clinfo
CommandLine Error: Option 'enable-value-profiling' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options

The same problem occurs for several ports on which OpenCL support might be an option and enabled, like graphics/blender (PR 223879), graphics/ImageMagick6 (PR 232357).
Comment 1 Brooks Davis freebsd_committer 2018-11-26 18:39:47 UTC
The message "Inconsistency in commandline options" is just a symptom of the fact that libLLVM can't be dlopen'd twice.  My suspicion is that if libOpenCL had a direct dependency on libLLVM.so that the right thing would happen, but I'm not sure.

Until upstream LLVM makes changes to support this, I don't consider this a bug in the LLVM ports.  OpenCL ports want to use LLVM in a way that isn't supported.
Comment 2 Brooks Davis freebsd_committer 2019-01-10 05:11:10 UTC
*** Bug 231228 has been marked as a duplicate of this bug. ***
Comment 3 O. Hartmann 2019-10-26 14:30:20 UTC
*** Bug 233494 has been marked as a duplicate of this bug. ***
Comment 4 O. Hartmann 2019-10-26 14:57:47 UTC
This problem seems to be a real pain in the ass.

I'm with 12-STABLE (12.1-STABLE #42 r354111: Sat Oct 26 06:19:21 CEST 2019 amd64) and 13-CURRENT (FreeBSD 13.0-CURRENT #62 r354083: Fri Oct 25 19:05:54 CEST 2019 am64), ports tree is at r515672.

It is NOT possible to have multiple OpenCL vendor libraries installed, of which each vendor OpenCL library is dynamically linked against LLVM 9.0.0 (or LLVM 8.0.0, it started with LLVM 4.0.0), for instance

lang/clover: AMD GPUs
lang/beignet: Intel iGPUs
lang/pocl: CPU

Using devel/ocl-icd ICD to dynamically select/switch the propper/fastest platform/device provided or even utilize all of them, if available, isn't possible on recent FreeBSD platforms. Even worse, some applications, like graphics/blender, also quit working with the error shown below if more than one OpenCL venodr library is linked against the same LLVM backend and installed:

# clinfo
: CommandLine Error: Option 'limited-coverage-experimental' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
Nothing to output !

The option 'limited-coverage-experimental' can be replaced with other options issued and depends on which one is issued first.

The problem is reported upstream (LLVM) here


and mentioned for POCL here


As long as lang/clover and lang/beignet are installed and linked against different version of LLVM, the problem doesn't reveal itself. As of today and with the reported revision of the portstree (see above), checking the installation of lang/beignet with

# ldd /usr/local/lib/beignet/libgbe.so
        libdrm_intel.so.1 => /usr/local/lib/libdrm_intel.so.1 (0x800691000)
        libdrm.so.2 => /usr/local/lib/libdrm.so.2 (0x8006b9000)
        libclang-cpp.so.9 => /usr/local/llvm90/lib/libclang-cpp.so.9 (0x801000000)
        libLLVM-9.so => /usr/local/llvm90/lib/libLLVM-9.so (0x803600000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x8006d0000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x8007ae000)
        libm.so.5 => /lib/libm.so.5 (0x800f8d000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x8007d5000)
        libthr.so.3 => /lib/libthr.so.3 (0x800fc0000)
        libc.so.7 => /lib/libc.so.7 (0x80024d000)
        libpciaccess.so.0 => /usr/local/lib/libpciaccess.so.0 (0x8007ec000)
        libedit.so.0 => /usr/local/lib/libedit.so.0 (0x8035a0000)
        libz.so.6 => /lib/libz.so.6 (0x8035d9000)
        librt.so.1 => /usr/lib/librt.so.1 (0x8007f6000)
        libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x800fed000)
        libncurses.so.8 => /lib/libncurses.so.8 (0x8079e1000)
        libelf.so.2 => /lib/libelf.so.2 (0x807a3f000)

shows ibLLVM-9.so 

and lang/clover gives

# ldd /usr/local/lib/libMesaOpenCL.so
        libdrm.so.2 => /usr/local/lib/libdrm.so.2 (0x800691000)
        libz.so.6 => /lib/libz.so.6 (0x8006a8000)
        libexpat.so.1 => /usr/local/lib/libexpat.so.1 (0x8006c4000)
        libelf.so.2 => /lib/libelf.so.2 (0x8006f1000)
        libLLVM-8.so => /usr/local/llvm80/lib/libLLVM-8.so (0x802400000)
        libthr.so.3 => /lib/libthr.so.3 (0x80070e000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x80225f000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x80073b000)
        libm.so.5 => /lib/libm.so.5 (0x800762000)
        libc.so.7 => /lib/libc.so.7 (0x80024d000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x800795000)
        libedit.so.0 => /usr/local/lib/libedit.so.0 (0x8007ac000)
        librt.so.1 => /usr/lib/librt.so.1 (0x8007e5000)
        libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x8007ed000)
        libncurses.so.8 => /lib/libncurses.so.8 (0x80233d000)

it is ibLLVM-8.so 

and for lang/pocl it is

# ldd /usr/local/lib/libpocl.so
        libdl.so.1 => /usr/lib/libdl.so.1 (0x800692000)
        libhwloc.so.5 => /usr/local/lib/libhwloc.so.5 (0x800696000)
        libm.so.5 => /lib/libm.so.5 (0x8006ca000)
        libLLVM-9.so => /usr/local/llvm90/lib/libLLVM-9.so (0x802a00000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x8006fd000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x802958000)
        libthr.so.3 => /lib/libthr.so.3 (0x80297f000)
        libc.so.7 => /lib/libc.so.7 (0x80024d000)
        libpciaccess.so.0 => /usr/local/lib/libpciaccess.so.0 (0x8007db000)
        libxml2.so.2 => /usr/local/lib/libxml2.so.2 (0x806de1000)
        libedit.so.0 => /usr/local/lib/libedit.so.0 (0x8029ac000)
        libz.so.6 => /lib/libz.so.6 (0x806f7e000)
        librt.so.1 => /usr/lib/librt.so.1 (0x8007e5000)
        libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x8007ed000)
        libncurses.so.8 => /lib/libncurses.so.8 (0x806f9a000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x8029e5000)
        liblzma.so.5 => /usr/lib/liblzma.so.5 (0x806ff8000)
        libelf.so.2 => /lib/libelf.so.2 (0x807027000)

and again libLLVM-9.so.

It is easy to confirm the phenomenon with devel/clinfo or benchmarks/clpeak. 

Having installed lang/pocl, lang/beignet and lang/clover with the dependencies shown above, no OpenCL related application works and throwing the error shown above. Leaving lang/pocl (LLVM-9) and lang/clover (LLVM-8), deleting lang/beignet (LLVM-9) makes the environment working again as expected. Compiling lang/pocl against devel/llvm80, makes lang/clover to be deleted to make the environment working again as expected, which can be easily prooven.

I consider the problem a serious one and it will popp up the day lang/clover AND lang/beignet refer to the same LLVM backend one day.