Bug 216745

Summary: devel/boost-libs: atomics are broken with clang 4.0 on i386
Product: Ports & Packages Reporter: Jan Beich <jbeich>
Component: Individual Port(s)Assignee: FreeBSD Office Team <office>
Status: Closed FIXED    
Severity: Affects Only Me CC: dbaio, dim, emaste, fernape, kib, office, rezny, yuri
Priority: --- Keywords: needs-patch
Version: Latest   
Hardware: i386   
OS: Any   
See Also: https://llvm.org/bugs/show_bug.cgi?id=31864
Bug Depends on:    
Bug Blocks: 216008    
Description Flags
src/performance_counters.cpp (preprocessed diff)
simple test case
Don't use implied zero displacements in boost's inline asm none

Description Jan Beich freebsd_committer 2017-02-03 02:07:46 UTC
linking qbittorrent
/usr/local/lib/libtorrent-rasterbar.so: undefined reference to `__atomic_fetch_add_8'
/usr/local/lib/libtorrent-rasterbar.so: undefined reference to `__atomic_store_8'
/usr/local/lib/libtorrent-rasterbar.so: undefined reference to `__atomic_load_8'
/usr/local/lib/libtorrent-rasterbar.so: undefined reference to `__atomic_compare_exchange_8'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Comment 1 Jan Beich freebsd_committer 2017-02-03 02:17:04 UTC
Created attachment 179554 [details]
src/performance_counters.cpp (preprocessed diff)
Comment 2 Yuri Victorovich freebsd_committer 2017-02-03 02:31:46 UTC
This looks like a compiler bug, isn't it?
Comment 3 Jan Beich freebsd_committer 2017-02-03 02:33:31 UTC
Comment on attachment 179554 [details]
src/performance_counters.cpp (preprocessed diff)

Boost.Atomic trips up on the following change:

$ clang39 -dM -E -</dev/null 2>&1 | fgrep __GCC_ATOMIC_LLONG_LOCK_FREE

$ cc -dM -E -</dev/null 2>&1 | fgrep __GCC_ATOMIC_LLONG_LOCK_FREE
Comment 4 Jan Beich freebsd_committer 2017-02-03 03:18:25 UTC
Dimitry, can you bisect or check if https://reviews.llvm.org/D28213 caused the above behavior? I'm not sure how to fix Boost...
Comment 5 Jan Beich freebsd_committer 2017-02-03 03:37:28 UTC
Created attachment 179555 [details]
simple test case

$ pkg install boost-libs llvm39 llvm40
$ export CPATH=/usr/local/include LIBRARY_PATH=/usr/local/lib
$ clang++39 -m32 boost_atomic.test.cc -lboost_atomic
$ clang++40 -m32 boost_atomic.test.cc -lboost_atomic
boost_atomic-850574.o: In function `main':
boost_atomic.test.cc:(.text+0x2ba): undefined reference to `__atomic_store_8'
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
Comment 6 Jan Beich freebsd_committer 2017-02-03 03:51:07 UTC
*** Bug 216746 has been marked as a duplicate of this bug. ***
Comment 7 Dimitry Andric freebsd_committer 2017-02-03 18:34:44 UTC
(In reply to Jan Beich (mail not working) from comment #4)
> Dimitry, can you bisect or check if https://reviews.llvm.org/D28213 caused
> the above behavior? I'm not sure how to fix Boost...

I don't think we should fix Boost, as gcc on i386 also seems to set __GCC_ATOMIC_LLONG_LOCK_FREE to 1 on FreeBSD:

    $ gcc6 -v
    Using built-in specs.
    Target: i386-portbld-freebsd12.0
    Configured with: /wrkdirs/usr/ports/lang/gcc6/work/gcc-6.3.0/configure --disable-multilib --disable-bootstrap --disable-nls --enable-gnu-indirect-function --libdir=/usr/local/lib/gcc6 --libexecdir=/usr/local/libexec/gcc6 --program-suffix=6 --with-as=/usr/local/bin/as --with-gmp=/usr/local --with-gxx-include-dir=/usr/local/lib/gcc6/include/c++/ --with-ld=/usr/local/bin/ld --with-pkgversion='FreeBSD Ports Collection' --with-system-zlib --disable-libgcj --enable-languages=c,c++,objc,fortran --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/man --infodir=/usr/local/info/gcc6 --build=i386-portbld-freebsd12.0
    Thread model: posix
    gcc version 6.3.0 (FreeBSD Ports Collection)

    $ gcc6 -dM -E -x c /dev/null | grep __GCC_ATOMIC_LLONG_LOCK_FREE

I have posted a similar comment on the LLVM review: 

but since it is already committed, I might have to open an upstream bug report.
I would like to have a thorough explanation though.

Maybe on Linux, there are only 32 bit x86 arches left that support CMPXCHG8B?  I think we still support those, but we don't have __atomic_xxx_8 functions in our libs to handle the cases where there isn't hardware support.
Comment 8 Dimitry Andric freebsd_committer 2017-02-04 20:18:37 UTC
Submitted upstream bug https://llvm.org/bugs/show_bug.cgi?id=31864, for which there is a fix in review: https://reviews.llvm.org/D29542
Comment 9 Konstantin Belousov freebsd_committer 2017-02-04 20:29:19 UTC
(In reply to Dimitry Andric from comment #7)
I believe that linux still supports 486.  More likely, the define means that gcc on linux defaults to pentium or even to 686.

You arguably do not want to run any office package on 486-class machine.  Putting aside the clang bug, I think that the practical solution, which also works on existing releases, is to add -march=pentiumpro (or whatever it spelled) to the CFLAGS and declare that port is not supported on any lesser CPU.
Comment 10 Dimitry Andric freebsd_committer 2017-02-15 18:30:29 UTC
Created attachment 180022 [details]
Don't use implied zero displacements in boost's inline asm

I would like to propose a two-pronged solution to this.

1) Let clang default to i586 on 32-bit x86, supporting cmpxchg8b and __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8, since in reality we have been doing this for years already.  That is, since clang was introduced in the tree, it has always generated cmpxchg8b opcodes for 64-bit atomic operations, even though it didn't set __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8.  We have also never had __atomic_xxx_8() functions for 32-bit x86 in our libc, or in a libatomic support library, and that is the reason for the link error in comment 0.

2) Apply the attached patch, so boost stops using the "movl 4+(%%edx), %%eax" inline assembly syntax, which does not work with clang.  The code path using this inline assembly is activated when __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 is defined.  (Note that this so-called "implied zero displacement" syntax also didn't work with GNU as from the binutils port, but when gcc processes the inline assembly, it might add some additional offset, making it valid).

Alternatively, we can try to find some sort of other workaround in the boost headers themselves, since it appears to emit some sort of mutex locking when you compile it with gcc 4.2.1, or other very old compilers.  This is rather ugly, though, and pessimizes all post i486 processors for the sake of real i486's.
Comment 11 Dimitry Andric freebsd_committer 2017-02-24 15:44:57 UTC
For now, upstream has reverted https://reviews.llvm.org/rL291477, and this will be merged into the release_40 branch.  So until some later time, this issue will not occur anymore.

I will update this bug as soon as the upstream revert is in the projects/clang400-import branch.
Comment 12 Dimitry Andric freebsd_committer 2017-02-25 22:27:22 UTC
As of r314269 the projects/clang400-import branch contains clang 4.0.0 r296202, which has the problematic upstream changed reverted.  I've tested this by rebuilding boost-libs and qbittorrent, but independent verification would be appreciated. :)
Comment 13 Jan Beich freebsd_committer 2017-02-27 16:02:29 UTC
I confirm. graphics/appleseed and net-p2p/qbittorrent build fine on clang 4.0 i386.
Comment 14 Danilo G. Baio freebsd_committer 2018-05-14 02:21:18 UTC

I am getting this error when building dns/knot2 (bug #227620) only in FreeBSD 12 i386, clang 6.0.0:

./.libs/libknotd.a(libknotd_la-query_module.o): In function `knotd_mod_stats_incr':
query_module.c:(.text+0x9d0): undefined reference to `__atomic_fetch_add_8'
./.libs/libknotd.a(libknotd_la-query_module.o): In function `knotd_mod_stats_decr':
query_module.c:(.text+0xa80): undefined reference to `__atomic_fetch_sub_8'
./.libs/libknotd.a(libknotd_la-query_module.o): In function `knotd_mod_stats_store':
query_module.c:(.text+0xb30): undefined reference to `__atomic_store_8'
cc: error: linker command failed with exit code 1 (use -v to see invocation)

Could it be a similar issue?
Comment 15 Fernando ApesteguĂ­a freebsd_committer 2018-05-30 15:10:06 UTC
Similar problem trying to build lang/ponyc on 10.4 amd64 (bug #228549)

c++ -o build/release/libponyrt.tests build/release/obj/tests/libponyrt/util.o build/release/obj/tests/libponyrt/ds/list.o build/release/obj/tests/libponyrt/ds/f
un.o build/release/obj/tests/libponyrt/ds/hash.o build/release/obj/tests/libponyrt/mem/pagemap.o build/release/obj/tests/libponyrt/mem/heap.o build/release/obj/tests/libponyrt/mem/pool.o -march=native -mtune=generic -mcx16 -L build/release -L /usr/local/lib  -lgtest -lponyrt -lpthread -rdynamic
build/release/libponyrt.a(pool.o): In function `pool_get':
src/libponyrt/mem/pool.c:(.text+0xbb): undefined reference to `__atomic_compare_exchange'
build/release/libponyrt.a(pool.o): In function `ponyint_pool_free':
src/libponyrt/mem/pool.c:(.text+0x221): undefined reference to `__atomic_compare_exchange'
src/libponyrt/mem/pool.c:(.text+0x26f): undefined reference to `__atomic_compare_exchange'
build/release/libponyrt.a(pool.o): In function `ponyint_pool_thread_cleanup':
src/libponyrt/mem/pool.c:(.text+0x7ef): undefined reference to `__atomic_compare_exchange'
src/libponyrt/mem/pool.c:(.text+0x8c0): undefined reference to `__atomic_compare_exchange'
build/release/libponyrt.a(pool.o):src/libponyrt/mem/pool.c:(.text+0x900): more undefined references to `__atomic_compare_exchange' follow
c++: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[1]: *** [Makefile:771: build/release/libponyrt.tests] Error 1
gmake[1]: *** Waiting for unfinished jobs....
gmake[1]: Leaving directory '/wrkdirs/usr/ports/lang/ponyc/work/ponyc-0.21.0'
===> Compilation failed unexpectedly.