Bug 192320 - Use of thread_local produces linking errors on system version of clang++
Summary: Use of thread_local produces linking errors on system version of clang++
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 10.0-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: Konstantin Belousov
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-08-01 22:09 UTC by Troy Heron
Modified: 2016-08-22 16:38 UTC (History)
13 users (show)

See Also:
koobs: mfc-stable11?
koobs: mfc-stable10?
koobs: mfc-stable9?


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Troy Heron 2014-08-01 22:09:39 UTC
During porting of a C++ application to FreeBSD I have discovered problem with using thread_local with the system version of Clang 3.3.

As below, it's use produces linking errors to the C++ ABI.

troy@freebsd10:~ # cat tl.cpp
#include <string>

thread_local std::string test;

int main()
{
}

troy@freebsd10:~ # clang++ -std=c++11 tl.cpp -o tl 
/tmp/tl-7sN7Nb.o: In function `__cxx_global_var_init':
tl.cpp:(.text+0xbb): undefined reference to `__cxa_thread_atexit'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
troy@freebsd10:~ # 

troy@freebsd10:~ # clang++ -v
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
Target: x86_64-unknown-freebsd10.0
Thread model: posix
troy@freebsd10:~ #
Comment 1 Dimitry Andric freebsd_committer 2014-08-02 10:27:12 UTC
Our __cxa_atexit() is in lib/libc/stdlib/atexit.c, but apparently we don't have a corresponding __cxa_thread_atexit() yet.  I'm not entirely sure where it should live, though: GNU puts it in libsupc++, so for use that would correspond to libcxxrt...
Comment 2 David Chisnall freebsd_committer 2014-08-02 14:41:07 UTC
I did write a __cxa_thread_atexit a while ago, but I haven't yet committed it (I'll try to find it).  I didn't commit it because the semantics of _Thread_local are very poorly defined in the presence of shared libraries.  I could think of three wrong ways of implementing it (one of which was chosen by libsupc++), but no right way.
Comment 3 Steve Wills freebsd_committer 2015-02-18 21:06:21 UTC
(In reply to David Chisnall from comment #2)

Any status update on this? I've run into this issue too.
Comment 4 David Chisnall freebsd_committer 2015-02-18 21:30:35 UTC
I'm still not completely sure what the correct solution for implementing this is.  

Clang will always lazily construct thread_local objects on first access (which means that you can't use them to track thread counts and so on, which limits their use somewhat), but the destructors will run when the thread exits.

Unfortunately, the destructors might not be signal safe and so there is no *good* way of running them on library unload.  A complete solution most likely involves locking the library in memory while it has outstanding thread-local objects.
Comment 5 Tijl Coosemans freebsd_committer 2015-02-19 08:57:04 UTC
(In reply to David Chisnall from comment #4)
That's what glibc seems to do.  When a thread calls dlclose its thread_local variables are destroyed, but if there are other threads with thread_local variables the DSO is kept in memory until those variables are destroyed (when the threads exit or call dlclose).

https://sourceware.org/glibc/wiki/Destructor%20support%20for%20thread_local%20variables
Comment 6 Rudolf-Walter Kiss-Szakacs 2015-10-28 19:26:19 UTC
I have also encountered this while trying to port something to FreeBSD. Is there no news?
Comment 7 Duraid Madina 2015-11-27 06:03:39 UTC
(In reply to David Chisnall from comment #4)

> A complete solution most likely involves locking the library in memory while it has outstanding thread-local objects.

This sounds like a perfectly decent solution - is it something you are able to implement? This bug is going to bite more frequently as more C++11 is written. :(
Comment 8 Hannes Hauswedell 2016-01-06 17:55:28 UTC
Indeed, a fix would be great! Thank you.
Comment 9 Mikhail Teterin freebsd_committer 2016-06-23 17:52:04 UTC
Please, be sure to bump OSVERSION, when this is committed -- so that the various work-arounds implemented in the ports in the mean time can be made conditional... Thanks!
Comment 10 Hannes Hauswedell 2016-08-12 14:18:20 UTC
This issue is now two years old and I have to work around it more and more often. If a "good solution" is so difficult to make, can we at least have some workaround? 

Thank you very much!
Comment 11 David Chisnall freebsd_committer 2016-08-12 14:20:49 UTC
Kib committed the a fix for this to head recently.
Comment 12 Kubilay Kocak freebsd_committer freebsd_triage 2016-08-13 06:47:32 UTC
Assign to committer that resolved.

@Kib, can you add a reference to the change please, and does this need or has it been MFC'd?

Set mfc-stable* flags to + where/when merged, or to - with comment if not required
Comment 13 Bryan Drewery freebsd_committer 2016-08-20 17:53:00 UTC
r303795 | kib | 2016-08-06 06:32:40 -0700 (Sat, 06 Aug 2016) | 37 lines
Changed paths:
   M /head/lib/libc/include/libc_private.h
   M /head/lib/libc/stdlib/Makefile.inc
   M /head/lib/libc/stdlib/Symbol.map
   A /head/lib/libc/stdlib/cxa_thread_atexit.c
   M /head/lib/libc/stdlib/exit.c
   M /head/lib/libc/tests/stdlib/Makefile
   M /head/lib/libc/tests/stdlib/Makefile.depend
   A /head/lib/libc/tests/stdlib/cxa_thread_atexit_nothr_test.cc
   A /head/lib/libc/tests/stdlib/cxa_thread_atexit_test.cc
   M /head/lib/libthr/thread/thr_exit.c

Add __cxa_thread_atexit(3) API implementation.

This is the backing feature to implement C++11 thread storage duration
specified by the thread_local keyword.  A destructor for given
thread-local object is registered to be executed at the thread
termination time using __cxa_thread_atexit().  Libc calls the
__cxa_thread_calls_dtors() during exit(3), before finalizers and
atexit functions, and libthr calls the function at the thread
termination time, after the stack unwinding and thread-specific key
destruction.

There are several uncertainties in the API which lacks a formal
specification.  Among them:
- is it allowed to register destructors during destructing;
        we allow, but limiting the nesting level.  If too many iterations
        detected, a diagnostic is issued to stderr and thread forcibly
        terminates for now.
- how to handle destructors which belong to an unloading dso;
        for now, we ignore destructor calls for such entries, and
        issue a diagnostic.  Linux does prevent dso unload until all
        threads with destructors from the dso terminated.
It is supposed that the diagnostics allow to detect real-world
applications relying on the above details and possibly adjust
our implementation.  Right now the choices were to provide the slim
API (but that rarely stands the practice test).

Tests are added to check generic functionality and to specify some of
the above implementation choices.

Submitted by:   Mahdi Mokhtari <mokhi64_gmail.com>
Reviewed by:    theraven
Discussed with: dim (detection of -std=c++11 supoort for tests)
Sponsored by:   The FreeBSD Foundation (my involvement)
MFC after:      2 weeks
Differential revisions: https://reviews.freebsd.org/D7224,
    https://reviews.freebsd.org/D7427
Comment 14 Mikhail Teterin freebsd_committer 2016-08-21 01:16:54 UTC
Some of the ports could benefit from knowing, the functionality is available. Can the OSVERSION be bumped to reflect it? Thanks!
Comment 15 commit-hook freebsd_committer 2016-08-22 15:52:44 UTC
A commit references this bug:

Author: bdrewery
Date: Mon Aug 22 15:52:04 UTC 2016
New revision: 304608
URL: https://svnweb.freebsd.org/changeset/base/304608

Log:
  Bump __FreeBSD_version for C++11 thread_local support in r303795.

  PR:		192320

Changes:
  head/sys/sys/param.h
Comment 16 commit-hook freebsd_committer 2016-08-22 15:53:48 UTC
A commit references this bug:

Author: bdrewery
Date: Mon Aug 22 15:53:33 UTC 2016
New revision: 304609
URL: https://svnweb.freebsd.org/changeset/base/304609

Log:
  MFC r304608:

    Bump __FreeBSD_version for C++11 thread_local support in r303795.

  PR:	        192320

Changes:
_U  stable/11/
  stable/11/sys/sys/param.h
Comment 17 commit-hook freebsd_committer 2016-08-22 16:04:52 UTC
A commit references this bug:

Author: bdrewery
Date: Mon Aug 22 16:04:25 UTC 2016
New revision: 304610
URL: https://svnweb.freebsd.org/changeset/base/304610

Log:
  MFS r304609:

    MFC r304608:

      Bump __FreeBSD_version for C++11 thread_local support in r303795.

  PR:             192320
  Approved by:	re (gjb)

Changes:
_U  releng/11.0/
  releng/11.0/sys/sys/param.h
Comment 18 commit-hook freebsd_committer 2016-08-22 16:35:58 UTC
A commit references this bug:

Author: bdrewery
Date: Mon Aug 22 16:35:51 UTC 2016
New revision: 304611
URL: https://svnweb.freebsd.org/changeset/base/304611

Log:
  MFC r304608:

    Bump __FreeBSD_version for C++11 thread_local support in r303795.

  PR:	        192320

Changes:
_U  stable/10/
  stable/10/sys/sys/param.h