Bug 235211

Summary: dlopen/libthr appears broken
Product: Base System Reporter: Kyle Evans <kevans>
Component: threadsAssignee: freebsd-threads (Nobody) <threads>
Status: Closed FIXED    
Severity: Affects Only Me CC: andrew, i+fbsd, kib
Priority: --- Keywords: patch
Version: CURRENT   
Hardware: Any   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235158

Description Kyle Evans freebsd_committer 2019-01-26 05:41:25 UTC
First observed with Lua and cqueue and reproduced on both 12.0-RELEASE as well as -CURRENT. It was reported that this is a regression from 11 -> 12.

Fairly minimal test case [1]; an application that dlopen() an .so linked against libthr and invokes a function in that .so that creates a new thread. The new thread appears to be created (see [2] for truss output) but there we stall without ever entering thread_start in the new thread.

[1] https://github.com/RhodiumToad/dynthr-test
[2] https://people.freebsd.org/~kevans/dynthr-broken.txt
Comment 1 Kyle Evans freebsd_committer 2019-01-26 05:43:01 UTC
CC kib@ because it looks like an rtld bug, perhaps.
Comment 2 Konstantin Belousov freebsd_committer 2019-01-27 04:52:05 UTC
Comment 3 commit-hook freebsd_committer 2019-01-29 22:47:42 UTC
A commit references this bug:

Author: kib
Date: Tue Jan 29 22:46:46 UTC 2019
New revision: 343566
URL: https://svnweb.freebsd.org/changeset/base/343566

  Untangle jemalloc and mutexes initialization.

  The need to use libc malloc(3) from some places in libthr always
  caused issues.  For instance, per-thread key allocation was switched to
  use plain mmap(2) to get storage, because some third party mallocs
  used keys for implementation of calloc(3).

  Even more important, libthr calls calloc(3) during initialization of
  pthread mutexes, and jemalloc uses pthread mutexes.  Jemalloc provides
  some way to both postpone the initialization, and to make
  initialization to use specialized allocator, but this is very fragile
  and often breaks.  See the referenced PR for another example.

  Add the small malloc implementation used by rtld, to libthr. Use it in
  thr_spec.c and for mutexes initialization. This avoids the issues with
  mutual dependencies between malloc and libthr in principle.  The
  drawback is that some more allocations are not interceptable for
  alternate malloc implementations.  There should be not too much memory
  use from this allocator, and the alternative, direct use of mmap(2) is
  obviously worse.

  PR:	235211
  MFC after:	2 weeks
  Sponsored by:	The FreeBSD Foundation
  Differential revision:	https://reviews.freebsd.org/D18988

Comment 4 andrew 2019-02-04 11:09:01 UTC
There's a report on the ARM list of a crash in /rescue/* on armv7 pointing at this new code. Offhand, isn't it a problem that handle_static_init calls atexit() which calls pthread_mutex_lock, before libthr's initialization is run?
Comment 6 Kyle Evans freebsd_committer 2020-02-21 13:50:37 UTC
The follow-up /rescue breakage was fixed by r343754 -- atexit appears to not be an issue.