Bug 235211 - dlopen/libthr appears broken
Summary: dlopen/libthr appears broken
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: threads (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-threads (Nobody)
URL:
Keywords: patch
Depends on:
Blocks:
 
Reported: 2019-01-26 05:41 UTC by Kyle Evans
Modified: 2020-02-21 13:50 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Kyle Evans freebsd_committer freebsd_triage 2019-01-26 05:41:25 UTC
First observed with Lua and cqueue and reproduced on both 12.0-RELEASE as well as -CURRENT. It was reported that this is a regression from 11 -> 12.

Fairly minimal test case [1]; an application that dlopen() an .so linked against libthr and invokes a function in that .so that creates a new thread. The new thread appears to be created (see [2] for truss output) but there we stall without ever entering thread_start in the new thread.

[1] https://github.com/RhodiumToad/dynthr-test
[2] https://people.freebsd.org/~kevans/dynthr-broken.txt
Comment 1 Kyle Evans freebsd_committer freebsd_triage 2019-01-26 05:43:01 UTC
CC kib@ because it looks like an rtld bug, perhaps.
Comment 2 Konstantin Belousov freebsd_committer freebsd_triage 2019-01-27 04:52:05 UTC
https://reviews.freebsd.org/D18988
Comment 3 commit-hook freebsd_committer freebsd_triage 2019-01-29 22:47:42 UTC
A commit references this bug:

Author: kib
Date: Tue Jan 29 22:46:46 UTC 2019
New revision: 343566
URL: https://svnweb.freebsd.org/changeset/base/343566

Log:
  Untangle jemalloc and mutexes initialization.

  The need to use libc malloc(3) from some places in libthr always
  caused issues.  For instance, per-thread key allocation was switched to
  use plain mmap(2) to get storage, because some third party mallocs
  used keys for implementation of calloc(3).

  Even more important, libthr calls calloc(3) during initialization of
  pthread mutexes, and jemalloc uses pthread mutexes.  Jemalloc provides
  some way to both postpone the initialization, and to make
  initialization to use specialized allocator, but this is very fragile
  and often breaks.  See the referenced PR for another example.

  Add the small malloc implementation used by rtld, to libthr. Use it in
  thr_spec.c and for mutexes initialization. This avoids the issues with
  mutual dependencies between malloc and libthr in principle.  The
  drawback is that some more allocations are not interceptable for
  alternate malloc implementations.  There should be not too much memory
  use from this allocator, and the alternative, direct use of mmap(2) is
  obviously worse.

  PR:	235211
  MFC after:	2 weeks
  Sponsored by:	The FreeBSD Foundation
  Differential revision:	https://reviews.freebsd.org/D18988

Changes:
  head/lib/libthr/Makefile
  head/lib/libthr/thread/Makefile.inc
  head/lib/libthr/thread/thr_fork.c
  head/lib/libthr/thread/thr_init.c
  head/lib/libthr/thread/thr_malloc.c
  head/lib/libthr/thread/thr_mutex.c
  head/lib/libthr/thread/thr_private.h
  head/lib/libthr/thread/thr_spec.c
Comment 4 Andrew "RhodiumToad" Gierth 2019-02-04 11:09:01 UTC
There's a report on the ARM list of a crash in /rescue/* on armv7 pointing at this new code. Offhand, isn't it a problem that handle_static_init calls atexit() which calls pthread_mutex_lock, before libthr's initialization is run?
Comment 5 Andrew "RhodiumToad" Gierth 2019-02-04 12:18:20 UTC
link to message:

https://docs.freebsd.org/cgi/mid.cgi?0C8976C2-8888-4340-84A3-F0F6F46BDD6D
Comment 6 Kyle Evans freebsd_committer freebsd_triage 2020-02-21 13:50:37 UTC
The follow-up /rescue breakage was fixed by r343754 -- atexit appears to not be an issue.