Comment 1 Jan Beich 2017-07-16 15:26:18 UTC

Created attachment 184402 [details]
"bt full" output

Comment 2 Alexey Dokuchaev 2018-01-22 06:51:50 UTC

The same thing happens with `games/quake2lnx'.  Similarly, if I link it explicitly against libpthread or run it as ``env LD_PRELOAD=/lib/libthr.so.3 quake2 ...'' it starts normally.

I'm reluctant to make changes to the port though before we know what particular change in jemalloc is breaking previously working code (ports), and if that code just worked by accident, or the bug had sneaked into jemalloc 5.0.0.

Comment 3 Alexey Dokuchaev 2018-01-22 14:09:23 UTC

Last two comments in https://github.com/jemalloc/jemalloc/issues/907 (October 10th-ish) might or might not be related.

Comment 4 Jan Beich 2018-01-23 05:03:17 UTC

Do we have tests for base r276630? For one, jemalloc 5.0.0 changed

  bool
  malloc_mutex_first_thread(void)
  {

  #ifdef JEMALLOC_MUTEX_INIT_CB
	  postpone_init = false;
	  while (postponed_mutexes != NULL) {
		  if (_pthread_mutex_init_calloc_cb(&postponed_mutexes->lock,
		      bootstrap_calloc) != 0)
			  return (true);
		  postponed_mutexes = postponed_mutexes->postponed_next;
	  }
  #endif
	  return (false);
  }

into

  bool
  malloc_mutex_first_thread(void) {

  #ifndef JEMALLOC_MUTEX_INIT_CB
	  return (malloc_mutex_first_thread());
  #else
	  return (false);
  #endif
  }

Comment 5 Alexey Dokuchaev 2018-01-23 09:55:50 UTC

(In reply to Jan Beich from comment #4)

FWIW, I've tried to play with malloc_mutex_first_thread() implementation, JEMALLOC_MUTEX_INIT_CB (+ some related macros), using different atomics implementation in jemalloc 5.0.0, all without any luck.

For the record, I'm including this advice from John Baldwin:
> I wonder if it is tripping over pthread_once not working in libc.  The stub
> for pthread_once in libc is a nop and has been for a long time.  I added
> a functioning stub (called _libc_once) to libc for it's internal use (there
> is a _once() wrapper that will call _pthread_once when using libpthread and
> _libc_once otherwise).  It might be interesting to build jemalloc with
> '-Dpthread_once=_once' to see if that makes a difference?
I've tried this as well and it did not help.

Comment 6 Jason Evans 2018-05-11 18:44:03 UTC

I don't have any specific ideas as to why this is happening.  Bootstrapping jemalloc on FreeBSD is really tricky, because pthreads mutexes use malloc, and jemalloc has to jump through hoops to initialize its mutexes in multiple phases.

It looks like beignet depends on the libpthread-stubs package.  Is it possible that the stubs don't sufficiently stub out the libc-internal hooks that jemalloc uses to bootstrap?  Here's a list of pthreads-related symbols jemalloc depends on (generated by building stand-alone jemalloc):

_pthread_mutex_init_calloc_cb
je_pthread_create_wrapper
pthread_cond_init
pthread_cond_signal
pthread_cond_timedwait
pthread_cond_wait
pthread_create
pthread_create_fptr
pthread_join
pthread_mutex_lock
pthread_mutex_trylock
pthread_mutex_unlock
pthread_sigmask

Comment 7 Jan Beich 2018-05-13 05:46:04 UTC

(In reply to Jason Evans from comment #6)
Some facts:
- libpthread-stubs is an alias to -pthread since ports r437182
- beignet doesn't use libpthread-stubs
- mesa-libs uses libpthread-stubs
- beignet optionally uses mesa-libs for cl_khr_gl_sharing
- libcl (from beignet) is already linked against libpthread
- ocl-icd uses dlopen() to pull libcl
- clinfo isn't linked against libpthread

After disabling beignet -> mesa-libs dependency clinfo crashes if not passed  LD_PRELOAD=/lib/libthr.so.3:

$ clinfo
Number of platforms                               1
  Platform Name                                   Intel Gen OCL Driver
  Platform Vendor                                 Intel
  Platform Version                                OpenCL 2.0 beignet 1.3
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_3d_image_writes cl_khr_image2d_from_buffer cl_khr_depth_images cl_khr_spir cl_khr_icd cl_intel_accelerator cl_intel_subgroups cl_intel_subgroups_short
  Platform Extensions function suffix             Intel
[New LWP 101911 of process 86314]

Thread 1 received signal SIGSEGV, Segmentation fault.
0x0000000802463650 in clang::CodeGen::LValue::MakeAddr(clang::CodeGen::Address, clang::QualType, clang::ASTContext&, clang::CodeGen::LValueBaseInfo, clang::CodeGen::TBAAAccessInfo) () from /usr/local/lib/beignet//libgbe.so
(gdb) backtrace
#0  0x0000000802463650 in clang::CodeGen::LValue::MakeAddr(clang::CodeGen::Address, clang::QualType, clang::ASTContext&, clang::CodeGen::LValueBaseInfo, clang::CodeGen::TBAAAccessInfo) () from /usr/local/lib/beignet//libgbe.so
#1  0x0000000802482287 in clang::CodeGen::CodeGenFunction::EmitDeclRefLValue(clang::DeclRefExpr const*) ()
   from /usr/local/lib/beignet//libgbe.so
#2  0x0000000802477b79 in clang::CodeGen::CodeGenFunction::EmitLValue(clang::Expr const*) ()
   from /usr/local/lib/beignet//libgbe.so
#3  0x000000080247fc13 in clang::CodeGen::CodeGenFunction::EmitCheckedLValue(clang::Expr const*, clang::CodeGen::CodeGenFunction::TypeCheckKind) () from /usr/local/lib/beignet//libgbe.so
#4  0x0000000802519c36 in clang::StmtVisitorBase<clang::make_ptr, (anonymous namespace)::ScalarExprEmitter, llvm::Value*>::Visit(clang::Stmt*) () from /usr/local/lib/beignet//libgbe.so
#5  0x00000008025205fa in (anonymous namespace)::ScalarExprEmitter::VisitCastExpr(clang::CastExpr*) ()
   from /usr/local/lib/beignet//libgbe.so
#6  0x0000000802510258 in clang::CodeGen::CodeGenFunction::EmitScalarExpr(clang::Expr const*, bool) ()
   from /usr/local/lib/beignet//libgbe.so
#7  0x000000080247f015 in clang::CodeGen::CodeGenFunction::EmitPointerWithAlignment(clang::Expr const*, clang::CodeGen::LValueBaseInfo*, clang::CodeGen::TBAAAccessInfo*) () from /usr/local/lib/beignet//libgbe.so
#8  0x0000000802480352 in clang::CodeGen::CodeGenFunction::EmitArraySubscriptExpr(clang::ArraySubscriptExpr const*, bool) () from /usr/local/lib/beignet//libgbe.so
#9  0x0000000802477c51 in clang::CodeGen::CodeGenFunction::EmitLValue(clang::Expr const*) ()
   from /usr/local/lib/beignet//libgbe.so
#10 0x000000080247fc13 in clang::CodeGen::CodeGenFunction::EmitCheckedLValue(clang::Expr const*, clang::CodeGen::CodeGenFunction::TypeCheckKind) () from /usr/local/lib/beignet//libgbe.so
#11 0x0000000802519c36 in clang::StmtVisitorBase<clang::make_ptr, (anonymous namespace)::ScalarExprEmitter, llvm::Value*>::Visit(clang::Stmt*) () from /usr/local/lib/beignet//libgbe.so
#12 0x00000008025205fa in (anonymous namespace)::ScalarExprEmitter::VisitCastExpr(clang::CastExpr*) ()
   from /usr/local/lib/beignet//libgbe.so
#13 0x0000000802515f57 in clang::StmtVisitorBase<clang::make_ptr, (anonymous namespace)::ScalarExprEmitter, llvm::Value*>::Visit(clang::Stmt*) () from /usr/local/lib/beignet//libgbe.so
#14 0x0000000802510258 in clang::CodeGen::CodeGenFunction::EmitScalarExpr(clang::Expr const*, bool) ()
   from /usr/local/lib/beignet//libgbe.so
#15 0x0000000802477572 in clang::CodeGen::CodeGenFunction::EmitAnyExpr(clang::Expr const*, clang::CodeGen::AggValueSlot, bool) () from /usr/local/lib/beignet//libgbe.so
#16 0x0000000802477524 in clang::CodeGen::CodeGenFunction::EmitIgnoredExpr(clang::Expr const*) ()
   from /usr/local/lib/beignet//libgbe.so
#17 0x00000008024c07b1 in clang::CodeGen::CodeGenFunction::EmitStmt(clang::Stmt const*, llvm::ArrayRef<clang::Attr const*>) () from /usr/local/lib/beignet//libgbe.so
#18 0x00000008024c933f in clang::CodeGen::CodeGenFunction::EmitCompoundStmtWithoutScope(clang::CompoundStmt const&, bool, clang::CodeGen::AggValueSlot) () from /usr/local/lib/beignet//libgbe.so
#19 0x00000008026085d5 in clang::CodeGen::CodeGenFunction::GenerateCode(clang::GlobalDecl, llvm::Function*, clang::CodeGen::CGFunctionInfo const&) () from /usr/local/lib/beignet//libgbe.so
#20 0x000000080261db9c in clang::CodeGen::CodeGenModule::EmitGlobalFunctionDefinition(clang::GlobalDecl, llvm::GlobalValue*) () from /usr/local/lib/beignet//libgbe.so
#21 0x0000000802617c64 in clang::CodeGen::CodeGenModule::EmitGlobalDefinition(clang::GlobalDecl, llvm::GlobalValue*) () from /usr/local/lib/beignet//libgbe.so
#22 0x0000000802620852 in clang::CodeGen::CodeGenModule::EmitTopLevelDecl(clang::Decl*) ()
   from /usr/local/lib/beignet//libgbe.so
#23 0x00000008026f1d3f in (anonymous namespace)::CodeGeneratorImpl::HandleTopLevelDecl(clang::DeclGroupRef) ()
   from /usr/local/lib/beignet//libgbe.so
#24 0x000000080243956a in clang::BackendConsumer::HandleTopLevelDecl(clang::DeclGroupRef) ()
   from /usr/local/lib/beignet//libgbe.so
#25 0x000000080303ac22 in clang::ParseAST(clang::Sema&, bool, bool) () from /usr/local/lib/beignet//libgbe.so
#26 0x000000080212f969 in clang::FrontendAction::Execute() () from /usr/local/lib/beignet//libgbe.so
#27 0x00000008020e6ec1 in clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) ()
   from /usr/local/lib/beignet//libgbe.so
#28 0x0000000801c84381 in gbe::buildModuleFromSource (
    source=0x8013392a0 "__kernel void self_test(__global int *buf) {  __local int tmp[3];  tmp[get_local_id(0)] = buf[get_local_id(0)];  barrier(CLK_LOCAL_MEM_FENCE);  buf[get_global_id(0)] = tmp[2 - get_local_id(0)] + buf[g"..., out_module=0x7fffffffda70, llvm_ctx=0x800257440, dumpLLVMFileName=..., dumpSPIRBinaryName=..., options=...,
    stringSize=1048576, err=0x8078435c0 "", errSize=0x801339288, oclVersion=120)
    at backend/src/backend/program.cpp:735
#29 0x0000000801c7035d in gbe::programNewFromSource (deviceID=6418,
    source=0x8013392a0 "__kernel void self_test(__global int *buf) {  __local int tmp[3];  tmp[get_local_id(0)] = buf[get_local_id(0)];  barrier(CLK_LOCAL_MEM_FENCE);  buf[get_global_id(0)] = tmp[2 - get_local_id(0)] + buf[g"..., stringSize=1048576, options=0x8008addaa "", err=0x8078435c0 "", errSize=0x801339288)
    at backend/src/backend/program.cpp:1058
#30 0x00000008008d2b77 in cl_program_build (p=0x8013391c0, options=0x8008addaa "") at src/cl_program.c:597
#31 0x00000008008c2cdd in clBuildProgram (program=0x8013391c0, num_devices=1, device_list=0x7fffffffe040,
    options=0x8008addaa "", pfn_notify=0x0, user_data=0x0) at src/cl_api.c:835
#32 0x00000008008e85d4 in cl_self_test (device=0x800945218 <intel_skl_gt2_device>,
    atomic_in_l3_flag=SELF_TEST_PASS) at src/cl_device_id.c:900
#33 0x00000008008e885e in cl_get_device_ids (platform=0x800941238 <intel_platform_data>, device_type=4294967295,
    num_entries=0, devices=0x0, num_devices=0x8033dbeb0) at src/cl_device_id.c:955
#34 0x00000008008c6093 in clGetDeviceIDs (platform=0x800941238 <intel_platform_data>, device_type=4294967295,
    num_entries=0, devices=0x0, num_devices=0x8033dbeb0) at src/cl_api_device_id.c:43
#35 0x000000080025fada in clGetDeviceIDs_hid () from /usr/local/lib/libOpenCL.so.1
#36 0x0000000000207071 in printPlatformInfo (p=0) at src/clinfo.c:459
#37 0x000000000020f0ed in main (argc=1, argv=0x7fffffffe278) at src/clinfo.c:2676

Sorry, I didn't build devel/llvm* with debug symbols this time.

Comment 8 Jan Beich 2018-05-13 06:03:33 UTC

dlopen'ing libpthread should be supported since base r276630.

Comment 9 Alexey Dokuchaev 2018-05-20 17:25:06 UTC

I've just tested Quake 2 again after jemalloc update to version 5.1.0 as per base r333477, it is still broken. :-(

Comment 10 Alexey Dokuchaev 2018-06-04 17:04:30 UTC

(In reply to Jason Evans from comment #6)
> I don't have any specific ideas as to why this is happening.  Bootstrapping
> jemalloc on FreeBSD is really tricky, because pthreads mutexes use malloc,
> and jemalloc has to jump through hoops to initialize its mutexes in multiple
> phases [...]
OK, but can you think of anything that had *changed* between 5.0.0 vs 4.5.0 in that area that could've caused this regression?  Looks like some software was working fine for years (that is the case at least for Quake2) until 5.0.0 hit the tree, so I want to know if 5.0.0 had revealed a bug in those programs, or rather introduced a bug into itself?

Comment 11 Alexey Dokuchaev 2019-01-30 11:59:51 UTC

It very much looks like base r343566 might finally fix this.  On the latest -CURRENT, Quake II starts normally again.

Comment 12 Jan Beich 2019-01-30 15:30:45 UTC

I confirm. beignet issue is masked (see below) after ports r473868, so I had to downgrade the port before testing base r343566.

$ ldd -a =clinfo
/usr/local/bin/clinfo:
        libOpenCL.so.1 => /usr/local/lib/libOpenCL.so.1 (0x800259000)
        libc.so.7 => /lib/libc.so.7 (0x800276000)
/usr/local/lib/libOpenCL.so.1:
        libdl.so.1 => /usr/lib/libdl.so.1 (0x800697000)
        libthr.so.3 => /lib/libthr.so.3 (0x80069b000)  <--- LOOK HERE
        libc.so.7 => /lib/libc.so.7 (0x800276000)
/usr/lib/libdl.so.1:
        libc.so.7 => /lib/libc.so.7 (0x800276000)
/lib/libthr.so.3:
        libc.so.7 => /lib/libc.so.7 (0x800276000)

Comment 13 Rene Ladan 2020-04-02 18:20:48 UTC

lang/beignet expired today and has been removed.