Created attachment 250181 [details] Console Output Code: void* kernel(void* par) { printf("kernel\n"); return NULL; } int main(int argc, const char* const* args){ pthread_t thread; pthread_create( &thread, NULL, kernel, NULL ); pthread_join(thread, NULL); printf("FreeBSD RELEASE 14.0 doesn't reach here.\n"); return 0; } Console Output: kernel The code above demonstrates that the call to pthread_join() with valid parameters terminates the program with a return code of 134. I have not tested this issue on bare metal so I don't know if my assigned severity level is valid. I'm using FreeBSD RELEASE 14.0 running on Proxmox with an 8 Core x86_64 Intel XEON chip. My libraries work fine on Windows, MacOS, Linux, and OpenBSD. pthread_join() is the only thing causing problems on FreeBSD thus far. Everything else seems to be functional.
I forgot to mention that I've tested this with both GCC (g++) and Clang (clang++).
This is not a valid complete code, at least it lacks required includes. After adding it, the fragment works fine for me. You need to provide complete reproduction, including full source and the exact build recipe. pthread_join() is too fundamental to be broken in this trivial way, it would cause almost all threading programs to fail. BTW, code 134 is SIGNALLED SIGABRT exit.
(In reply to Konstantin Belousov from comment #2) I apologize; this is the first time I've ever filed a bug report to a project. I'm aware of the return code message. I'm 100 percent sure that this has to do with the FreeBSD or it's compiler toolchain/libraries. main.cpp: #include "iostream" // iostream's static library has issues on freebsd. #include "cstdio" #include "pthread.h" #include "pthread_np.h" void* kernel(void* par) { printf("kernel\n"); return NULL; } int main(int argc, const char* const* args){ pthread_t thread; pthread_create( &thread, NULL, kernel, NULL ); pthread_join(thread, NULL); printf("FreeBSD RELEASE 14.0 doesn't reach here.\n"); return 0; } build.sh: #!/bin/sh g++ -std=c++20 -static-libgcc -static-libstdc++ -static -pthread -O3 "main.cpp" -o "main_g++" clang++ -std=c++20 -static-libgcc -static-libstdc++ -static -pthread -O3 "main.cpp" -o "main_clang++" Problem: I've narrowed the issue down to "iostream" and it's static libraries causing problems. The issue goes away if you remove the flags to statically link libraries or remove the "iostream" header from the list of includes. Also, please keep in mind that this is all still being done in a virtual machine so I don't know if the issue persists on bare metal.
So this issue only happens for the static binary built with gcc. The thing that fails is the gcc unwinder: #0 0x00000000004d4dca in thr_kill () #1 0x00000000004c226f in raise () #2 0x00000000004df6f9 in abort () #3 0x0000000000402bef in uw_init_context_1 ( context=context@entry=0x7fffdfffdd50, outer_cfa=outer_cfa@entry=0x7fffdfffdf80, outer_ra=0x4b2326 <thread_unwind+54>) at ../../../gcc-13.2.0/libgcc/unwind-dw2.c:1336 #4 0x00000000004ad986 in _Unwind_ForcedUnwind (exc=0x800818940, stop=0x4b24d0 <thread_unwind_stop>, stop_argument=0x0) at ../../../gcc-13.2.0/libgcc/unwind.inc:212 #5 0x00000000004b2326 in thread_unwind () #6 0x00000000004b228c in _pthread_exit_mask () #7 0x00000000004b21fb in pthread_exit () #8 0x00000000004b1e6d in thread_start () #9 0x0000000000000000 in ?? () It fails because gcc' _Unwind_IteratePhdrCallback() insists on finding PT_GNU_EH_FRAME which is missed for gcc-compiled binary: Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align LOAD 0x0000000000000000 0x0000000000400000 0x0000000000400000 0x00000000001523d8 0x00000000001523d8 R E 0x1000 LOAD 0x0000000000153000 0x0000000000553000 0x0000000000553000 0x0000000000050274 0x0000000000273c30 RW 0x1000 NOTE 0x0000000000000158 0x0000000000400158 0x0000000000400158 0x0000000000000048 0x0000000000000048 R 0x4 TLS 0x00000000001973f0 0x00000000005973f0 0x00000000005973f0 0x0000000000001850 0x0000000000002080 R 0x10 GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 RW 0x10 The EH_FRAME segment is present for clang++. I have no idea why.
I can reproduce this with GCC 10, 11, 12, and 13. I cannot reproduce this with Clang 16 or 17. $ clang++16 -std=c++20 -static-libgcc -static-libstdc++ -static -pthread -O3 "main.cpp" -o "main_clang++" && ./main_clang++ clang-16: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument] clang-16: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument] kernel FreeBSD RELEASE 14.0 doesn't reach here. $ clang++17 -std=c++20 -static-libgcc -static-libstdc++ -static -pthread -O3 "main.cpp" -o "main_clang++" && ./main_clang++ clang++: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument] clang++: warning: argument unused during compilation: '-static-libstdc++' [-Wunused-command-line-argument] kernel FreeBSD RELEASE 14.0 doesn't reach here. It's reproducible with any exception with GCC: static-except.cc: int fn(void) { throw (int)0; } int main(int argc, char *argv[]) { try { fn(); } catch (int i) { return 0; } return 1; } $ g++13 -static static-except.cc -o static-except && ./static-except Abort trap (core dumped) I'll change this to a lang/gcc bug
If I link using Clang as the compiler driver (using either the ld.lld default, or ld.bfd) it works. The linker command line in the bfd case is: "/usr/local/bin/ld.bfd" --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o static-except /usr/lib/crt1.o /usr/lib/crti.o /usr/lib/crtbegin.o -L/usr/lib static-except.o -lc++ -lm -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/crtend.o /usr/lib/crtn.o On a quick look it appears GCC fails to pass `--eh-frame-header`
(In reply to Ed Maste from comment #6) -lc++ is not using libstdc++ but is using clang's libc++ instead. The command is not comparible to the failing context's use of libstdc++ .
(In reply to Ed Maste from comment #6) -Wl,--eh-frame-hdr indeed help with the unwinder in thread_exit. There is one more bug meantime, (gdb) r Starting program: /usr/home/kostik/work/DEV/tests/pr278551 [New LWP 308137 of process 11609] kernel [LWP 308137 of process 11609 exited] FreeBSD RELEASE 14.0 doesn't reach here. Thread 1 received signal SIGSEGV, Segmentation fault. Invalid permissions for mapped object. 0x00000000005aec00 in (anonymous namespace)::moneypunct_cache_ct () (gdb) bt #0 0x00000000005aec00 in (anonymous namespace)::moneypunct_cache_ct () #1 0x00000000004032bf in __do_global_dtors_aux () #2 0x00000000005400a5 in _fini () #3 0x00000000004dfaff in __cxa_finalize () #4 0x00000000004dfede in exit () #5 0x00000000004bf93f in __libc_start1 () #6 0x0000000000403298 in _start ()
from g++13 -dumpspecs it looks like --eh-frame-header is not included for statically linked binaries *link: %{!static|static-pie:--eh-frame-hdr} %{m32:-m elf_i386_fbsd}%{!m32:-m elf_x86_64_fbsd} %{p:%nconsider using '-pg' instead of '-p' with gprof(1)} %{v:-V} %{assert*} %{R*} %{rpath*} %{defsym*} %{shared:-Bshareable %{h*} %{soname*}} %{!shared: %{!static: %{rdynamic:-export-dynamic} -dynamic-linker %(fbsd_dynamic_linker) } %{static:-Bstatic}} %{symbolic:-Bsymbolic} I've opened a GCC bug for this: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114839
(In reply to Mohammed Goder from comment #0) If gcc is to be involved, a question may be if -Wl,-rpath=/usr/local/lib/gcc* for the appropriate * substitution is needed.
I did not reproduce a problem with g++13, with -Wl,--eh-frame-hdr: $ g++13 -Wl,--eh-frame-hdr -std=c++20 -static-libgcc -static-libstdc++ -static -pthread -O3 "main.cpp" -o "main_g++" && ./main_g++ kernel FreeBSD RELEASE 14.0 doesn't reach here. $
(In reply to Ed Maste from comment #11) Perhaps you do use libc++ from base indeed. I tried with the manual gcc build from sources, and got the destructor faulting.
I'm using the gcc13 package, and adding -v to my g++13 command line shows the link invocation as: /usr/local/libexec/gcc13/gcc/x86_64-portbld-freebsd14.0/13.2.0/collect2 -plugin /usr/local/libexec/gcc13/gcc/x86_64-portbld-freebsd14.0/13.2.0/liblto_plugin.so -plugin-opt=/usr/local/libexec/gcc13/gcc/x86_64-portbld-freebsd14.0/13.2.0/lto-wrapper -plugin-opt=-fresolution=/tmp//cc3JPuk2.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -m elf_x86_64_fbsd -V -Bstatic -o main_g++ /usr/lib/crt1.o /usr/lib/crti.o /usr/lib/crtbeginT.o -L/usr/local/lib/gcc13/gcc/x86_64-portbld-freebsd14.0/13.2.0 -L/usr/local/lib/gcc13/gcc/x86_64-portbld-freebsd14.0/13.2.0/../../../../../x86_64-portbld-freebsd14.0/lib -L/usr/local/lib/gcc13/gcc/x86_64-portbld-freebsd14.0/13.2.0/../../.. --eh-frame-hdr /tmp//ccQ9SSiv.o -lstdc++ -lm -lgcc -lgcc_eh -lpthread -lc -lgcc -lgcc_eh /usr/local/lib/gcc13/gcc/x86_64-portbld-freebsd14.0/13.2.0/crtend.o /usr/lib/crtn.o So looks like it should be libstdc++. Perhaps there's a newly-introduced issue, or a patch that exists only in the ports tree?
Thank you guys for your hard work and dedication. I found the issue with clang++. The "-fsanitize=safe-stack" flag causes a segfault on FreeBSD. GCC's equivalent "-mshstk" works fine and Clang's safe-stack works on OpenBSD, Linux, Windows, and MacOS. Should I make another bug report on FreeBSD's bug tracker or should this go to the clang devs?
(In reply to Mohammed Goder from comment #14) Actually slight correction. Clang's safe-stack does not work on OpenBSD.
(In reply to Mohammed Goder from comment #14) > Should I make another bug report on FreeBSD's bug tracker or should this go to the clang > devs? If you can get a minimized reproducer and exact reproduction steps, please go ahead and submit a FreeBSD bug. Issues that affect 3rd party FreeBSD base system software could be clearly upstream issues, could be clearly FreeBSD issues, or (as is the case here) might not be obvious or might be somewhere in between, so it's most reasonable for FreeBSD folks to do an initial triage before sending upstream.
(In reply to Ed Maste from comment #16) I did some further testing and I'm pretty sure that it's a clang issue. I remember using the flag on Windows, MacOS, and Linux but it seems as though either my memory is bad or they removed support for the flag on Windows and MacOS; and left it in an unstable state on Linux. My memory might be conflating using GCC's shadow stack with clang's safe-stack. Either way; Thanks a bunch. I'll leave you guys to solve the GCC problem.
FYI: I got more source code path information from a gdb based run. The context used g++13 but the system has symbols as well. I keep a main-src worktree that holds a copy of the main branch materials. Thus the /usr/main-src/ in my paths. (gdb) run Starting program: /usr/home/root/c_tests/g++-static-link-test [New LWP 249477 of process 56713] kernel Thread 2 received signal SIGABRT, Aborted. Sent by thr_kill() from pid 56713 and user 0. [Switching to LWP 249477 of process 56713] thr_kill () at thr_kill.S:4 4 RSYSCALL(thr_kill) (gdb) bt #0 thr_kill () at thr_kill.S:4 #1 0x00000000004c283f in __raise (s=s@entry=6) at /usr/main-src/lib/libc/gen/raise.c:48 #2 0x00000000004df3d9 in abort () at /usr/main-src/lib/libc/stdlib/abort.c:61 #3 0x0000000000402be5 in uw_init_context_1 (context=context@entry=0x7fffdfffdd50, outer_cfa=outer_cfa@entry=0x7fffdfffdf80, outer_ra=0x4b2fc6 <thread_unwind+54>) at /wrkdirs/usr/ports/lang/gcc13/work/gcc-13.2.0/libgcc/unwind-dw2.c:1336 #4 0x00000000004ae786 in _Unwind_ForcedUnwind (exc=0x800818940, stop=0x4b3170 <thread_unwind_stop>, stop_argument=stop_argument@entry=0x0) at /wrkdirs/usr/ports/lang/gcc13/work/gcc-13.2.0/libgcc/unwind.inc:212 #5 0x00000000004b2fc6 in thread_unwind () at /usr/main-src/lib/libthr/thread/thr_exit.c:172 #6 0x00000000004b2f2c in _pthread_exit_mask (status=0x0, mask=mask@entry=0x0) at /usr/main-src/lib/libthr/thread/thr_exit.c:254 #7 0x00000000004b2e9b in _Tthr_exit (status=0x3ce85) at /usr/main-src/lib/libthr/thread/thr_exit.c:206 #8 0x00000000004b2b0d in thread_start (curthread=0x800818700) at /usr/main-src/lib/libthr/thread/thr_create.c:289 #9 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffdfffe000
(In reply to Mark Millard from comment #18) Whaat says that the two /wrkdirs/usr/ports/lang/gcc13/work/gcc-13.2.0/libgcc/* source files referenced are compatible with the otherwise-FreeBSD-specific source code files referenced?
So should this be reported upstream? If so, am I supposed to do that or will you guys be handling the rest of this?
(In reply to Mohammed Goder from comment #20) Niether With proper command line options the trivial test case variant below works just fine on FreeBSD. // File: lang-gcc-g++-exception-handling.cpp // On FreeBSD: // Works: g++13 -static -Wl,--eh-frame-hdr lang-gcc-g++-exception-handling.cpp -o lang-gcc-g++-exception-handling && ./lang-gcc-g++-exception-handling // FAILS: g++13 -static lang-gcc-g++-exception-handling.cpp -o lang-gcc-g++-exception-handling && ./lang-gcc-g++-exception-handling int main(int argc, char *argv[]) { try { throw (int)0; } catch (int i) { return 0; } return 1; } Such is also true with -static-libgcc -static-libstdc++ added to the command lines. I do not know if -Wl,--eh-frame-hdr should be implicit/automatic for FreeBSD. My test context was based on: # uname -apKU FreeBSD aarch64-main-pbase 15.0-CURRENT FreeBSD 15.0-CURRENT #5 main-n269589-9dcf39575efb-dirty: Sun Apr 21 01:42:00 PDT 2024 root@aarch64-main-pbase:/usr/obj/BUILDs/main-CA76-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA76 arm64 aarch64 1500018 1500018 # ~/fbsd-based-on-what-commit.sh -C /usr/ports 62a76b7dc95a (HEAD -> main, freebsd/main, freebsd/HEAD) graphics/mahotas: Update to 1.4.15 Author: Wen Heping <wen@FreeBSD.org> Commit: Wen Heping <wen@FreeBSD.org> CommitDate: 2024-04-22 00:04:50 +0000 branch: main merge-base: 62a76b7dc95aa8c2a74b06f92b0a8b752e3b1848 merge-base: CommitDate: 2024-04-22 00:04:50 +0000 n661234 (--first-parent --count for merge-base) I'll note that the same is true for: // File: lang-gcc-g++-exception-handling-with-threads.cpp // On FreeBSD: // Works: g++13 -static -Wl,--eh-frame-hdr -pthread lang-gcc-g++-exception-handling-with-threads.cpp -o lang-gcc-g++-exception-handling-with-threads && ./lang-gcc-g++-exception-handling-with-threads // FAILS: g++13 -static -pthread lang-gcc-g++-exception-handling-with-threads.cpp -o lang-gcc-g++-exception-handling-with-threads && ./lang-gcc-g++-exception-handling-with-threads #include "iostream" #include "cstdio" #include "pthread.h" #include "pthread_np.h" void* kernel(void* par) { printf("kernel\n"); return NULL; } int main(int argc, const char* const* args){ pthread_t thread; pthread_create( &thread, NULL, kernel, NULL ); pthread_join(thread, NULL); printf("FreeBSD doesn't reach here.\n"); return 0; }
(In reply to Mark Millard from comment #21) Probably not obvious for my prior note: The final -Wl,--eh-frame-hdr case does, in fact, also print the line: FreeBSD doesn't reach here.
(In reply to Mark Millard from comment #21) Another combination that works uses libc++ instead (pthread example again): g++13 -static -pthread -stdlib=libc++ lang-gcc-g++-exception-handling-with-threads.cpp -o lang-gcc-g++-exception-handling-with-threads && ./lang-gcc-g++-exception-handling-with-threads
*** Bug 273398 has been marked as a duplicate of this bug. ***
(In reply to Mohammed Goder from comment #20) > So should this be reported upstream? I reported this upstream as https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114839. You don't need to do anything further. Kostik may have found another issue (see comment #12) that we'll have to take care of.