I'm running unmodified head at r320900 with GENERIC-NODEBUG configuration. All 32 bit binaries don't work: $ cat hello.c #include <stdio.h> int main() { printf("Hello, world!\n"); return 0; } $ cc -o hello64 hello.c $ cc -m32 -o hello32 hello.c $ ./hello64 Hello, world! $ ./hello32 Segmentation fault (core dumped) Core was generated by `./hello32'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000000 in ?? () (gdb) bt #0 0x00000000 in ?? () #1 0x201cd67f in _elf_aux_info (aux=<optimized out>, buf=0x201e1a00 <__stack_chk_guard>, buflen=32) at /usr/src/lib/libc/gen/auxv.c:126 #2 0x20158297 in __guard_setup () at /usr/src/lib/libc/secure/stack_protector.c:62 #3 0x201cf562 in ?? () from /usr/lib32/libc.so.7 #4 0x201cf57d in _init () from /usr/lib32/libc.so.7 #5 0x00000000 in ?? ()
More info: 32 bit binaries work when /usr/lib32/libc.so.7 is linked with ld.bfd, but crash when /usr/lib32/libc.so.7 is linked with ld.lld. I'm pasting both here.
Created attachment 184361 [details] libc.so.7 linked with LLD
Created attachment 184362 [details] libc.so.7 linked with BFD
To confirm, the problem exists with stock head (i.e., linked with bfd) and you included the lld-linked libc.so.7 as well just to be comprehensive?
No, sorry. It's broken with ld.lld and works with ld.bfd.
The LLD case is stock head with WITH_LLD_IS_LD and WITH_LLD_BOOTSTRAP set.
LLVM PR23214 (https://bugs.llvm.org/show_bug.cgi?id=23214) depends on individual outstanding LLD issues affecting FreeBSD and has an occasionally-updated status update. The most recent for i386 is: 4) Usable as the linker for the FreeBSD/i386 kernel+modules. lld-linked kernel boots. Modules build but are untested. Linking the base system succeeds, but applications all segfault with assertion errors from jemalloc. No further investigation or testing has yet been done. That is, there are lld bugs that affect i386, and amd64 with -m32. It works for linking simple binaries - e.g.: ref11-i386% cc -fuse-ld=lld hello.c ref11-i386% ./a.out Hello, world but it looks like something's broken linking libc.
This is the hello.c I'm using: #include <stdio.h> int main() { printf("Hello, world!\n"); return 0; } When compiled with "cc -m32 -o hello hello.c" on amd64 system, the resulting program crashes when libc is linked with ld.lld. This issue also affects libthr. When a binary is linked against ld.bfd-produced libc and ld.lld-produced libthr, it still crashes, so for now at least 32 bit libc and libthr need to be linked with ld.bfd.
(In reply to Nikolai Lifanov from comment #8) Right. I mean that you can link a hello world using lld (via -fuse-ld=lld) on a stock head system -- that is, one which has a bfd-linked libc.so.
The failure I see with lld-linked i386 is: # sh <jemalloc>: /usr/home/emaste/src/freebsd-wip/contrib/jemalloc/include/jemalloc/internal/tsd.h:240: Failed assertion: "!malloc_slow && tsd_tcache_enabled_get(tsd) && tsd_reentrancy_level_get(tsd) == 0"
comparing the symbol tables between the two (nm -D <file> | cut -c10- | sort): --- lld.sym 2017-07-19 12:21:24.989171000 -0400 +++ bfd.sym 2017-07-19 12:21:19.240946000 -0400 @@ -1,3 +1,11 @@ +A FBSD_1.0 +A FBSD_1.1 +A FBSD_1.2 +A FBSD_1.3 +A FBSD_1.4 +A FBSD_1.5 +A FBSDprivate_1.0 +A _end B _PathLocale B _ThreadRuneLocale B __des_crypt_LOCAL and relocations (readelf -r <file> | sed -E 's/[0-9a-f]{4,}/###/g' | sort): volta% diff -u1 lld.reloc bfd.reloc --- lld.reloc 2017-07-19 12:24:27.888070000 -0400 +++ bfd.reloc 2017-07-19 12:24:34.228979000 -0400 @@ -83,2 +83,4 @@ ### ### R_386_32 ### __sys_writev +### ### R_386_32 ### _end +### ### R_386_32 ### _end ### ### R_386_32 ### endgrent @@ -855,2 +857,3 @@ ### ### R_386_JUMP_SLOT ### setbuf +### ### R_386_JUMP_SLOT ### setcontext ### ### R_386_JUMP_SLOT ### setegid @@ -3365,3 +3368,2 @@ ### ### R_386_RELATIVE ### -### ### R_386_TLS_DTPMOD32 ### ### ### R_386_TLS_DTPMOD32 ### (Nikolai, were the attached libc.so.7s built from the same source tree rev?)
Yes, these were built from the same revision: $ diff -u <(ident libc.so.7.bfd) <(ident libc.so.7.lld) --- /tmp//sh-np.v0yaqh 2017-07-19 14:50:00.866753000 -0400 +++ /tmp//sh-np.OfqXT7 2017-07-19 14:50:00.896432000 -0400 @@ -1,4 +1,4 @@ -libc.so.7.bfd: +libc.so.7.lld: $FreeBSD: head/lib/csu/i386/crti.S 217105 2011-01-07 16:07:51Z kib $ $FreeBSD: head/lib/libc/string/wmemmove.c 188080 2009-02-03 17:58:20Z danger $ $FreeBSD: head/lib/libc/string/wmemcmp.c 188080 2009-02-03 17:58:20Z danger $
In r322589 I added a hack/workaround to force the use of ld.bfd for linking i386 libc, until the root cause can be identified and fixed.
I saw that. Thank you!
I think you also need to apply a similar hack to libthr.
(In reply to Nikolai Lifanov from comment #15) Do you have an example of the failure you see from libthr?
Yes, i386-wine* ports don't work with LLD-linked libthr.
(In reply to Nikolai Lifanov from comment #17) Do you have an example of how it fails though? Does it segfault, or just not run correctly in some way that provides no hint at the underlying problem? Debugging the failure in libc is is proving tricky, so I'm hoping for a conveniently debuggable failure in something other than libc (and hoping that it has the same root cause).
I just retried it with latest head, libc linked with BFD and libthr linked with LLD 5.0.0 and I can't reproduce the problem with i386-wine* ports anymore. Previously, wine would just segfault.
Fixed by r326897