This is a repost of https://github.com/llvm/llvm-project/issues/46697. I'm creating a PR here, because it turns out that LTO itself works fine when using lld from ports, but is broken with the base lld (lld 15 is also affected). It looks like it has affected powerpc64 ever since the GCC -> LLVM switch (so LLVM 9), but I never tried linking with ports lld until now. I have a simple test program: int main() { return 0; } and do: root@talos-powerpc64:~ # clang -fuse-ld=/usr/local/bin/ld.lld15 -flto test.c root@talos-powerpc64:~ # ./a.out root@talos-powerpc64:~ # clang -flto test.c root@talos-powerpc64:~ # ./a.out Segmentation fault (core dumped) I also deassembled both binaries and did a diff: --- baselld.S 2023-02-09 23:53:53.171611000 +0100 +++ portslld.S 2023-02-09 23:53:58.827662000 +0100 @@ -1,5 +1,5 @@ -a.out2: file format elf64-powerpc +a.out: file format elf64-powerpc Disassembly of section .text: @@ -20,7 +20,7 @@ 10010558: fb df 00 60 std 30, 96(31) 1001055c: 7c be 2b 78 mr 30, 5 ; if (environ == NULL) -10010560: e8 a4 80 70 ld 5, -32656(4) +10010560: e8 a4 80 58 ld 5, -32680(4) 10010564: fb 9f 00 50 std 28, 80(31) 10010568: 28 25 00 00 cmpldi 5, 0 1001056c: 7c 7c 1b 78 mr 28, 3 @@ -33,7 +33,7 @@ ; if (environ == NULL) 10010588: 40 82 00 0c bf 2, 0x10010594 <_start+0x68> ; environ = env; -1001058c: 38 64 80 70 addi 3, 4, -32656 +1001058c: 38 64 80 58 addi 3, 4, -32680 10010590: fb c3 00 00 std 30, 0(3) ; if (argc > 0 && argv[0] != NULL) { 10010594: 2c 1c 00 01 cmpwi 28, 1 @@ -65,7 +65,7 @@ 100105e0: 41 82 00 10 bt 2, 0x100105f0 <_start+0xc4> ; __ps_strings = ps_strings; 100105e4: 3c 62 00 01 addis 3, 2, 1 -100105e8: 38 63 80 68 addi 3, 3, -32664 +100105e8: 38 63 80 50 addi 3, 3, -32688 100105ec: f9 03 00 00 std 8, 0(3) ; if (&_DYNAMIC != NULL) 100105f0: 3c 62 ff ff addis 3, 2, -1 @@ -95,8 +95,8 @@ 10010640: 3f 42 00 01 addis 26, 2, 1 ; for (; aux->a_type != AT_NULL; aux++) { 10010644: 7c 63 f2 14 add 3, 3, 30 -10010648: 38 9b 80 78 addi 4, 27, -32648 -1001064c: 38 ba 80 7c addi 5, 26, -32644 +10010648: 38 9b 80 60 addi 4, 27, -32672 +1001064c: 38 ba 80 64 addi 5, 26, -32668 10010650: 38 63 ff f8 addi 3, 3, -8 10010654: 48 00 00 10 b 0x10010664 <_start+0x138> 10010658: 7c a7 2b 78 mr 7, 5 @@ -144,9 +144,9 @@ ; target = ((ifunc_resolver_t)ptr)(cpu_features, cpu_features2, 100106e0: 7d 89 03 a6 mtctr 12 100106e4: 39 20 00 00 li 9, 0 -100106e8: 80 7b 80 78 lwz 3, -32648(27) +100106e8: 80 7b 80 60 lwz 3, -32672(27) 100106ec: 39 40 00 00 li 10, 0 -100106f0: 80 9a 80 7c lwz 4, -32644(26) +100106f0: 80 9a 80 64 lwz 4, -32668(26) 100106f4: 4e 80 04 21 bctrl 100106f8: e8 41 00 18 ld 2, 24(1) ; *where = target; @@ -169,10 +169,10 @@ 10010734: 4e 80 04 21 bctrl 10010738: e8 41 00 18 ld 2, 24(1) ; exit(main(argc, argv, env)); -1001073c: 3c 62 00 01 addis 3, 2, 1 +1001073c: 3c 62 ff fe addis 3, 2, -2 10010740: 7f a4 eb 78 mr 4, 29 10010744: 7f c5 f3 78 mr 5, 30 -10010748: 39 83 80 50 addi 12, 3, -32688 +10010748: 39 83 7e 30 addi 12, 3, 32304 1001074c: 7f 83 e3 78 mr 3, 28 10010750: 7d 89 03 a6 mtctr 12 10010754: 4e 80 04 21 bctrl @@ -452,6 +452,8 @@ 10010b04: eb e1 ff f8 ld 31, -8(1) 10010b08: 4e 80 00 20 blr ... + +0000000010010b18 <main>: 10010b18: 38 60 00 00 li 3, 0 10010b1c: 90 61 ff f4 stw 3, -12(1) 10010b20: 38 60 00 00 li 3, 0 For some reason, main function is missing when linked with the base lld. Since this issue is non-existent with lld from ports, it must be because of how we built LLVM. It only affects powerpc64. powerpc and powerpc64le are not affected. If it's necessary to have access to some powerpc64 system for debugging, I can arrange it.
Also, using -flto=thin doesn't make a difference.
I get the same problem using base ld.lld v11.0.1 (FreeBSD 13.0-RELEASE) $ clang --version FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe) Target: powerpc64-unknown-freebsd13.0 Thread model: posix InstalledDir: /usr/bin $ ld.lld --version LLD 11.0.1 (FreeBSD llvmorg-11.0.1-0-g43ff75f2c3fe-1300007) (compatible with GNU linkers)
Alfredo, did you have any idea about this? In our copy of lld there are no powerpc64 specific changes, but in the llvm sources themselves we have these two diff chunks: --- a/llvm/lib/CodeGen/TargetLoweringBase.cpp +++ b/llvm/lib/CodeGen/TargetLoweringBase.cpp @@ -1995,7 +1995,7 @@ void TargetLoweringBase::insertSSPDeclarations(Module // FreeBSD has "__stack_chk_guard" defined externally on libc.so if (TM.getRelocationModel() == Reloc::Static && !TM.getTargetTriple().isWindowsGNUEnvironment() && - !TM.getTargetTriple().isOSFreeBSD()) + !(TM.getTargetTriple().isPPC64() && TM.getTargetTriple().isOSFreeBSD())) GV->setDSOLocal(true); } } --- a/llvm/lib/Target/PowerPC/PPCISelLowering.cpp +++ b/llvm/lib/Target/PowerPC/PPCISelLowering.cpp @@ -16726,6 +16726,8 @@ bool PPCTargetLowering::isFMAFasterThanFMulAndFAdd(con bool PPCTargetLowering::isFMAFasterThanFMulAndFAdd(const Function &F, Type *Ty) const { + if (Subtarget.hasSPE()) + return false; switch (Ty->getScalarType()->getTypeID()) { case Type::FloatTyID: case Type::DoubleTyID: I don't think it is likely the isFMAFasterThanFMulAndFAdd() fix will have any influence on your test case, but the insertSSPDeclarations() one might have...
(In reply to Piotr Kubaj from comment #0) > If it's necessary to have access to some powerpc64 system for debugging, I can arrange it. I think it might be handy. It looks like the ref[14|13|12]-powerpc64 machines no longer exist, so I can't use them anymore. Of course clang hosted on amd64 can target powerpc64 just fine, however, to be able to link I need at least C startup objects (crt*.o) and a libc: "/usr/bin/ld" --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --enable-new-dtags -o empty /usr/lib/crt1.o /usr/lib/crti.o /usr/lib/crtbegin.o -L/usr/lib -plugin /usr/bin/../lib/LLVMgold.so -plugin-opt=mcpu=ppc64 /home/dim/tmp/empty-b574db.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/crtend.o /usr/lib/crtn.o ld: error: /home/dim/tmp/empty-b574db.o is incompatible with /usr/lib/crt1.o clang: error: linker command failed with exit code 1 (use -v to see invocation)
Login data sent via email.
On Piotr's machine I built clang and lld 15.0.7 from source, and if you compile an empty program with -flto with that, it *also* crashes. Only the ports version of llvm15 works correctly! It looks like the crucial patches in the devel/llvm ports are: * extra-patch-clang_lib_Driver_ToolChains_Clang.cpp * patch-lib_Target_PowerPC_PPCTargetMachine.cpp These appear to be adaptations of https://cgit.freebsd.org/src/commit/?id=e4399d169acc ("[PowerPC64] Starting from FreeBSD 13.0, default to ELFv2 ABI") by jhibbits. The changes from that commit got lost in a previous import, https://cgit.freebsd.org/src/commit/?id=5ffd83dbcc34 ("Merge llvm, clang, compiler-rt, libc++, libunwind, lld, lldb and openmp master 2e10b7a39b9, the last commit before the llvmorg-12-init tag, from which release/11.x was branched"). Apparently I had promised to reapply all our custom patches again from there, but I missed this PowerPC specific one... So I'll put back e4399d169acc in main, before MFC'ing the lot. And also, as promised in that commit, it shall be upstreamed. :)
Oh, I somehow thought this issue was present since the original LLVM switch, anyway thanks for rootcausing it so quickly! Since this also affects stable/13 and releng/13.2, please also cherry-pick the patches there.
There is one thing that I don't fully understand though: why does the port's extra-patch-clang_lib_Driver_ToolChains_Clang.cpp *remove* the check for FreeBSD >= 13? E.g. it is just: --- clang/lib/Driver/ToolChains/Clang.cpp.orig 2021-08-11 19:51:00.122735000 +0200 +++ clang/lib/Driver/ToolChains/Clang.cpp 2021-08-11 19:51:24.346107000 +0200 @@ -1921,8 +1921,7 @@ if (T.isOSBinFormatELF()) { switch (getToolChain().getArch()) { case llvm::Triple::ppc64: { - if ((T.isOSFreeBSD() && T.getOSMajorVersion() >= 13) || - T.isOSOpenBSD() || T.isMusl()) + if (T.isOSFreeBSD() || T.isOSOpenBSD() || T.isMusl()) ABIName = "elfv2"; else ABIName = "elfv1"; The other patch, patch-lib_Target_PowerPC_PPCTargetMachine.cpp, does exactly the reverse! I.e. it uses elfv1 for FreeBSD < 13, and elfv2 for >= 13. Is the Clang.cpp patch actually needed? I think you might get in trouble if you *compile* stuff for elfv2 and then use LTO at link time where it selects elfv1? Brooks, any idea where that one came from? Note that Justin's original change to clang/lib/Basic/Targets/PPC.h also checked the OS version and use elfv1 for < 13, elfv2 for >= 13: --- a/contrib/llvm-project/clang/lib/Basic/Targets/PPC.h +++ b/contrib/llvm-project/clang/lib/Basic/Targets/PPC.h @@ -375,12 +375,29 @@ class LLVM_LIBRARY_VISIBILITY PPC64TargetInfo : public PPCTargetInfo { IntMaxType = SignedLong; Int64Type = SignedLong; + if (Triple.getEnvironment() != llvm::Triple::UnknownEnvironment) { + switch (Triple.getEnvironment()){ + case llvm::Triple::ELFv1: + ABI = "elfv1"; + break; + default: + ABI = "elfv2"; + break; + } + } else { + if ((Triple.getOS() == llvm::Triple::FreeBSD) && + (Triple.getOSMajorVersion() < 13)) { + ABI = "elfv1"; + } else { + ABI = "elfv2"; + } + } + + if ((Triple.getArch() == llvm::Triple::ppc64le)) { resetDataLayout("e-m:e-i64:64-n32:64"); - ABI = "elfv2"; } else { resetDataLayout("E-m:e-i64:64-n32:64"); - ABI = Triple.getEnvironment() == llvm::Triple::ELFv2 ? "elfv2" : "elfv1"; } if (Triple.getOS() == llvm::Triple::AIX) but this block of code doesn't exist anymore as-is, it has been refactored.
(In reply to Dimitry Andric from comment #8) Regarding extra-patch-clang_lib_Driver_ToolChains_Clang.cpp, please see 3b5dff3ea583726dca0709140447c653dcc90706. Without that patch, Firefox didn't build. I'm not sure whether it's necessary now, but it surely was before.
I built a base llvm with: diff --git a/contrib/llvm-project/clang/lib/Basic/Targets/PPC.h b/contrib/llvm-project/clang/lib/Basic/Targets/PPC.h index 8148762f446b..66069418f93f 100644 --- a/contrib/llvm-project/clang/lib/Basic/Targets/PPC.h +++ b/contrib/llvm-project/clang/lib/Basic/Targets/PPC.h @@ -429,7 +429,7 @@ class LLVM_LIBRARY_VISIBILITY PPC64TargetInfo : public PPCTargetInfo { ABI = "elfv2"; } else { DataLayout = "E-m:e-i64:64-n32:64"; - ABI = "elfv1"; + ABI = "elfv2"; } if (Triple.isOSFreeBSD() || Triple.isOSOpenBSD() || Triple.isMusl()) { diff --git a/contrib/llvm-project/llvm/lib/Target/PowerPC/PPCTargetMachine.cpp b/contrib/llvm-project/llvm/lib/Target/PowerPC/PPCTargetMachine.cpp index fe396cbfc011..78bd0aeb99fc 100644 --- a/contrib/llvm-project/llvm/lib/Target/PowerPC/PPCTargetMachine.cpp +++ b/contrib/llvm-project/llvm/lib/Target/PowerPC/PPCTargetMachine.cpp @@ -236,9 +236,8 @@ static PPCTargetMachine::PPCABI computeTargetABI(const Triple &TT, switch (TT.getArch()) { case Triple::ppc64le: - return PPCTargetMachine::PPC_ABI_ELFv2; case Triple::ppc64: - return PPCTargetMachine::PPC_ABI_ELFv1; + return PPCTargetMachine::PPC_ABI_ELFv2; default: return PPCTargetMachine::PPC_ABI_UNKNOWN; } This indeed fixes LTO issues. It probably breaks compilation for older FreeBSD, but I don't think this is a big issue. Any user who wants to build for older FreeBSD can just use a chroot or jail with older FreeBSD. It should also be possible to force elfv1 by adding -mabi=elfv1 (I haven't checked that). Would it be possible to commit that diff?
(In reply to Piotr Kubaj from comment #10) I think the original diff for llvm's PPCTargetMachine.cpp is better, because it checks the OSMajorVersion in the target triple. So if you compile for powerpc64-unknown-freebsd12 it will automatically use ELFv1, and for powerpc64-unknown-freebsd13 and higher it will use ELFv2. I think the same logic should apply to the toolchain part in clang's PPC.h, but I'll defer to you and any other PowerPC experts. I can't really make sure that it works correctly, so if you tell me that it does, I will believe you :)
(In reply to Dimitry Andric from comment #11) The problem is that, if you compile for powerpc64-unknown-freebsd (no version), elfv1 is used. This is why I committed extra-patch-clang_lib_Driver_ToolChains_Clang.cpp to ports, it actually fixed Firefox build.
Maybe we could switch the logic: - if no version is given, default to elfv2, - if the version is given and < 13, use elfv1.
Would this patch be acceptable? diff --git a/contrib/llvm-project/clang/lib/Basic/Targets/PPC.h b/contrib/llvm-project/clang/lib/Basic/Targets/PPC.h index 8148762f446b..62a06bd493be 100644 --- a/contrib/llvm-project/clang/lib/Basic/Targets/PPC.h +++ b/contrib/llvm-project/clang/lib/Basic/Targets/PPC.h @@ -427,6 +427,9 @@ class LLVM_LIBRARY_VISIBILITY PPC64TargetInfo : public PPCTargetInfo { } else if ((Triple.getArch() == llvm::Triple::ppc64le)) { DataLayout = "e-m:e-i64:64-n32:64"; ABI = "elfv2"; + } else if (Triple.isOSFreeBSD() && (Triple.getOSMajorVersion() == 0 || Triple.getOSMajorVersion() >= 13)) { + DataLayout = "E-m:e-i64:64-n32:64"; + ABI = "elfv2"; } else { DataLayout = "E-m:e-i64:64-n32:64"; ABI = "elfv1"; diff --git a/contrib/llvm-project/llvm/lib/Target/PowerPC/PPCTargetMachine.cpp b/contrib/llvm-project/llvm/lib/Target/PowerPC/PPCTargetMachine.cpp index fe396cbfc011..a9bb96d47c7e 100644 --- a/contrib/llvm-project/llvm/lib/Target/PowerPC/PPCTargetMachine.cpp +++ b/contrib/llvm-project/llvm/lib/Target/PowerPC/PPCTargetMachine.cpp @@ -231,8 +231,11 @@ static PPCTargetMachine::PPCABI computeTargetABI(const Triple &TT, assert(Options.MCOptions.getABIName().empty() && "Unknown target-abi option!"); - if (TT.isMacOSX()) + if (TT.isMacOSX()) { return PPCTargetMachine::PPC_ABI_UNKNOWN; + } else if (TT.isOSFreeBSD() && TT.getArch() == Triple::ppc64 && (TT.getOSMajorVersion() == 0 || TT.getOSMajorVersion() >= 13)) { + return PPCTargetMachine::PPC_ABI_ELFv2; + } switch (TT.getArch()) { case Triple::ppc64le:
(In reply to Piotr Kubaj from comment #14) Yeah, I think that should cover all cases? It would also be nice to upstream this, as it's annoying to keep carrying a patch that, well, makes stuff work. :)
(In reply to Dimitry Andric from comment #15) Sure, I will do that. But I'd like to first get this committed to FreeBSD, since it would be nice if it could make it to releng/13.2 before the RELEASE. Then we could drop LTO hacks for powerpc64 from ports. Does your comment mean approval for src commit?
(In reply to Piotr Kubaj from comment #16) Yes, please go ahead and mark it for MFC!
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=a1ffc2fe9ce54a498c410dcab86495569dbaa7cc commit a1ffc2fe9ce54a498c410dcab86495569dbaa7cc Author: Piotr Kubaj <pkubaj@FreeBSD.org> AuthorDate: 2023-02-16 23:49:43 +0000 Commit: Piotr Kubaj <pkubaj@FreeBSD.org> CommitDate: 2023-02-16 23:53:30 +0000 llvm: make sure to use ELFv2 ABI on powerpc64 Currently LLVM is more or less set up to use ELFv2, but it still defaults to ELFv1 in some places. This causes lld to generate broken binaries when used with LTO. PR: 269455 Approved by: dim MFC after: 3 days contrib/llvm-project/clang/lib/Basic/Targets/PPC.h | 3 +++ contrib/llvm-project/llvm/lib/Target/PowerPC/PPCTargetMachine.cpp | 5 ++++- 2 files changed, 7 insertions(+), 1 deletion(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=8f39cbf4f30c9a8cd3f593dc2b373a289785d4b6 commit 8f39cbf4f30c9a8cd3f593dc2b373a289785d4b6 Author: Piotr Kubaj <pkubaj@FreeBSD.org> AuthorDate: 2023-02-16 23:49:43 +0000 Commit: Piotr Kubaj <pkubaj@FreeBSD.org> CommitDate: 2023-02-21 01:19:33 +0000 llvm: make sure to use ELFv2 ABI on powerpc64 Currently LLVM is more or less set up to use ELFv2, but it still defaults to ELFv1 in some places. This causes lld to generate broken binaries when used with LTO. PR: 269455 Approved by: dim MFC after: 3 days (cherry picked from commit a1ffc2fe9ce54a498c410dcab86495569dbaa7cc) contrib/llvm-project/clang/lib/Basic/Targets/PPC.h | 3 +++ contrib/llvm-project/llvm/lib/Target/PowerPC/PPCTargetMachine.cpp | 5 ++++- 2 files changed, 7 insertions(+), 1 deletion(-)
A commit in branch releng/13.2 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=50796dea719a4ca2e26376eaab67eafd94532fee commit 50796dea719a4ca2e26376eaab67eafd94532fee Author: Piotr Kubaj <pkubaj@FreeBSD.org> AuthorDate: 2023-02-16 23:49:43 +0000 Commit: Piotr Kubaj <pkubaj@FreeBSD.org> CommitDate: 2023-02-21 13:38:43 +0000 llvm: make sure to use ELFv2 ABI on powerpc64 Currently LLVM is more or less set up to use ELFv2, but it still defaults to ELFv1 in some places. This causes lld to generate broken binaries when used with LTO. PR: 269455 Approved by: dim, re (cpercival) MFC after: 3 days (cherry picked from commit a1ffc2fe9ce54a498c410dcab86495569dbaa7cc) (cherry picked from commit 8f39cbf4f30c9a8cd3f593dc2b373a289785d4b6) contrib/llvm-project/clang/lib/Basic/Targets/PPC.h | 3 +++ contrib/llvm-project/llvm/lib/Target/PowerPC/PPCTargetMachine.cpp | 5 ++++- 2 files changed, 7 insertions(+), 1 deletion(-)
Fixed, thanks for help!