Bug 257638 - lld fails to parse big endian compressed debug sections
Summary: lld fails to parse big endian compressed debug sections
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-08-05 15:36 UTC by Ed Maste
Modified: 2021-08-17 00:00 UTC (History)
0 users

See Also:


Attachments
lld reproducer tarball (18.00 KB, application/x-tar)
2021-08-06 00:41 UTC, Ed Maste
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ed Maste freebsd_committer 2021-08-05 15:36:03 UTC
I introduced compressed debug in:

commit c910570e757305ba4a974b628bcd8fd4278908a7
Author: Ed Maste <emaste@FreeBSD.org>
Date:   Mon Mar 1 12:25:22 2021 -0500

    Use compressed debug in standalone userland debug files by default
    
    The compiler supports CFLAGS=-gz=zlib to compress .debug sections in
    object files, libraries, and binaries.  Enable it to reduce disk usage
    for standalone debug files (and /usr/obj).
    
    Reviewed by:    dim, kevans
    Sponsored by:   The FreeBSD Foundation
    Differential Revision:  https://reviews.freebsd.org/D29002

but the build failed on powerpc and mips and so was reverted:

commit 89ed2ecb14ceabc27883282cf96559a9e7d52717
Author: Ed Maste <emaste@FreeBSD.org>
Date:   Thu Aug 5 08:34:03 2021 -0400

    Disable compressed debug by default
    
    In c910570e7573 I enabled compressed debug sections, but it broke mips
    and powerpc.  Disable it for now.
    
    Reported by:    jenkins, manu
    Sponsored by:   The FreeBSD Foundation

Failure examples:

FreeBSD-main-powerpc-build - Build #20581

 --- Scrt1.o ---
ld  -o Scrt1.o -r Scrt1_c.o crtbrand.o feature_note.o ignore_init_note.o
ld: error: Scrt1_c.o:(.debug_loc): unsupported compression type
ld: error: Scrt1_c.o:(.debug_abbrev): unsupported compression type
ld: error: Scrt1_c.o:(.debug_info): unsupported compression type
ld: error: Scrt1_c.o:(.debug_str): unsupported compression type
ld: error: Scrt1_c.o:(.debug_line): unsupported compression type

These ones also failed with *crt objects:
FreeBSD-main-powerpc64-build
FreeBSD-main-mips64-build
FreeBSD-main-mips-build
FreeBSD-main-powerpcspe-build

This one was in stand:

FreeBSD-main-powerpc64le-build

 --- all_subdir_stand ---
ld: error: /usr/obj/usr/src/powerpc.powerpc64le/stand/ficl/libficl.a(ficl.o):(.debug_loc): unsupported compression type
ld: error: /usr/obj/usr/src/powerpc.powerpc64le/stand/ficl/libficl.a(ficl.o):(.debug_abbrev): unsupported compression type
ld: error: /usr/obj/usr/src/powerpc.powerpc64le/stand/ficl/libficl.a(ficl.o):(.debug_info): unsupported compression type
ld: error: /usr/obj/usr/src/powerpc.powerpc64le/stand/ficl/libficl.a(ficl.o):(.debug_str): unsupported compression type
ld: error: /usr/obj/usr/src/powerpc.powerpc64le/stand/ficl/libficl.a(ficl.o):(.debug_line): unsupported compression type
ld: error: /usr/obj/usr/src/powerpc.powerpc64le/stand/ficl/libficl.a(dict.o):(.debug_loc): unsupported compression type
ld: error: /usr/obj/usr/src/powerpc.powerpc64le/stand/ficl/libficl.a(dict.o):(.debug_abbrev): unsupported compression type
ld: error: /usr/obj/usr/src/powerpc.powerpc64le/stand/ficl/libficl.a(dict.o):(.debug_info): unsupported compression type
ld: error: /usr/obj/usr/src/powerpc.powerpc64le/stand/ficl/libficl.a(dict.o):(.debug_ranges): unsupported compression type
ld: error: /usr/obj/usr/src/powerpc.powerpc64le/stand/ficl/libficl.a(dict.o):(.debug_str): unsupported compression type
Comment 1 Ed Maste freebsd_committer 2021-08-06 00:41:50 UTC
Created attachment 226979 [details]
lld reproducer tarball

lld reproducer attached, obtained via
$ LLD_REPRODUCE=freebsd-pr-257638.tar make

response.txt contains the command line to reproduce:
$ ld.lld $(cat response.txt)
ld.lld: error: usr/obj/usr/home/emaste/src/freebsd-git/main/mips.mips64/lib/csu/mips/Scrt1_c.o:(.debug_loc): unsupported compression type
ld.lld: error: usr/obj/usr/home/emaste/src/freebsd-git/main/mips.mips64/lib/csu/mips/Scrt1_c.o:(.debug_abbrev): unsupported compression type
ld.lld: error: usr/obj/usr/home/emaste/src/freebsd-git/main/mips.mips64/lib/csu/mips/Scrt1_c.o:(.debug_info): unsupported compression type
ld.lld: error: usr/obj/usr/home/emaste/src/freebsd-git/main/mips.mips64/lib/csu/mips/Scrt1_c.o:(.debug_str): unsupported compression type
ld.lld: error: usr/obj/usr/home/emaste/src/freebsd-git/main/mips.mips64/lib/csu/mips/Scrt1_c.o:(.debug_line): unsupported compression type

It appears the compressed header is not being parsed correctly with respect to endianness
Comment 2 Ed Maste freebsd_committer 2021-08-06 01:21:20 UTC
Upstream bug report at https://bugs.llvm.org/show_bug.cgi?id=51369
Comment 3 Ed Maste freebsd_committer 2021-08-06 14:07:39 UTC
Upstream patch in review https://reviews.llvm.org/D107635
Comment 4 Ed Maste freebsd_committer 2021-08-07 13:45:04 UTC
dim merged the fix:

commit d69d07569ee2404705ff4c4abb45dc0b884eeb35
Author: Dimitry Andric <dim@FreeBSD.org>
Date:   Sat Aug 7 13:14:11 2021 +0200

    Apply upstream lld fix for compressed input sections on BE targets
    
    Merge commit c6ebc651b6fa from llvm git (by Simon Atanasyan):
    
      [LLD] Support compressed input sections on big-endian targets
    
      This patch enables compressed input sections on big-endian targets by
      checking the target endianness and selecting an appropriate `Chdr`
      structure.
    
      Fixes PR51369
    
      Differential Revision: https://reviews.llvm.org/D107635
    
    Reported by:    emaste
    MFC after:      3 days
Comment 5 Ed Maste freebsd_committer 2021-08-07 15:39:31 UTC
Review to reenable for LE targets only: https://reviews.freebsd.org/D31454

For some time we may have to deal with unfixed lld installed from packages and as jrtc27 points out ports may link against base system .a archives, so we may want to just avoid compressed debug on BE for now.
Comment 6 commit-hook freebsd_committer 2021-08-08 17:36:08 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=47363e99d3d312c510902b68d5cf3094ddc7bb76

commit 47363e99d3d312c510902b68d5cf3094ddc7bb76
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2021-08-07 14:46:07 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2021-08-08 17:31:28 +0000

    Enable compressed debug on little-endian targets

    Compressed debug was enabled by default in commit c910570e7573, but
    broke the build on big-endian targets, and so was disabled in
    89ed2ecb14ce.

    Older versions of LLD fail with big-endian compressed debug sections.
    This was fixed in LLD upstream (commit c6ebc651b6fa) and merged to
    FreeBSD main (commit d69d07569ee2) by dim.

    External toolchains (e.g. the llvm12 package) will not yet have the fix.
    These may be used to link against base system .a archives, so compressed
    debug sections would cause trouble even though the base system is fixed.

    Enable compressed debug sections again, for little-endian targets only.
    As discussed on freebsd-hackers[1] I expect to undo this in the future
    (using compressed debug everywhere), once fixed versions of lld are
    widely available.

    Note that to be pedantically correct we should check both the compiler
    and the linker for compressed debug support, but given the external
    toolchain constraint the extra complexity does not seem worthwhile.

    [1] https://lists.freebsd.org/archives/freebsd-hackers/2021-August/000188.html

    PR:             257638
    Reported by:    jrtc27 [impact of .a archives]
    Discussed with: imp
    Relnotes:       Yes
    Sponsored by:   The FreeBSD Foundation
    Differential Revision:  https://reviews.freebsd.org/D31454

    Revert "Disable compressed debug by default"

    This reverts commit 89ed2ecb14ceabc27883282cf96559a9e7d52717.

 share/mk/bsd.compiler.mk | 9 ++++++++-
 share/mk/bsd.sys.mk      | 2 +-
 2 files changed, 9 insertions(+), 2 deletions(-)