Bug 218198 - devel/aarch64-binutils: update to 2.28 breaks buildworld with aarch64
Summary: devel/aarch64-binutils: update to 2.28 breaks buildworld with aarch64
Status: Closed Overcome By Events
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: arm64 Any
: --- Affects Only Me
Assignee: Baptiste Daroussin
URL:
Keywords:
Depends on:
Blocks: 201763
  Show dependency treegraph
 
Reported: 2017-03-28 22:08 UTC by Enji Cooper
Modified: 2019-05-27 21:55 UTC (History)
5 users (show)

See Also:
bugzilla: maintainer-feedback? (bapt)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Enji Cooper freebsd_committer 2017-03-28 22:08:26 UTC
- Originally reported here: https://lists.freebsd.org/pipermail/freebsd-ports/2017-March/107859.html .
- Also seen in the arm64 job, starting here: https://ci.freebsd.org/job/FreeBSD-head-aarch64-build/1433/console .

The OP reported that they needed to downgrade the port version in order for the arm64 cross-compilation to go through.
Comment 1 Anton Yuzhaninov 2017-04-01 15:02:54 UTC
FYI: this llvm ticket can be related to problem with building aarch64:
https://bugs.llvm.org//show_bug.cgi?id=21077
Comment 2 Mark Millard 2017-04-01 21:31:03 UTC
(In reply to Anton Yuzhaninov from comment #1)

As far as I can tell if llvm's bugzilla 21077 were a
problem still then binutils 2.27.? versions would not
work either.

But reverting to the latest 2.27 vintage materials for
binutils does work. (Keeping clang and such unchanged.)

Something is different between 2.27 and 2.28 as far as
I can tell from the overall behavior of substitution-
testing the alternatives.

This suggests that 2.28 binutils is what would change
to fix the problem. (But it is no proof of such as
the unique solution.)
Comment 3 Anton Yuzhaninov 2017-04-02 01:30:55 UTC
Build error happens on linking shared object file with TLS variable and debug info.

For releng/11.0 there is a message:
/usr/local/aarch64-freebsd/bin/ld: setrunelocale.So(.debug_info+0x3d): R_AARCH64_ABS64 used with TLS symbol _ThreadRuneLocale

readelf --relocs /path/to/setrunelocale.So
...
Relocation section with addend (.rela.debug_info):
r_offset     r_info       r_type              st_value         st_name + r_addend
...
00000000003d 002800000101 R_AARCH64_ABS64     0000000000000000 _ThreadRuneLocale + 0

Question: Is R_AARCH64_ABS64 right relocation code here?
Comment 4 Mark Millard 2017-04-02 02:16:28 UTC
(In reply to Anton Yuzhaninov from comment #3)

The original report on the list that is referenced is for getting (on head):
(and, yes, R_AARCH64_ABS64 is part of what is reported)

Building /usr/obj/pine64_clang/arm64.aarch64/usr/src/lib/libc/libc.so.7.full
--- libc.so.7.full ---
building shared library libc.so.7
/usr/local/aarch64-freebsd/bin/ld: getutxent.pico(.debug_info+0x3b): R_AARCH64_ABS64 used with TLS symbol udb
/usr/local/aarch64-freebsd/bin/ld: getutxent.pico(.debug_info+0x58): R_AARCH64_ABS64 used with TLS symbol uf
/usr/local/aarch64-freebsd/bin/ld: utxdb.pico(.debug_info+0x5a): R_AARCH64_ABS64 used with TLS symbol futx_to_utx.ut
/usr/local/aarch64-freebsd/bin/ld: jemalloc_tsd.pico(.debug_info+0x3c): R_AARCH64_ABS64 used with TLS symbol __je_tsd_tls
/usr/local/aarch64-freebsd/bin/ld: jemalloc_tsd.pico(.debug_info+0x146e): R_AARCH64_ABS64 used with TLS symbol __je_tsd_initialized
/usr/local/aarch64-freebsd/bin/ld: cxa_thread_atexit_impl.pico(.debug_info+0x3b): R_AARCH64_ABS64 used with TLS symbol dtors
/usr/local/aarch64-freebsd/bin/ld: xlocale.pico(.debug_info+0x403): R_AARCH64_ABS64 used with TLS symbol __thread_locale
/usr/local/aarch64-freebsd/bin/ld: setrunelocale.pico(.debug_info+0x3c): R_AARCH64_ABS64 used with TLS symbol _ThreadRuneLocale
cc: error: linker command failed with exit code 1 (use -v to see invocation)


But this only happens with 2.28 binutils. With the prior 2.27 binutils it
does not happen. The problem is not just the inputs to binutils but what
it does with them, 2.27 does something different than 2.28.

Whatever the overall changes, at least that difference needs to be
explained for 2.27 vs. 2.28 and covered by what is discovered and
fixed. The changes might not be limited to binutils but 2.27 does
not require anything else to be changed in order to work for
linking libc.so.7.full .

Even the official FreeBSD build server activities were working
with 2.27 until 2.28 was put in place.
Comment 5 Mark Millard 2017-04-03 04:17:26 UTC
(In reply to Mark Millard from comment #4)

https://lists.freebsd.org/pipermail/freebsd-ports/2017-April/107988.html
from Anton Yuzhaninov citrin+bsd at citrin.ru on Sat Apr 1 15:57:06 UTC 2017
reports:

workaround working for me is

WITHOUT_DEBUG_FILES=yes in /etc/src.conf

but there is a bug somewhere (in linker from binutils or in llvm).

[End  report]


It is very interesting if that means that the combination:

WITH_DEBUG=
WITHOUT_DEBUG_FILES=

works as it implies that moving the debug information to
separate files is involved in the problem but the debug
information itself is not a problem.
Comment 6 Anton Yuzhaninov 2017-04-03 17:06:29 UTC
(In reply to Mark Millard from comment #5)
There is a problem with linking object files containing debug info. Moving debug data to separate files happens later.
Comment 7 Anton Yuzhaninov 2017-04-03 17:14:48 UTC
I've build stable/11 aarch64 with binutils 2.27 - there is same errors in logs, but it is ignored:

/usr/local/aarch64-freebsd/bin/ld: getutxent.pico(.debug_info+0x3b): R_AARCH64_ABS64 used with TLS symbol udb
/usr/local/aarch64-freebsd/bin/ld: getutxent.pico(.debug_info+0x58): R_AARCH64_ABS64 used with TLS symbol uf
/usr/local/aarch64-freebsd/bin/ld: utxdb.pico(.debug_info+0x5a): R_AARCH64_ABS64 used with TLS symbol futx_to_utx.ut
/usr/local/aarch64-freebsd/bin/ld: jemalloc_tsd.pico(.debug_info+0x38): R_AARCH64_ABS64 used with TLS symbol __je_tsd_tls
/usr/local/aarch64-freebsd/bin/ld: jemalloc_tsd.pico(.debug_info+0x142f): R_AARCH64_ABS64 used with TLS symbol __je_tsd_initialized
/usr/local/aarch64-freebsd/bin/ld: cxa_thread_atexit_impl.pico(.debug_info+0x3b): R_AARCH64_ABS64 used with TLS symbol dtors
/usr/local/aarch64-freebsd/bin/ld: xlocale.pico(.debug_info+0x403): R_AARCH64_ABS64 used with TLS symbol __thread_locale
/usr/local/aarch64-freebsd/bin/ld: setrunelocale.pico(.debug_info+0x3c): R_AARCH64_ABS64 used with TLS symbol _ThreadRuneLocale
/usr/local/aarch64-freebsd/bin/objcopy --only-keep-debug libc.so.7.full libc.so.7.debug
/usr/local/aarch64-freebsd/bin/objcopy --strip-debug --add-gnu-debuglink=libc.so.7.debug  libc.so.7.full libc.so.7
building special pic c library
...

The difference between 2.27 and 2.28 is in exit code. ld in booth 2.27 and 2.28 shows the same warning "R_AARCH64_ABS64 used with TLS symbol" but for 2.27 exit code is 0 and for 2.28 exit code is 1.
Comment 8 Mark Millard 2017-04-03 17:30:14 UTC
(In reply to Anton Yuzhaninov from comment #6)

So WITHOUT_DEBUG_FILES only has an indirect
contribution of possibly indirectly leading
to not having WITH_DEBUG.

That is more what I would have expected: WITH_DEBUG
is the real context for the issue.
Comment 9 Mark Millard 2017-04-03 17:39:54 UTC
(In reply to Anton Yuzhaninov from comment #7)

Interesting. Great find.

The implication is that for 2.27 the non-debug information
was linked into a usable state but debug information was
apparently (partially) broken: the 2.27 based builds were
operable.

Likely then 2.28 has a similar status but for the handling
of the exit code stopping things (unless 2.28 no longer
leaves the files behind).

That answers what my notes were about and leaves the
kind of thing your notes have been about.

[I spent the last 2 days recovering from a virtual machine
corruption so my investigation activity and time spent
thinking about this were greatly limited the last to days.
It shows in the quality of my notes.]
Comment 10 Anton Yuzhaninov 2017-04-03 17:41:13 UTC
There is a change in binutils affected ld exit code:
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=4519d071387f374932616b588ddb4ec8cabe2a52
Comment 11 Mark Millard 2017-05-17 23:50:54 UTC
(In reply to Mark Millard from comment #9)

A workaround for svnlite/svn folks that use
source builds is to do something like:

svnlite revert -r436731 /usr/ports/devel/*binutils

to revert to the source for 2.27 vintage binutils
and then rebuild the binutils that you use. Then
use these older binutils.

Because of slave-port relationships the main
host binutils needs to be reverted as part
of this, not just the target machine's binutils.

This will still produce the warnings but changes
the return code from ld to allow things to
continue with somewhat messed up debug
information.

2.28's return code is probably correct long term
and this is just a temporary hack until the input
to ld is correct so that 2.28 has nothing to
cause the non-zero return code.
Comment 12 Mark Millard 2017-05-18 00:02:35 UTC
(In reply to Mark Millard from comment #11)


Update, not revert:

svnlite update -r436731 /usr/ports/devel/*binutils



I wish I could edit out confusions that I create.
Comment 13 Li-Wen Hsu freebsd_committer 2019-01-27 10:49:52 UTC
devel/aarch64-binutils is 2.31.1 now and I think the issue has been fixed. Can we close this ticket?
Comment 14 Mark Millard 2019-01-27 10:59:44 UTC
(In reply to Li-Wen Hsu from comment #13)

With the default now being use of lld, I wonder if the aarch64-binutils
based ld is being tested much.
Comment 15 Li-Wen Hsu freebsd_committer 2019-05-27 20:06:58 UTC
With these packages:

aarch64-binutils-2.32_1,1
aarch64-gcc-6.4.0_5
aarch64-xtoolchain-gcc-0.4_1

We still have problem when build by:

# make -j32 -DNO_CLEAN CROSS_TOOLCHAIN=aarch64-gcc TARGET=arm64 TARGET_ARCH=aarch64 buildworld buildkernel

--- libc.so.7.full ---
/usr/local/bin/aarch64-unknown-freebsd12.0-ld: /usr/local/libexec/gcc/aarch64-unknown-freebsd12.0/6.4.0/liblto_plugin.so: error loading plugin: Service unavailable
collect2: error: ld returned 1 exit status
*** [libc.so.7.full] Error code 1

I think this is another issue?
Comment 16 Mark Millard 2019-05-27 21:55:21 UTC
(In reply to Li-Wen Hsu from comment #15)

The message:

error loading plugin: Service unavailable

happens when (for example):

aarch64-binutils-2.32_1,1

was built such that ld was a static executable.

See the later comments in:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237688

where someone else recently went through this until
they rebuilt ld without it being static.