Bug 237068 - /usr/local/bin/ld: BFD (GNU Binutils) 2.30 assertion fail elflink.c:2824
Summary: /usr/local/bin/ld: BFD (GNU Binutils) 2.30 assertion fail elflink.c:2824
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: Normal Affects Some People
Assignee: freebsd-toolchain mailing list
URL:
Keywords: needs-qa
Depends on:
Blocks: 235891
  Show dependency treegraph
 
Reported: 2019-04-07 05:03 UTC by Kubilay Kocak
Modified: 2019-08-25 20:39 UTC (History)
8 users (show)

See Also:


Attachments
devel/llvm60 head-armv6-default (beefy) log (107.18 KB, application/gzip)
2019-04-07 05:06 UTC, Kubilay Kocak
no flags Details
Remove binutils-do-not-provide-shared-section-symbols patch (2.65 KB, patch)
2019-05-21 16:17 UTC, Dimitry Andric
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Kubilay Kocak freebsd_committer freebsd_triage 2019-04-07 05:03:17 UTC
Multiple reports of software (llvm60, at least) failing to link on 11, 12, current on multiple architectures with the following error:

/usr/local/bin/ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935

Affected software/environments reported so far:

lang/julia: link errors on stable/11 after clang 7.0.1 (bundled llvm60, amd64, bug 235891)

"devel/llvm60 on powerpcspe" (bug 235891 comment 7)

devel/llvm60 on armv6 (head-armv6-default, beefy8, log attached)
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-07 05:06:23 UTC
Created attachment 203437 [details]
devel/llvm60 head-armv6-default (beefy) log
Comment 2 Dimitry Andric freebsd_committer 2019-04-07 09:22:07 UTC
Does anybody have a reproducible test case that doesn't involve setting up a virtual arm box, and going through a complete llvm build?  For example, a tarball containing the .o and .a files which BFD ld uses for linking in this assertion case.

Also, is it reproducible with an older version version of BFD ld? If not, any ideas where it regressed?
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-07 10:17:49 UTC
(In reply to Dimitry Andric from comment #2)

Without arm, but still requiring an llvm build is (so far) bug 235891, which is:

- lang/julia
- stable/11 amd64 (post clang 7 import after base r344213)
- The port has a bundled llvm60

Original reporter of bug 235891 (CC'd) may be able to provide .o's and .a's

The earliest pkg-fallout notification I could find was:

https://lists.freebsd.org/pipermail/freebsd-pkg-fallout/Week-of-Mon-20181210/1013748.html

This is right around the time of base r341825 ("Upgrade our copies of clang, llvm, lld, lldb, compiler-rt and libc++ tothe upstream release_70 branch")
Comment 4 Ed Maste freebsd_committer 2019-04-08 12:45:07 UTC
Is there an associated GNU binutils ld bug report?
Comment 5 Dimitry Andric freebsd_committer 2019-04-08 13:56:45 UTC
(In reply to Ed Maste from comment #4)
> Is there an associated GNU binutils ld bug report?

Not yet, I'm working on a small test case. The issue appears to be caused by a versioned weak symbol:

  llvm::hashing::detail::get_execution_seed()::seed@@JL_LLVM_6.0

I added some debug printfs around the failing assertion, with binutils master, and this shows:

DBG h  : llvm::hashing::detail::get_execution_seed()::seed@@JL_LLVM_6.0
DBG def: __bss_start@@JL_LLVM_6.0
/home/dim/ins/binutils-master-e392bad3e/bin/ld: BFD (GNU Binutils) 2.32.51.20190407 assertion fail /share/dim/src/binutils-gdb/bfd/elflink.c:2974
DBG h  : llvm::hashing::detail::get_execution_seed()::seed@@JL_LLVM_6.0
DBG def: __bss_start@@JL_LLVM_6.0
/home/dim/ins/binutils-master-e392bad3e/bin/ld: BFD (GNU Binutils) 2.32.51.20190407 assertion fail /share/dim/src/binutils-gdb/bfd/elflink.c:2974

The 'seed' variable is a static variable in an inline function in include/llvm/ADT/Hashing.h:

/// \brief A global, fixed seed-override variable.
///
/// This variable can be set using the \see llvm::set_fixed_execution_seed
/// function. See that function for details. Do not, under any circumstances,
/// set or read this variable.
extern size_t fixed_seed_override;

inline size_t get_execution_seed() {
  // FIXME: This needs to be a per-execution seed. This is just a placeholder
  // implementation. Switching to a per-execution seed is likely to flush out
  // instability bugs and so will happen as its own commit.
  //
  // However, if there is a fixed seed override set the first time this is
  // called, return that instead of the per-execution seed.
  const uint64_t seed_prime = 0xff51afd7ed558ccdULL;
  static size_t seed = fixed_seed_override ? fixed_seed_override
                                           : (size_t)seed_prime;
  return seed;
}

In libLLVM-6.0.so's .symtab, it looks like:

$ readelf -sW /wrkdirs/share/dim/ports/lang/julia/work/julia-1.0.3/deps/scratch/llvm-6.0.0/build_Release/lib/libLLVM-6.0.so | grep _ZZN4llvm7hashing6detail18get_execution_seedEvE4seed
3490: 00000000027c5380     8 OBJECT  WEAK   DEFAULT   28 _ZZN4llvm7hashing6detail18get_execution_seedEvE4seed@@JL_LLVM_6.0
31450: 00000000027c5380     8 OBJECT  WEAK   DEFAULT   28 _ZZN4llvm7hashing6detail18get_execution_seedEvE4seed

For some reason, there is both a symbol with versionJL_LLVM_6.0, and one without.

libLLVM-6.0.so gets built with the following very simple version script:

JL_LLVM_6.0 { global: *; };
Comment 6 Jan Beich freebsd_committer 2019-04-19 21:29:47 UTC
devel/llvm60 alone causes 6882 ports skipped. Brooks, can you apply ports r492319 in the meantime? See https://people.freebsd.org/~jbeich/logs/head-armv6/llvm60-6.0.1_6.log
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-20 06:24:58 UTC
(In reply to Jan Beich from comment #6)

And please reference this PR on that and any related commits
Comment 8 Mark Millard 2019-05-19 19:43:43 UTC
I see this sort of thing on powerpc64 with 2.32 as well.

The code reporting the problem is the last BFD_ASSERT
in:

static bfd_boolean
_bfd_elf_fix_symbol_flags (struct elf_link_hash_entry *h,
                          struct elf_info_failed *eif)
{
. . .
 if (h->is_weakalias)
   {
     struct elf_link_hash_entry *def = weakdef (h);

     /* If the real definition is defined by a regular object file,
        don't do anything special.  See the longer description in
        _bfd_elf_adjust_dynamic_symbol, below.  */
     if (def->def_regular)
       {
         h = def;
         while ((h = h->u.alias) != def)
           h->is_weakalias = 0;
       }
     else
       {
         while (h->root.type == bfd_link_hash_indirect)
           h = (struct elf_link_hash_entry *) h->root.u.i.link;
         BFD_ASSERT (h->root.type == bfd_link_hash_defined
                     || h->root.type == bfd_link_hash_defweak);
         BFD_ASSERT (def->def_dynamic);
         BFD_ASSERT (def->root.type == bfd_link_hash_defined);
         (*bed->elf_backend_copy_indirect_symbol) (eif->info, def, h);
       }
   }

 return TRUE;
}

For:

/usr/bin/powerpc64-unknown-freebsd13.0-ld "--eh-frame-hdr" "-Bshareable" "--enable-new-dtags" "-o" "lib/libLTO.so.6.0.1" "/usr/lib/crti.o" "/usr/lib/crtbeginS.o" "-L/usr/local/lib" "-L/usr/lib" "-z" "origin" "-O3" "--gc-sections" "--version-script" "/wrkdirs/usr/ports/devel/llvm60/work/.build/tools/lto/LTO.exports" "-soname" "libLTO.so.6" "tools/lto/CMakeFiles/LTO.dir/LTODisassembler.cpp.o" "tools/lto/CMakeFiles/LTO.dir/lto.cpp.o" "-rpath" "\$ORIGIN/../lib:/usr/local/lib" "lib/libLLVM-6.0.so" "-lc++" "-lm" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "/usr/lib/crtendS.o" "/usr/lib/crtn.o"

the backtrace at the first message report is:

Breakpoint 1, error_handler_internal (fmt=0x101dbc48 "BFD %s assertion fail %s:%d", ap=0x3fffffffffffcdc8 "") at bfd.c:1093
1093	{
(gdb) bt
#0  error_handler_internal (fmt=0x101dbc48 "BFD %s assertion fail %s:%d", ap=0x3fffffffffffcdc8 "") at bfd.c:1093
#1  0x00000000100204e8 in ld_bfd_error_handler (fmt=<optimized out>, ap=<optimized out>) at ./ldmain.c:187
#2  0x0000000010049b7c in _bfd_error_handler (fmt=<optimized out>) at bfd.c:1178
#3  0x000000001002047c in ld_bfd_assert_handler (fmt=<optimized out>, bfdver=<optimized out>, file=<optimized out>, line=<optimized out>) at ./ldmain.c:177
#4  0x0000000010049de4 in bfd_assert (file=<optimized out>, line=<optimized out>) at bfd.c:1429
#5  0x00000000100a520c in _bfd_elf_fix_symbol_flags (h=h@entry=0x811e77b70, eif=eif@entry=0x3fffffffffffcf70) at elflink.c:2935
#6  0x00000000100a54a0 in _bfd_elf_link_assign_sym_version (h=0x811e77b70, data=0x3fffffffffffd148) at elflink.c:2363
#7  0x0000000010053d28 in bfd_link_hash_traverse (htab=0x810e50000, func=@0x10223720: 0x100a5460 <_bfd_elf_link_assign_sym_version>, info=<optimized out>) at linker.c:669
#8  0x00000000100ab92c in bfd_elf_size_dynamic_sections (output_bfd=0x810e4b000, soname=0x3fffffffffffdc87 "libLTO.so.6", rpath=0x8105fee80 "$ORIGIN/../lib:/usr/local/lib", filter_shlib=0x0, 
    audit=0x0, depaudit=0x0, auxiliary_filters=0x0, info=0x102324d8 <link_info>, sinterpptr=0x3fffffffffffd280) at elflink.c:6316
#9  0x0000000010032b34 in gldelf64ppc_fbsd_before_allocation () at eelf64ppc_fbsd.c:2107
#10 ppc_before_allocation () at eelf64ppc_fbsd.c:404
#11 0x00000000100264e0 in ldemul_before_allocation () at ldemul.c:82
#12 0x000000001001d644 in lang_process () at ldlang.c:7593
#13 0x00000000100047b8 in main (argc=<optimized out>, argv=<optimized out>) at ./ldmain.c:440


For _bfd_elf_fix_symbol_flags the later gdb output shows that *h has:

_ZZN4llvm7hashing6detail18get_execution_seedEvE4seed@@LLVM_6.0
root.type = bfd_link_hash_defweak
is_weakalias = 1

*h->u.alias has:
__bss_start@@LLVM_6.0
root.type = bfd_link_hash_indirect
is_weakalias = 0

So that was def (via weakdef) and def->root.type==bfd_link_hash_indirect .

(Not that I know what to do next with such information.)

The supporting details:

static inline struct elf_link_hash_entry *
weakdef (struct elf_link_hash_entry *h)
{
  while (h->is_weakalias)
    h = h->u.alias;
  return h;
}

(gdb) down
#5  0x00000000100a520c in _bfd_elf_fix_symbol_flags (h=h@entry=0x811e77b70, eif=eif@entry=0x3fffffffffffcf70) at elflink.c:2935
2935		  BFD_ASSERT (def->root.type == bfd_link_hash_defined);

(gdb) print def
$1 = <optimized out>

(gdb) print *h
$5 = {root = {root = {next = 0x0, string = 0x811e77b30 "_ZZN4llvm7hashing6detail18get_execution_seedEvE4seed@@LLVM_6.0", hash = 1230121370}, type = bfd_link_hash_defweak, non_ir_ref_regular = 0, 
    non_ir_ref_dynamic = 0, linker_def = 0, ldscript_def = 0, rel_from_abs = 0, u = {undef = {next = 0x0, abfd = 0x811072340}, def = {next = 0x0, section = 0x811072340, value = 0}, i = {next = 0x0, 
        link = 0x811072340, warning = 0x0}, c = {next = 0x0, p = 0x811072340, size = 0}}}, indx = -1, dynindx = -1, got = {refcount = 0, offset = 0, glist = 0x0, plist = 0x0}, plt = {refcount = 0, 
    offset = 0, glist = 0x0, plist = 0x0}, size = 8, type = 1, other = 0, target_internal = 0, ref_regular = 0, def_regular = 0, ref_dynamic = 0, def_dynamic = 1, ref_regular_nonweak = 0, 
  dynamic_adjusted = 0, needs_copy = 0, needs_plt = 0, non_elf = 0, versioned = versioned, forced_local = 0, dynamic = 0, mark = 0, non_got_ref = 0, dynamic_def = 1, ref_dynamic_nonweak = 0, 
  pointer_equality_needed = 0, unique_global = 0, protected_def = 0, start_stop = 0, is_weakalias = 1, dynstr_index = 0, u = {alias = 0x8119bc690, elf_hash_value = 34655159952}, verinfo = {
    verdef = 0x811073448, vertree = 0x811073448}, u2 = {start_stop_section = 0x0, vtable = 0x0}}

(gdb) print *(eif->info)
$7 = {type = type_dll, symbolic = 0, nocopyreloc = 0, export_dynamic = 0, create_default_symver = 0, gc_sections = 1, gc_keep_exported = 0, notice_all = 0, lto_plugin_active = 0, 
  strip_discarded = 1, dynamic_data = 0, resolve_section_groups = 1, strip = strip_none, discard = discard_sec_merge, elf_stt_common = unchanged, common_skip_ar_symbols = bfd_link_common_skip_none, 
  unresolved_syms_in_objects = RM_IGNORE, unresolved_syms_in_shared_libs = RM_IGNORE, static_link = 0, keep_memory = 1, emitrelocations = 0, relro = 0, separate_code = 0, eh_frame_hdr_type = 1, 
  warn_shared_textrel = 0, error_textrel = 0, emit_hash = 1, emit_gnu_hash = 0, reduce_memory_overheads = 0, traditional_format = 0, combreloc = 1, default_imported_symver = 0, new_dtags = 1, 
  no_ld_generated_unwind_info = 0, task_link = 0, allow_multiple_definition = 0, prohibit_multiple_definition_absolute = 0, allow_undefined_version = 1, dynamic = 0, execstack = 0, noexecstack = 0, 
  optimize = 1, print_gc_sections = 0, warn_alternate_em = 0, user_phdrs = 0, load_phdrs = 1, check_relocs_after_open_input = 1, bndplt = 0, ibtplt = 0, ibt = 0, shstk = 0, nointerp = 0, 
  no_reloc_overflow_check = 0, call_nop_as_suffix = 0, inhibit_common_definition = 0, has_map_file = 0, call_nop_byte = 0 '\000', wrap_char = 46 '.', path_separator = 58 ':', 
  compress_debug = COMPRESS_DEBUG_NONE, stacksize = 0, disable_target_specific_optimizations = -1, callbacks = 0x10229318 <link_callbacks>, hash = 0x810e50000, keep_hash = 0x0, notice_hash = 0x0, 
  wrap_hash = 0x0, ignore_hash = 0x0, output_bfd = 0x810e4b000, out_implib_bfd = 0x0, input_bfds = 0x810e4b140, input_bfds_tail = 0x810e4cc70, create_object_symbols_section = 0x0, 
  gc_sym_list = 0x810e491a0, base_file = 0x0, init_function = 0x10143078 "_init", fini_function = 0x10143080 "_fini", relax_pass = 1, relax_trip = 0, extern_protected_data = -1, 
  dynamic_undefined_weak = -1, pei386_auto_import = -1, pei386_runtime_pseudo_reloc = 0, spare_dynamic_tags = 5, flags = 1, flags_1 = 128, relro_start = 0, relro_end = 0, dynamic_list = 0x0, 
  version_info = 0x810e22000}

(gdb) print *h->u.alias
$8 = {root = {root = {next = 0x8128f5e28, string = 0x8119bc678 "__bss_start@@LLVM_6.0", hash = 266808839}, type = bfd_link_hash_indirect, non_ir_ref_regular = 0, non_ir_ref_dynamic = 0, 
    linker_def = 0, ldscript_def = 1, rel_from_abs = 1, u = {undef = {next = 0x8119bc690, abfd = 0x8119bc748}, def = {next = 0x8119bc690, section = 0x8119bc748, value = 158833}, i = {
        next = 0x8119bc690, link = 0x8119bc748, warning = 0x26c71 <error: Cannot access memory at address 0x26c71>}, c = {next = 0x8119bc690, p = 0x8119bc748, size = 158833}}}, indx = -1, 
  dynindx = -1, got = {refcount = 0, offset = 0, glist = 0x0, plist = 0x0}, plt = {refcount = 0, offset = 0, glist = 0x0, plist = 0x0}, size = 0, type = 0, other = 0, target_internal = 0, 
  ref_regular = 0, def_regular = 0, ref_dynamic = 1, def_dynamic = 1, ref_regular_nonweak = 0, dynamic_adjusted = 0, needs_copy = 0, needs_plt = 0, non_elf = 0, versioned = versioned, 
  forced_local = 0, dynamic = 0, mark = 0, non_got_ref = 0, dynamic_def = 1, ref_dynamic_nonweak = 0, pointer_equality_needed = 0, unique_global = 0, protected_def = 0, start_stop = 0, 
  is_weakalias = 0, dynstr_index = 0, u = {alias = 0x811e77b70, elf_hash_value = 34660121456}, verinfo = {verdef = 0x811073448, vertree = 0x811073448}, u2 = {start_stop_section = 0x0, vtable = 0x0}}
Comment 9 Mark Millard 2019-05-19 19:58:54 UTC
(In reply to Mark Millard from comment #8)

Notes for the 2nd message and what follows. What follows
after the messaging appears interesting, noting the
comment's content.

(gdb) bt    
#0  error_handler_internal (fmt=0x101dbc48 "BFD %s assertion fail %s:%d", ap=0x3fffffffffffcdf8 "") at bfd.c:1093
#1  0x00000000100204e8 in ld_bfd_error_handler (fmt=<optimized out>, ap=<optimized out>) at ./ldmain.c:187
#2  0x0000000010049b7c in _bfd_error_handler (fmt=<optimized out>) at bfd.c:1178
#3  0x000000001002047c in ld_bfd_assert_handler (fmt=<optimized out>, bfdver=<optimized out>, file=<optimized out>, line=<optimized out>) at ./ldmain.c:177
#4  0x0000000010049de4 in bfd_assert (file=<optimized out>, line=<optimized out>) at bfd.c:1429
#5  0x00000000100a520c in _bfd_elf_fix_symbol_flags (h=h@entry=0x811e77b70, eif=eif@entry=0x3fffffffffffd0e0) at elflink.c:2935
#6  0x00000000100a5800 in _bfd_elf_adjust_dynamic_symbol (h=0x811e77b70, data=0x3fffffffffffd0e0) at elflink.c:2962
#7  0x0000000010053d28 in bfd_link_hash_traverse (htab=0x810e50000, func=@0x10223738: 0x100a57b0 <_bfd_elf_adjust_dynamic_symbol>, info=<optimized out>) at linker.c:669
#8  0x00000000100ac588 in bfd_elf_size_dynamic_sections (output_bfd=0x810e4b000, soname=<optimized out>, rpath=0x8105fee80 "$ORIGIN/../lib:/usr/local/lib", filter_shlib=0x0, audit=0x0, depaudit=0x0, 
    auxiliary_filters=<optimized out>, info=0x102324d8 <link_info>, sinterpptr=<optimized out>) at elflink.c:6847
#9  0x0000000010032b34 in gldelf64ppc_fbsd_before_allocation () at eelf64ppc_fbsd.c:2107
#10 ppc_before_allocation () at eelf64ppc_fbsd.c:404
#11 0x00000000100264e0 in ldemul_before_allocation () at ldemul.c:82
#12 0x000000001001d644 in lang_process () at ldlang.c:7593
#13 0x00000000100047b8 in main (argc=<optimized out>, argv=<optimized out>) at ./ldmain.c:440

At #5 *h shows:

_ZZN4llvm7hashing6detail18get_execution_seedEvE4seed@@LLVM_6.0
root.type = bfd_link_hash_defweak
is_weakalias = 1

*h->u.alias shows:

__bss_start@@LLVM_6.0
root.type = bfd_link_hash_indirect
is_weakalias = 0

So again: def->root.type== bfd_link_hash_indirect


The code continues after the two BFD_ASSERT messages.

If eventually reaches the xexit in main's :

477	  /* Even if we're producing relocatable output, some non-fatal errors should
478	     be reported in the exit status.  (What non-fatal errors, if any, do we
479	     want to ignore for relocatable output?)  */
480	  if (!config.make_executable && !force_make_executable)
481	    {
482	      if (verbose)
483		einfo (_("%P: link errors found, deleting executable `%s'\n"),
484		       output_filename);
485	
486	      /* The file will be removed by ld_cleanup.  */
487	      xexit (1);
488	    }
Comment 10 Mark Millard 2019-05-19 21:36:43 UTC
(In reply to Mark Millard from comment #9)

The only source code that I find with:

_ZZN4llvm7hashing6detail18get_execution_seedEvE4seed

(So: namespace llvm::hashing::detail:: ) is:

/wrkdirs/usr/ports/devel/llvm60/work/llvm-6.0.1.src/include/llvm/ADT/Hashing.h 's:

. . .
namespace llvm {
. . .

// All of the implementation details of actually computing the various hash
// code values are held within this namespace. These routines are included in
// the header file mainly to allow inlining and constant propagation.
namespace hashing {
namespace detail {
. . .

/// \brief A global, fixed seed-override variable.
///
/// This variable can be set using the \see llvm::set_fixed_execution_seed
/// function. See that function for details. Do not, under any circumstances,
/// set or read this variable.
extern size_t fixed_seed_override;

inline size_t get_execution_seed() {
  // FIXME: This needs to be a per-execution seed. This is just a placeholder
  // implementation. Switching to a per-execution seed is likely to flush out
  // instability bugs and so will happen as its own commit.
  //
  // However, if there is a fixed seed override set the first time this is
  // called, return that instead of the per-execution seed.
  const uint64_t seed_prime = 0xff51afd7ed558ccdULL;
  static size_t seed = fixed_seed_override ? fixed_seed_override
                                           : (size_t)seed_prime;
  return seed;
}
Comment 11 Mark Millard 2019-05-19 22:24:42 UTC
(In reply to Mark Millard from comment #10)

So far going down my own path seems to have confirmed
Dimitry Andric's comment #5 still applies, even for
powerpc64, but with the additional detail:

QUOTE
For _bfd_elf_fix_symbol_flags the later gdb output shows that *h has:

_ZZN4llvm7hashing6detail18get_execution_seedEvE4seed@@LLVM_6.0
root.type = bfd_link_hash_defweak
is_weakalias = 1

*h->u.alias has:
__bss_start@@LLVM_6.0
root.type = bfd_link_hash_indirect
is_weakalias = 0

So that was def (via weakdef) and def->root.type==bfd_link_hash_indirect .
END QUOTE.

So the @@LLVM_6.0 name is involved and so is an alias to:
__bss_start@@LLVM_6.0 for bfd_link_hash_indirect.
Comment 12 Mark Millard 2019-05-19 23:25:22 UTC
(In reply to Mark Millard from comment #11)

I've reduced the link command to:

# /usr/bin/powerpc64-unknown-freebsd13.0-ld "-Bshareable" "-o" "lib/libLTO.so.6.0.1" "lib/libLLVM-6.0.so"
/usr/bin/powerpc64-unknown-freebsd13.0-ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935
/usr/bin/powerpc64-unknown-freebsd13.0-ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935

The only reason for -Bsharable is that there are more messages without it,
relating to _start, __progname, and environ.

So it seems lib/libLLVM-6.0.so is sufficient context for the
BFD_ASSERT to fail.


# readelf -sW lib/libLLVM-6.0.so | grep 'get_execution_seed.*seed' | more
 12020: 0000000003c20f00     8 OBJECT  WEAK   DEFAULT   26 _ZZN4llvm7hashing6detail18get_execution_seedEvE4seed@@LLVM_6.0 (2)
 21501: 0000000003c20f08     8 OBJECT  WEAK   DEFAULT   26 _ZGVZN4llvm7hashing6detail18get_execution_seedEvE4seed@@LLVM_6.0 (2)
 68864: 0000000003c20f00     8 OBJECT  WEAK   DEFAULT   26 _ZZN4llvm7hashing6detail18get_execution_seedEvE4seed
 78345: 0000000003c20f08     8 OBJECT  WEAK   DEFAULT   26 _ZGVZN4llvm7hashing6detail18get_execution_seedEvE4seed

# readelf -sW lib/libLLVM-6.0.so | grep 'bss' | more
  6278: 0000000003c20f00     0 NOTYPE  GLOBAL DEFAULT   26 __bss_start@@LLVM_6.0 (2)
 63122: 0000000003c20f00     0 NOTYPE  GLOBAL DEFAULT   26 __bss_start

Also:

  [26] .bss              NOBITS           0000000003c20f00  03c10f00
       0000000000056010  0000000000000000  WA       0     0     8
Comment 13 Mark Millard 2019-05-20 01:09:22 UTC
(In reply to Mark Millard from comment #12)

The following 2 link commands, producing and
using an vastly-smaller lib/libLLVM-6.0.so
reproduce the messages:

"/usr/bin/powerpc64-unknown-freebsd13.0-ld" \
"-Bshareable" \
"-o" "lib/libLLVM-6.0.so" \
"/usr/lib/crtbeginS.o" \
"--version-script" "/wrkdirs/usr/ports/devel/llvm60/work/.build/./lib/tools/llvm-shlib/simple_version_script.map" \
"--whole-archive" \
"lib/libLLVMCore.a" \
"--no-whole-archive" \
"-lm"

/usr/bin/powerpc64-unknown-freebsd13.0-ld \
"-Bshareable" \
"-o" "lib/libLTO.so.6.0.1" \
"lib/libLLVM-6.0.so"

It looks like lib/libLLVMCore.a would be next for
making it smaller.
Comment 14 Mark Millard 2019-05-20 01:37:57 UTC
(In reply to Mark Millard from comment #13)

Exploring shrinking lib/libLLVMCore.a did not take
as long. Only Constants.cpp.o and Core.cpp.o are
required. The following sequence produces the messages:
(the tail is as in comment #13 but is repeated here)

rm lib/libLLVMCore.a

/usr/bin/ar qc lib/libLLVMCore.a \
lib/IR/CMakeFiles/LLVMCore.dir/Constants.cpp.o \
lib/IR/CMakeFiles/LLVMCore.dir/Core.cpp.o

/usr/bin/ranlib lib/libLLVMCore.a

"/usr/bin/powerpc64-unknown-freebsd13.0-ld" \
"-Bshareable" \
"-o" "lib/libLLVM-6.0.so" \
"/usr/lib/crtbeginS.o" \
"--version-script" "/wrkdirs/usr/ports/devel/llvm60/work/.build/./lib/tools/llvm-shlib/simple_version_script.map" \
"--whole-archive" \
"lib/libLLVMCore.a" \
"--no-whole-archive" \
"-lm"

/usr/bin/powerpc64-unknown-freebsd13.0-ld \
"-Bshareable" \
"-o" "lib/libLTO.so.6.0.1" \
"lib/libLLVM-6.0.so"
Comment 15 Mark Millard 2019-05-20 01:46:55 UTC
(In reply to Mark Millard from comment #14)

lib/IR/CMakeFiles/LLVMCore.dir/Core.cpp.o is not needed to
show the problem. So just:

rm lib/libLLVMCore.a

/usr/bin/ar qc lib/libLLVMCore.a \
lib/IR/CMakeFiles/LLVMCore.dir/Constants.cpp.o

"/usr/bin/powerpc64-unknown-freebsd13.0-ld" \
"-Bshareable" \
"-o" "lib/libLLVM-6.0.so" \
"/usr/lib/crtbeginS.o" \
"--version-script" "/wrkdirs/usr/ports/devel/llvm60/work/.build/./lib/tools/llvm-shlib/simple_version_script.map" \
"--whole-archive" \
"lib/libLLVMCore.a" \
"--no-whole-archive" \
"-lm"

/usr/bin/powerpc64-unknown-freebsd13.0-ld \
"-Bshareable" \
"-o" "lib/libLTO.so.6.0.1" \
"lib/libLLVM-6.0.so"
Comment 16 Mark Millard 2019-05-20 04:35:48 UTC
(In reply to Mark Millard from comment #15)

The following small source code file and the few steps
to build/link it produce the message:

# more small_link_failure.cpp
unsigned long fixed_seed_override= 0xdeadc0de;

inline unsigned long get_execution_seed()
{
  const  unsigned long seed_prime = 0xff51afd7ed558ccdULL;
  static unsigned long seed = fixed_seed_override ? fixed_seed_override
                                           : (unsigned long)seed_prime;
  return seed;
}

unsigned long f()
{
  return get_execution_seed();
}

# c++ -c small_link_failure.cpp

# rm -f small_link_failure.a
# /usr/bin/ar qc small_link_failure.a small_link_failure.o

# "/usr/bin/powerpc64-unknown-freebsd13.0-ld" \
"-Bshareable" \
"-o" "small_link_failure-1.0.so" \
"/usr/lib/crtbeginS.o" \
"--version-script" "/wrkdirs/usr/ports/devel/llvm60/work/.build/./lib/tools/llvm-shlib/simple_version_script.map" \
"--whole-archive" \
"small_link_failure.a" \
"--no-whole-archive" \
"-lm"

# /usr/bin/powerpc64-unknown-freebsd13.0-ld \
"-Bshareable" \
"-o" "small_link_failure.so.1.0.1" \
"small_link_failure-1.0.so"

The result is:

/usr/bin/powerpc64-unknown-freebsd13.0-ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935
/usr/bin/powerpc64-unknown-freebsd13.0-ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935
Comment 17 Mark Millard 2019-05-20 05:18:21 UTC
(In reply to Mark Millard from comment #16)

Using:

g++8 -c small_link_failure.cpp

instead of the system-clang-8 c++ also reproduces
getting the messages after the other steps.

Using the program:

/usr/local/bin/ld

(from devel/binutils) instead of:

/usr/bin/powerpc64-unknown-freebsd13.0-ld

(from base/binutils) also reproduces getting the
messages.
Comment 18 Dimitry Andric freebsd_committer 2019-05-20 20:31:16 UTC
(In reply to Mark Millard from comment #16)
> The following small source code file and the few steps
> to build/link it produce the message:
> 
> # more small_link_failure.cpp

Thanks Mark, this test case is very useful.  With a quite recent binutils master, it does *not* reproduce, but with the ports version, it does.

I'm bisecting binutils git to see where it was fixed.
Comment 19 Mark Millard 2019-05-20 20:38:12 UTC
(In reply to Mark Millard from comment #16)

The below is about alternate fixed_seed_override
definitions/declarations in the small example
(not llvm60) and the consequences, in particular
not getting the BFD_ASSERTs. They are not proposed
fixes: I'm just reporting what was important about
the fixed_seed_override definition that I used in
the small example.

Instead:

unsigned long fixed_seed_override= 0x0;
or:
extern unsigned long fixed_seed_override;

so they instead do not contribute to .data .

These change small_link_failure-1.0.so such that the
BFD_ASSERT messages are not produced. The "0xdeadc0de"
vs. other two end up as reported below.

The file offset changes from as seen in (note the matching
0x10cc8 vs. the symbol tables for the 0xdeadc0de based
example):
. . .
  [19] .data             PROGBITS         0000000000010ca8  00000ca8
       0000000000000020  0000000000000000  WA       0     0     8
  [20] .bss              NOBITS           0000000000010cc8  00000cc8
       0000000000000010  0000000000000000  WA       0     0     8
. . .
Symbol table (.dynsym) contains 16 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
. . .
     7: 0000000000010cc8     0 NOTYPE  GLOBAL DEFAULT   20 __bss_start@@LLVM_6.0 (2)
. . .
Symbol table (.symtab) contains 52 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
. . .
    43: 0000000000010cc8     0 NOTYPE  GLOBAL DEFAULT   20 __bss_start
. . .

to as seen in (note the 0x10cc0 vs. 0x10cb9's in the symbol tables):
. . .
  [19] .data             PROGBITS         0000000000010ca8  00000ca8
       0000000000000011  0000000000000000  WA       0     0     8
  [20] .bss              NOBITS           0000000000010cc0  00000cb9
       0000000000000018  0000000000000000  WA       0     0     8
. . .
Symbol table (.dynsym) contains 16 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
. . .
     7: 0000000000010cb9     0 NOTYPE  GLOBAL DEFAULT   20 __bss_start@@LLVM_6.0 (2)
. . .
Symbol table (.symtab) contains 52 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
. . .
    43: 0000000000010cb9     0 NOTYPE  GLOBAL DEFAULT   20 __bss_start
. . .

(Also note the odd .data size 0x11 compared to the earlier 0x20.)

So the definition of fixed_seed_override with a non-zero value
being involved is an important part of setting up the small
example of rejecting __bss_start@@LLVM_6.0 use internally.

(I had to discover this to make the small example C++ program.)
Comment 20 Dimitry Andric freebsd_committer 2019-05-20 21:10:25 UTC
Ok, bisection shows that your test case is fixed by this commit, which is unfortunately quite huge due to all the changed test coverage:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=fd161d860f1df7140153eab4726705cc3e2727b0

> Define various symbols conditionally in shared libraries
> 
> The values of symbols in shared libraries like _end, _edata, and
> __bss_start are generally not that useful outside of the shared
> library.  This patch defines them conditionally with PROVIDE, since a
> shared library might need the local value.  An example is glibc ld.so
> local access to "_begin", "_etext" and "_end".  (ld.so gains access to
> the local values by making the references using hidden visibility.
> That makes the definitions hidden too.)
> 
> We can't use PROVIDE_HIDDEN in the linker scripts because the shared
> library might need the value of the symbol in the executable.  An
> example is freebsd libc dynamic access to "_end".

See also https://sourceware.org/bugzilla/show_bug.cgi?id=23161

However, it seems that this fix is is already in binutils 2.32, which is the version of the current port.  So I don't really understand why the port also asserts.
Comment 21 Dimitry Andric freebsd_committer 2019-05-20 21:15:12 UTC
Ah, it seems that the upstream fix, in particular the one in ld/scripttempl/elf.sc here:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blobdiff;f=ld/scripttempl/elf.sc;h=444aef29423c6ba73753dbd5d222423954523a36;hp=5ccdd415f39ed4d7d17c9806c3aba3ef3a6cdda1;hb=fd161d860f1df7140153eab4726705cc3e2727b0;hpb=4f652d7b9a15cd3aae25de38f51baafe7e529646

was undone by Baptiste in this commit:

https://svnweb.freebsd.org/ports?view=revision&revision=490859

It even says "Reported by: antoine, dim", but I can't really remember what I had noticed at the time...  Must be getting old and forgetful :)

In any case, this is likely the cause for the assertion, but I'm not sure what other bad things might happen if the binutils-do-not-provide-shared-section-symbols.patch file is dropped.
Comment 22 Antoine Brodin freebsd_committer 2019-05-20 21:20:35 UTC
(In reply to Dimitry Andric from comment #21)

Without the patch,  lang/gcc* were failing with this error on at least i386:

/usr/local/bin/ld: /wrkdirs/usr/ports/lang/gcc48/work/.build/./gcc/libgcc_s.so: _end: invalid version 31393 (max 12)
Comment 24 Mark Millard 2019-05-20 22:04:08 UTC
(In reply to Antoine Brodin from comment #22)

That looks to me like possibly lack of a symbol version script
controlling what is exported from the relevant .so , creating
symbol conflicts.

It appears to me that trying to make that automatic interfers
with C++ specific functionality as things are. (get_execution_seed
and its handling of seed mixed with use of fixed_seed_override
is not valid for C, just for C++. I'm not aware how to cause the
link BFD_ASSERT messages via C, but I've not tried.)

But how many symbol version scripts would be required if the
automatic avoidance was not in place? (Messy?)
Comment 25 Mark Millard 2019-05-20 22:09:11 UTC
(In reply to Antoine Brodin from comment #23)

https://svnweb.freebsd.org/ports?view=revision&revision=490859

indicates that it got the patch from fedora.
Comment 26 Antoine Brodin freebsd_committer 2019-05-20 22:14:54 UTC
Fedora later removed the patch and used this one instead:

https://src.fedoraproject.org/rpms/binutils/c/0cdbf6312224a298ecff9e74e008f83d75c85025?branch=master
Comment 27 Dimitry Andric freebsd_committer 2019-05-21 16:17:58 UTC
Created attachment 204518 [details]
Remove binutils-do-not-provide-shared-section-symbols patch

(In reply to Antoine Brodin from comment #26)
> Fedora later removed the patch and used this one instead:
> 
> https://src.fedoraproject.org/rpms/binutils/c/
> 0cdbf6312224a298ecff9e74e008f83d75c85025?branch=master

Right, and binutils itself had it here:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=ecf99cc09236316db27fd8ffaedca864f81ea081

This code is also in our 2.32 version, so it won't need to be applied separately.

I tried removing the binutils-do-not-provide-shared-section-symbols.patch, rebuilt devel/binutils with it, and then I can successfully build devel/llvm60, lang/julia, lang/gcc48 and lang/gcc.  Patch attached.
Comment 28 Mark Millard 2019-05-22 13:40:29 UTC
(In reply to Dimitry Andric from comment #27)

I've not had your success, although things are different
with the change.

devel/llvm60 and devel/llvm80 still produce:

/usr/bin/powerpc64-unknown-freebsd13.0-ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935
/usr/bin/powerpc64-unknown-freebsd13.0-ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935


But the two commands from comment #13 based on
producing a reduced lib/libLLVM-6.0.so from just
lib/libLLVMCore.a do not produce the messages in
the new context. (Nor does my simple test case.)
For reference:

"/usr/bin/powerpc64-unknown-freebsd13.0-ld" \
"-Bshareable" \
"-o" "lib/libLLVM-6.0.so" \
"/usr/lib/crtbeginS.o" \
"--version-script" "/wrkdirs/usr/ports/devel/llvm60/work/.build/./lib/tools/llvm-shlib/simple_version_script.map" \
"--whole-archive" \
"lib/libLLVMCore.a" \
"--no-whole-archive" \
"-lm"

/usr/bin/powerpc64-unknown-freebsd13.0-ld \
"-Bshareable" \
"-o" "lib/libLTO.so.6.0.1" \
"lib/libLLVM-6.0.so"

does not fail the BFD_ASSERT but with the full
lib/libLLVM-6.0.so it does fail.

So it looks like I'll be going back through another reduction
to find (closer to) what is now necessary to fail the
BFD_ASSERT. That may be a while and take a while.
Comment 29 commit-hook freebsd_committer 2019-05-22 14:55:26 UTC
A commit references this bug:

Author: antoine
Date: Wed May 22 14:54:56 UTC 2019
New revision: 502274
URL: https://svnweb.freebsd.org/changeset/ports/502274

Log:
  Remove binutils-do-not-provide-shared-section-symbols.patch, the issue it was
  working around was fixed in binutils version 2.32

  PR:		237068
  Submitted by:	dim

Changes:
  head/devel/binutils/Makefile
  head/devel/binutils/files/binutils-do-not-provide-shared-section-symbols.patch
Comment 30 Mark Millard 2019-05-22 15:20:50 UTC
(In reply to Mark Millard from comment #28)

I'm able to reproduce the messages with the
following for the llvm60 context (no simple
program yet):

"/usr/bin/powerpc64-unknown-freebsd13.0-ld" \
"-Bshareable" \
"-o" "lib/libLLVM-6.0.so" \
"/usr/lib/crtbeginS.o" \
"-L/usr/lib" \
"--version-script" "/wrkdirs/usr/ports/devel/llvm60/work/.build/./lib/tools/llvm-shlib/simple_version_script.map" \
"--whole-archive" \
"lib/libLLVMCore.a" \
"--no-whole-archive" \
"-lm" \
"-lc++" 

/usr/bin/powerpc64-unknown-freebsd13.0-ld \
"-Bshareable" \
"-o" "lib/libLTO.so.6.0.1" \
"lib/libLLVM-6.0.so"


What is additional from comment #13 is the -lc++ . Without
it above I do not get the BFD_ASSERT messages.

But the above is with lib/libLLVMCore.a not being simplified
either: it still has everything from llvm60 that it normally
had. So I've more simplification to do just for
lib/libLLVMCore.a .
Comment 31 Mark Millard 2019-05-22 17:43:21 UTC
(In reply to Mark Millard from comment #30)

For both comment #15 (llvm60 based) and #16
(small example) simply adding the -lc++ is
enough to produce the BFD_ASSERT messages
in my context.

So for the small example variant:

# c++ -c small_link_failure.cpp

# rm -f small_link_failure.a

# /usr/bin/ar qc small_link_failure.a small_link_failure.o

# rm -f small_link_failure.a
# /usr/bin/ar qc small_link_failure.a small_link_failure.o

# /usr/bin/powerpc64-unknown-freebsd13.0-ld  \
"-Bshareable" \
"-o" "small_link_failure-1.0.so" \
"/usr/lib/crtbeginS.o" \
"--version-script" "/wrkdirs/usr/ports/devel/llvm60/work/.build/./lib/tools/llvm-shlib/simple_version_script.map" \
"--whole-archive" \
"small_link_failure.a" \
"--no-whole-archive" \
"-lm" \
"-lc++"

# /usr/bin/powerpc64-unknown-freebsd13.0-ld  \
"-Bshareable" \
"-o" "small_link_failure.so.1.0.1" \
"small_link_failure-1.0.so"

produces as part of the last:

/usr/local/bin/ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935
/usr/local/bin/ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935

The backtrace is still:

(gdb) bt
#0  error_handler_internal (fmt=0x101d91fd "BFD %s assertion fail %s:%d", ap=0x3fffffffffffcf78 "") at bfd.c:1098
#1  0x00000000100202a8 in ld_bfd_error_handler (fmt=<optimized out>, ap=<optimized out>) at ./ldmain.c:187
#2  0x0000000010047630 in _bfd_error_handler (fmt=<optimized out>) at bfd.c:1178
#3  0x0000000010049a54 in _bfd_default_assert_handler (bfd_formatmsg=<optimized out>, bfd_version=<optimized out>, bfd_file=<optimized out>, bfd_line=<optimized out>) at bfd.c:1258
#4  0x0000000010020234 in ld_bfd_assert_handler (fmt=<optimized out>, bfdver=<optimized out>, file=<optimized out>, line=<optimized out>) at ./ldmain.c:177
#5  0x0000000010047890 in bfd_assert (file=<optimized out>, line=<optimized out>) at bfd.c:1429
#6  0x00000000100af998 in _bfd_elf_fix_symbol_flags (h=0x8105d74b8, eif=0x3fffffffffffd1e0) at elflink.c:2935
#7  0x00000000100a001c in _bfd_elf_link_assign_sym_version (h=0x8105d74b8, data=0x3fffffffffffd428) at elflink.c:2363
#8  0x00000000100505d0 in bfd_link_hash_traverse (htab=0x810e2a000, func=@0x1022a138: 0x1009ffb4 <_bfd_elf_link_assign_sym_version>, info=0x3fffffffffffd428) at linker.c:669
#9  0x000000001009da80 in bfd_elf_size_dynamic_sections (output_bfd=0x810e25000, soname=0x0, rpath=0x0, filter_shlib=0x0, audit=0x0, depaudit=0x0, auxiliary_filters=0x0, info=0x10238a38 <link_info>, 
    sinterpptr=0x3fffffffffffd548) at elflink.c:6316
#10 0x000000001002df00 in gldelf64ppc_fbsd_before_allocation () at eelf64ppc_fbsd.c:2095
#11 ppc_before_allocation () at eelf64ppc_fbsd.c:404
#12 0x0000000010025cac in ldemul_before_allocation () at ldemul.c:82
#13 0x0000000010016604 in lang_process () at ldlang.c:7593
#14 0x000000001001fd64 in main (argc=<optimized out>, argv=<optimized out>) at ./ldmain.c:440


And for:

(gdb) up 6 
#6  0x00000000100af998 in _bfd_elf_fix_symbol_flags (h=0x8105d74b8, eif=0x3fffffffffffd1e0) at elflink.c:2935
2935		  BFD_ASSERT (def->root.type == bfd_link_hash_defined);

I still get *h with:

_ZZ18get_execution_seedvE4seed@@LLVM_6.0
root.type = bfd_link_hash_defweak
is_weakalias = 1

*h->u.alias with:

__bss_start@@LLVM_6.0
root.type = bfd_link_hash_indirect


I see via readelf -aegW :

. . .
  [20] .data             PROGBITS        0000000000010da8 000da8 000020 00  WA  0   0  8
  [21] .bss              NOBITS          0000000000010dc8 000dc8 000010 00  WA  0   0  8
. . .
Symbol table (.dynsym) contains 16 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
. . .
     8: 0000000000010dc8     0 NOTYPE  GLOBAL DEFAULT   21 __bss_start@@LLVM_6.0 (2)
     9: 0000000000010dc0     8 OBJECT  GLOBAL DEFAULT   20 fixed_seed_override@@LLVM_6.0 (2)
. . .
Symbol table (.symtab) contains 53 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
. . .
    45: 0000000000010dc8     0 NOTYPE  GLOBAL DEFAULT   21 __bss_start
    46: 0000000000010dc0     8 OBJECT  GLOBAL DEFAULT   20 fixed_seed_override
. . .


Without the -lc++ (so no messages are generated) there
is no _bss_start at all in the readelf output, in fact
the only lines with "bss" are the two lines:

   01     .ctors .dtors .jcr .dynamic .opd .got .plt .data .bss 
. . .
  [20] .bss              NOBITS          0000000000010cc8 000cc8 000010 00  WA  0   0  8

which -lc++ is not listed for producing small_link_failure-1.0.so .
(Before the linker update this did have the symbols.)



It leaves me wondering if I need to rebuild and reinstall world
based on the updated linker in order to avoid having bad material
in system libraries. That may be my next exploration. (In my
powerpc64 context the system linker was/is from base/binutil .)

The same sort of thing might apply to the likes of g++8 and its
libraries if built under the prior linker variant.
Comment 32 Mark Millard 2019-05-22 23:26:45 UTC
(In reply to Mark Millard from comment #31)

buildworld then installworld based on the updated
ld being used was enough to get rid of -lc++ use
leading to the BFD_ASSERT messages for the small
test case.

(The old PowerMac will not have rebuilt devel/llvm60
or devel/llvm80 for some time.)
Comment 33 Antoine Brodin freebsd_committer 2019-05-23 05:38:31 UTC
There is still an assertion when building llvm60 on armv6

http://beefy8.nyi.freebsd.org/data/head-armv6-default/p502274_s348109/logs/errors/llvm60-6.0.1_6.log
Comment 34 Mark Millard 2019-05-23 07:11:01 UTC
(In reply to Mark Millard from comment #32)

The devel/llvm60 build still failed on the old
PowerMac:

/usr/bin/powerpc64-unknown-freebsd13.0-ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935
/usr/bin/powerpc64-unknown-freebsd13.0-ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935

at the same step as prior attempted builds.

(Poudriere is still writing out the tar of the failure.)
Comment 35 Mark Millard 2019-05-23 07:52:02 UTC
(In reply to Mark Millard from comment #34)

Turns out that the devel/libedit package needs to be
rebuilt in poudriere so that -ledit does not pick up
old ld output and put it into the build.

So it is another example of a library eliminated in
the small example contexts that matters in the full
devel/llvm60 context, much like -lc++, but not a
system library reference (rpath picks out the port).

Adding in the -ledit into my somewhat smaller
examples showed the problem.
Comment 36 Mark Millard 2019-05-23 08:41:49 UTC
(In reply to Antoine Brodin from comment #33)

I'll note one other possible oddity I see in that log
file (and also in the logs from my builds): use of
\$ORIGIN in -rpath assignments:

-Wl,-rpath,"\$ORIGIN/../lib:/usr/local/lib"
Comment 37 Antoine Brodin freebsd_committer 2019-05-23 14:15:15 UTC
devel/llvm60 builds fine on head-armv6 after ports r502329 :

http://beefy8.nyi.freebsd.org/data/head-armv6-default/p502330_s348151/logs/llvm60-6.0.1_6.log

Still, there is an incompatibility between llvm and binutils' ld.
Comment 38 Mark Millard 2019-05-23 16:15:23 UTC
(In reply to Mark Millard from comment #35)

powerpc64 devel/llvm60 progress!

devel/llvm60 made to to the packaging stage, finally . . .

[01]: devel/llvm60  | llvm60-6.0.1_6 package (00:00:16 / 07:44:21)

To get there, I first forced the poudriere jail to rebuild/update
its:

devel/libedit textproc/libxml2 converters/libiconv

that devel/llvm60 depends on.

Then I started devel/llvm60 (and devel/llvm80) again.

(devel/llvm80 is still building.)


(I also forced rebuild/updates of: math/gmp math/mpfr math/mpc .)
Comment 39 Mark Millard 2019-05-23 17:57:53 UTC
(In reply to Mark Millard from comment #38)

Both devel/llvm60 and devel/llvm80 finished in my
powerpc64 context:

[08:14:08] [01] [08:11:48] Finished devel/llvm60 | llvm60-6.0.1_6: Success
. . .
[09:20:45] [02] [09:18:25] Finished devel/llvm80 | llvm80-8.0.0_2: Success


This was based on first making sure that all the system and
ports libraries involved had been rebuilt/updated based on
the updated ld (via the patch for devel/binutils).

(I started this effort before the patch had made it to svn.)
Comment 40 Mark Millard 2019-05-26 20:08:43 UTC
(In reply to Dimitry Andric from comment #27)

(In reply to Antoine Brodin from comment #37)

I've not managed to get 32-bit powerpc FreeBSD to build
llvm60 (or other such) using a gcc8/g++8 related toolchain:
same assertion.

I have reduced a reproduction technique for the g++8 context
for 32-bit powerpc FreeBSD -r347549 down to:

/usr/local/bin/ld \
-Bshareable \
-o lib/libLLVM-6.0.so \
/usr/local/lib/gcc8/gcc/powerpc-portbld-freebsd13.0/8.3.0/crtbeginS.o \
--gc-sections \
-soname libLLVM-6.0.so \
--version-script /wrkdirs/usr/ports/devel/llvm60/work/.build/./lib/tools/llvm-shlib/simple_version_script.map \
--whole-archive \
lib/libLLVMSupport.a \
--no-whole-archive \
-ltinfo \
/usr/local/lib/gcc8/gcc/powerpc-portbld-freebsd13.0/8.3.0/crtendS.o

/usr/local/bin/ld \
-Bshareable \
-o lib/libLTO.so.6.0.1 \
/usr/local/lib/gcc8/gcc/powerpc-portbld-freebsd13.0/8.3.0/crtbeginS.o \
--gc-sections \
-soname \
libLTO.so.6 \
lib/libLLVM-6.0.so \
/usr/local/lib/gcc8/gcc/powerpc-portbld-freebsd13.0/8.3.0/crtendS.o

where lib/libLLVM-6.0.so ends up with:

# ldd lib/libLLVM-6.0.so
lib/libLLVM-6.0.so:
	libncurses.so.8 => /lib/libncurses.so.8 (0x41e00000)
	libc.so.7 => /lib/libc.so.7 (0x41862000)

# readelf -asW /lib/libncurses.so.8 | egrep '(bss|Symbol)'
   01     .ctors .dtors .jcr .data.rel.ro .dynamic .data .got .sdata .sbss .plt .bss 
  [21] .sbss             NOBITS          00057448 047448 000070 00  WA  0   0  4
  [23] .bss              NOBITS          00058410 047448 00085c 00  WA  0   0  8
Symbol table (.dynsym) contains 725 entries:
   474: 0000000000057448     0 NOTYPE  GLOBAL DEFAULT  ABS __bss_start
Symbol table (.symtab) contains 752 entries:
   501: 0000000000057448     0 NOTYPE  GLOBAL DEFAULT  ABS __bss_start

(Note: __bss_start has the .sbss 00057448 .)

This ends up producing a lib/libLLVM-6.0.so with:

# readelf -asW lib/libLLVM-6.0.so | egrep '(bss|Symbol )'
   01     .tbss .init_array .ctors .dtors .data.rel.ro .got2 .dynamic .data .got .sdata .sbss .plt .bss 
   03     .tbss 
  [13] .tbss             NOBITS          001522dc 1422dc 000004 00 WAT  0   0  4
  [23] .sbss             NOBITS          001574b8 1474b8 000025 00  WA  0   0  8
  [25] .bss              NOBITS          0015ac78 1474b8 001364 00  WA  0   0  8
Symbol table (.dynsym) contains 2510 entries:
  1390: 00000000001574b8     0 NOTYPE  GLOBAL DEFAULT   23 __bss_start@@LLVM_6.0 (2)
Symbol table (.symtab) contains 3680 entries:
  2560: 00000000001574b8     0 NOTYPE  GLOBAL DEFAULT   23 __bss_start

In other words, the .dynsym ends up with the @@LLVM_6.0 added.

Removing either lib/libLLVMSupport.a or -ltinfo no longer gets
the assert but does not include as much.
Comment 41 Mark Millard 2019-05-26 21:10:05 UTC
(In reply to Mark Millard from comment #40)

I should have noted: this is based on a gcc 4.2.1
toolchain FreeBSD context, not one of my more modern
toolchain experiments. So:

GNU ld (GNU Binutils) 2.32
# /usr/bin/ld -v
GNU ld 2.17.50 [FreeBSD] 2007-07-03


Thus, for example, the /lib/libncurses.so.8 in:

# ldd lib/libLLVM-6.0.so
lib/libLLVM-6.0.so:
	libncurses.so.8 => /lib/libncurses.so.8 (0x41e00000)
	libc.so.7 => /lib/libc.so.7 (0x41862000)

was not produced by the more modern ld :

# /usr/local/bin/ld -v
GNU ld (GNU Binutils) 2.32

but /lib/libncurses.so.8 was used by 2.32 in forming
lib/libLLVM-6.0.so .
Comment 42 Mark Millard 2019-05-26 22:28:23 UTC
(In reply to Mark Millard from comment #41)

Looks like the old 2.17..50 [FreeBSD] 2007-07-03
ld puts __bss_satrt in the .dynsym information
fairly generally:

# find /lib* /usr/lib*  -exec objdump -T {} \; 2>&1 | egrep '(file format elf32-p| D .*__bss_start)' | grep -C1 __bss_start | more
/lib/libnv.so.0:     file format elf32-powerpc-freebsd
0002430c g    D  *ABS*  00000000  Base        __bss_start
/lib/libalias_ftp.so:     file format elf32-powerpc-freebsd
000131ac g    D  *ABS*  00000000  Base        __bss_start
/lib/libzpool.so.2:     file format elf32-powerpc-freebsd
0023c238 g    D  *ABS*  00000000  Base        __bss_start
/lib/libutil.so.9:     file format elf32-powerpc-freebsd
00022610 g    D  *ABS*  00000000  Base        __bss_start
/lib/libncurses.so.8:     file format elf32-powerpc-freebsd
00057448 g    D  *ABS*  00000000  Base        __bss_start
/lib/libipsec.so.4:     file format elf32-powerpc-freebsd
00017404 g    D  *ABS*  00000000  Base        __bss_start
/lib/libctf.so.2:     file format elf32-powerpc-freebsd
0001d460 g    D  *ABS*  00000000  Base        __bss_start
/lib/libncursesw.so.8:     file format elf32-powerpc-freebsd
0006648c g    D  *ABS*  00000000  Base        __bss_start
/lib/libdevstat.so.7:     file format elf32-powerpc-freebsd
00016454 g    D  *ABS*  00000000  Base        __bss_start
/lib/libbsdxml.so.4:     file format elf32-powerpc-freebsd
00035a1c g    D  *ABS*  00000000  Base        __bss_start
/lib/librss.so.1:     file format elf32-powerpc-freebsd
000127a4 g    D  *ABS*  00000000  Base        __bss_start
. . . (I'll not list it all) . . .

So using 2.17.50 is sort of like using 2.32 was before the
removal of the temporary patch, if I gather right: removal
of the temporary patch made 2.32 stop generating these
.dynsym entries and once the libraries involved were rebuilt
the prolem went away in the 2.32 based context.

Looks like the blame may go on 2.17.50's ld. That makes a
mess for 32-bit powerpc built with gcc 4.2.1's tool chain.
Comment 43 Justin Hibbits freebsd_committer 2019-07-09 16:40:46 UTC
I'm still trying to build devel/llvm60 on my powerpcspe machine, and hitting this issue.  I know for a fact that this built just fine back in December, with binutils 2.30 then.  However, rolling back devel/binutils, then also lang/gcc8, to December known-good, it still fails.  So, it appears something changed in base between December and February, that started triggering this assert.
Comment 44 Mark Millard 2019-07-09 17:57:56 UTC
(In reply to Justin Hibbits from comment #43)

In your environment, does the find command from
comment #42 show the __bss_start in the .dynsym
information in anything involved in the link?

Vs.: have you gotten the message without that
being involved?

Comment #38 listed where I eliminated such
__bss_start information in order to have llvm60
and llvm80 build. That should at least be a start
at what to look for having .dynsym information.

As far as I can tell, having __bss_start in the
.dynsym information for the link is sufficient to
have the problem. But it might not be the only
way to have it.
Comment 45 Justin Hibbits freebsd_committer 2019-07-09 19:06:10 UTC
Yes, I do see those symbols, just as in comment #42.  I also see the symbols in the powerpc64 libraries, and powerpc64 builds llvm60 just fine.
Comment 46 Mark Millard 2019-07-09 19:35:16 UTC
(In reply to Justin Hibbits from comment #45)

Okay. Note that beyond eliminating such from what -lc++
pulled in (coment #32 that I fogot to mention), I had to
eliminate such from ports (comment #38 that I mentioned):

For sure: devel/libedit textproc/libxml2 converters/libiconv
Possibly: math/gmp math/mpfr math/mpc

I'm not aware of __bss_start in the .dynsym
information being anywhere else impacting my tests.

I've not seen the problem again in any context after
rebuilding everything (system and ports) to not have
__bss_start in the .dynsym information. But I did have
to rebuild everything involved by using a toolchain
that did not add __bss_start in the .dynsym information,
including in poudriere so that the ports involved would
avoid such as well.

If there is another way for the problem to happen, isolating
it may be easier from an environment that does not have any
examples involved with __bss_start in the .dynsym information .
Unfortunately I've no clue how to cause the problem in such
a context.
Comment 47 Justin Hibbits freebsd_committer 2019-07-09 19:40:52 UTC
Figuring out what symbol it's choking on would help immensely.  Unfortunately, I haven't found the link between the elf_link_hash_entry and a symbol name yet.  Dimitry, might you know anything?
Comment 48 Mark Millard 2019-07-09 19:57:29 UTC
(In reply to Justin Hibbits from comment #47)

Dimitry's comment #% indicated that he used printf's.

My comment #8 shows using gdb. Just after the backtrace
I reported (of course your symbol might be different):

QUOTE
For _bfd_elf_fix_symbol_flags the later gdb output shows that *h has:

_ZZN4llvm7hashing6detail18get_execution_seedEvE4seed@@LLVM_6.0
root.type = bfd_link_hash_defweak
is_weakalias = 1

*h->u.alias has:
__bss_start@@LLVM_6.0
root.type = bfd_link_hash_indirect
is_weakalias = 0
END QUOTE

The full print *h (with notes about from where in the backtrace)
was shown later in the comment.
Comment 49 Mark Millard 2019-07-09 20:01:53 UTC
(In reply to Mark Millard from comment #48)

I should have mentioned that print *h->u.alias was
also shown at the last print in the comment.
Comment 50 Mark Millard 2019-07-09 20:24:58 UTC
(In reply to Mark Millard from comment #49)

In case it helps for having debug information around,
in /usr/ports/Mk/bsd.port.mk I use:

 STRIP_CMD=	${TRUE}
 .endif
 DEBUG_FLAGS?=	-g
+.if defined(ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG)
+CFLAGS:=		${CFLAGS} ${DEBUG_FLAGS}
+.else
 CFLAGS:=		${CFLAGS:N-O*:N-fno-strict*} ${DEBUG_FLAGS}
+.endif
 .if defined(INSTALL_TARGET)
 INSTALL_TARGET:=	${INSTALL_TARGET:S/^install-strip$/install/g}
 .endif

I define ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG to
get the changed build behavior for ports for
poudriere-based builds:

# more /usr/local/etc/poudriere.d/make.conf
WANT_QT_VERBOSE_CONFIGURE=1
#
DEFAULT_VERSIONS+=perl5=5.28 gcc=9 llvm=80
WRKDIRPREFIX?=/wrkdirs
#
# From a local /usr/ports/Mk/bsd.port.mk extension:
ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG=
#
.if ${.CURDIR:M*/devel/llvm*}
#WITH_DEBUG=
.elif ${.CURDIR:M*/lang/cling*}
#WITH_DEBUG=
.elif ${.CURDIR:M*/www/*webkit*}
#WITH_DEBUG=
.else
WITH_DEBUG=
.endif
MALLOC_PRODUCTION=
Comment 51 Justin Hibbits freebsd_committer 2019-07-09 21:15:13 UTC
Mark, sorry I missed that.  It seems to me that a weak alias to an indirect reference should resolve through the indirect reference.  So I think the while (h->root.type == bfd_link_hash_indirect) block needs to be copied and used for def as well.
Comment 52 Mark Millard 2019-07-09 23:33:33 UTC
(In reply to Justin Hibbits from comment #51)

I do not know.

 if (h->is_weakalias)
   {
     struct elf_link_hash_entry *def = weakdef (h);

// Here, so that def->def_regular is not based on bfd_link_hash_indirect ?
// (Not a "real definition"?)

     /* If the real definition is defined by a regular object file,
        don't do anything special.  See the longer description in
        _bfd_elf_adjust_dynamic_symbol, below.  */
     if (def->def_regular)
       {
         h = def;
         while ((h = h->u.alias) != def)
           h->is_weakalias = 0;
       }
     else
       {
         while (h->root.type == bfd_link_hash_indirect)
           h = (struct elf_link_hash_entry *) h->root.u.i.link;

// Here instead?

         BFD_ASSERT (h->root.type == bfd_link_hash_defined
                     || h->root.type == bfd_link_hash_defweak);
         BFD_ASSERT (def->def_dynamic);
         BFD_ASSERT (def->root.type == bfd_link_hash_defined);
         (*bed->elf_backend_copy_indirect_symbol) (eif->info, def, h);
       }
   }


Might:

         while (def->root.type == bfd_link_hash_indirect)
           def = (struct elf_link_hash_entry *) def->root.u.i.link;

(validly) reach something else that is not an example of bfd_link_hash_defined?
Should def->root.type == bfd_link_hash_indirect have been possible in the
first place?

I did not figure out that much about the context for the code.