Bug 233769 - Possible build race: ld: error: unable to find library -lgcc_s
Summary: Possible build race: ld: error: unable to find library -lgcc_s
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-12-04 09:58 UTC by Li-Wen Hsu
Modified: 2019-07-30 20:47 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Li-Wen Hsu freebsd_committer 2018-12-04 09:58:06 UTC
Failed runs:

https://ci.freebsd.org/job/FreeBSD-head-i386-build/9799/console
https://ci.freebsd.org/job/FreeBSD-head-armv7-build/2129/console

Output (from FreeBSD-head-i386-build/9799)

The order looks correct but might be the timing is too close or some path issues.

00:10:50.832 --- lib/libgcc_s__L ---
00:10:50.832 sh /usr/src/tools/install.sh -l rs -o root -g wheel -m 755  /usr/obj/usr/src/i386.i386/tmp/lib/libgcc_s.so.1  /usr/obj/usr/src/i386.i386/tmp/usr/lib/libgcc_s.so

...

00:10:50.836 --- lib/libgssapi__L ---
00:10:50.836 /usr/obj/usr/src/i386.i386/tmp/usr/bin/ld: error: unable to find library -lgcc_s

...

00:10:50.838 --- lib/libgssapi__L ---
00:10:50.838 cc: error: linker command failed with exit code 1 (use -v to see invocation)
Comment 1 Mark Millard 2018-12-04 20:17:48 UTC
Back on 2018-June-18 Bryan Drewery wrote in a response for
something else that looked initially to be a possible race:

If it was -lgcc_s then it's a known rare build race due to
tools/install.sh not handling -S.
Comment 2 Ed Maste freebsd_committer 2019-03-26 00:16:19 UTC
Occurred in a private CI build of mine: https://cirrus-ci.com/task/6381946174177280
Comment 3 Ed Maste freebsd_committer 2019-03-26 03:40:26 UTC
Indeed, the build installs libgcc_s twice to the same location:

% grep 'install.*libgcc_s.so$' ~/Downloads/6381946174177280-main.log
sh /tmp/cirrus-ci-build/tools/install.sh -l rs -o root -g wheel -m 755  /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/lib/libgcc_s.so.1  /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/usr/lib/libgcc_s.so
sh /tmp/cirrus-ci-build/tools/install.sh -l rs -o root -g wheel -m 755  /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/lib/libgcc_s.so.1  /usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp/usr/lib/libgcc_s.so
Comment 4 Ed Maste freebsd_committer 2019-03-26 16:54:12 UTC
In a successful build we actually install libgcc_s.so three times to the same location.
Comment 5 Mark Millard 2019-03-26 19:50:30 UTC
(In reply to Ed Maste from comment #4)

Is there a installworld vs. buildworld confusion? (I'd expect ld to be a
buildworld issue, not a installworld one.)

I see from recent activity, using:

egrep -r '(--- installworld ---|install.*libgcc_s.so)' /root/sys_typescripts/


/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:--- installworld ---
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:--- installworld ---
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install  -s -o root -g wheel -m 444   -S  libgcc_s.so.1 /lib/
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install  -o root -g wheel -m 444    libgcc_s.so.1.debug /usr/lib/debug/lib/
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -l rs -o root -g wheel -m 755  /lib/libgcc_s.so.1  /usr/lib/libgcc_s.so
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install  -s -o root -g wheel -m 444   -S  libgcc_s.so.1 /usr/lib32/
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install  -o root -g wheel -m 444    libgcc_s.so.1.debug /usr/lib/debug/usr/lib32/
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-21:16:45:49:install -l rs -o root -g wheel -m 755  libgcc_s.so.1 /usr/lib32/libgcc_s.so
/root/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-host-2018-10-22:15:07:14:Command: env __MAKE_CONF=/root/src.configs/make.conf SRCCONF=/dev/null SRC_ENV_CONF=/root/src.configs/src.conf.

(So all from the same log file: one for installworld.)

No matches found in files without installworld showing. Similarly for
all the log history the grep went through. (I run buildworld and
installworld in separate runs, not together, so separate log files in
the directory.)

The logs span amd64, armv7, aarch64, powerpc64, and powerpc builds.
I see no evidence of buildworld doing install's of libgcc_s.so materials.
Comment 6 Ed Maste freebsd_committer 2019-03-26 19:57:40 UTC
(In reply to Mark Millard from comment #5)
The installs are done during buildworld as part of the libraries target. In a submake the _prereq_libs, _startup_libs and _generic_libs targets are built in turn and each of these builds and installs libgcc_s.so (see the ${_lib}__PL and ${_lib}__L targets in Makefile.inc1)
Comment 7 Mark Millard 2019-03-26 20:56:45 UTC
(In reply to Ed Maste from comment #6)

Ahh, I needed to look in the .meta files. Sorry.

What you are reporting may mean that some .meta files are
written more than once, so I'd see content from only the
last pass for each. Messy.

For reference (for a first time build of a head -r345520
based tree):

# grep -lr 'CMD .*install.*libgcc_s\.so' /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind
/usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/libgcc_s.so.1.meta
/usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/_libinstall.meta
/usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/libgcc_s.so.1.meta
/usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/_libinstall.meta

So 4 meta files, 2 for lib32 activity and 2 not:

# Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/libgcc_s.so.1.meta
CMD /usr/local/powerpc64-unknown-freebsd13.0/bin/objcopy --strip-debug --add-gnu-debuglink=libgcc_s.so.1.debug  libgcc_s.so.1.full libgcc_s.so.1
CMD @sh /usr/src/tools/install.sh -l s -o root -g wheel -m 444  libgcc_s.so.1 libgcc_s.so
CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc
TARGET libgcc_s.so.1

# Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc/_libinstall.meta
CMD sh /usr/src/tools/install.sh   -o root -g wheel -m 444   -S  libgcc_s.so.1 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/lib/
CMD sh /usr/src/tools/install.sh  -o root -g wheel -m 444    libgcc_s.so.1.debug /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/usr/lib/debug/lib/
CMD sh /usr/src/tools/install.sh -l rs -o root -g wheel -m 755  /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/lib/libgcc_s.so.1  /usr/obj/powerpc64
vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/usr/lib/libgcc_s.so
CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/gnu/lib/libgcc
TARGET _libinstall

# Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/libgcc_s.so.1.meta
CMD /usr/local/powerpc64-unknown-freebsd13.0/bin/objcopy --strip-debug --add-gnu-debuglink=libgcc_s.so.1.debug  libgcc_s.so.1.full libgcc_s.so.1
CMD @sh /usr/src/tools/install.sh -l s -o root -g wheel -m 444  libgcc_s.so.1 libgcc_s.so
CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc
TARGET libgcc_s.so.1

# Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc/_libinstall.meta
CMD sh /usr/src/tools/install.sh   -o root -g wheel -m 444   -S  libgcc_s.so.1 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/tmp/usr/lib32/
CMD sh /usr/src/tools/install.sh  -o root -g wheel -m 444    libgcc_s.so.1.debug /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/tmp/usr/lib/de
bug/usr/lib32/
CMD sh /usr/src/tools/install.sh -l rs -o root -g wheel -m 755  libgcc_s.so.1 /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/tmp/usr/lib32/lib
gcc_s.so
CWD /usr/obj/powerpc64vtsc_clang_altbinutils-oldunwind/powerpc.powerpc64/usr/src/powerpc.powerpc64/obj-lib32/gnu/lib/libgcc
TARGET _libinstall
Comment 8 Mark Millard 2019-03-26 21:46:15 UTC
(In reply to Mark Millard from comment #7)

And, finally what I should have looked for in the overall
logs, using a aarch64 build this time (no lib32):

# egrep '(libgcc_s\.so|libgcc_s.*_libinstall)' ~/sys_typescripts/typescript_make_cortexA72_nodebug_clang_bootstrap-amd64-host-2019-03-26:01:34:22
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/libgcc_s.so.1.full
--- libgcc_s.so.1.full ---
building shared library libgcc_s.so.1
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/libgcc_s.so.1.debug
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/libgcc_s.so.1
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_s/_libinstall

So 3 _libinstall 's for libgcc_s .

Looking at other things in the build there are a lot of pairs of _libinstall 's
and libgcc_eh has 3:

FBSDFSSD# egrep '/_libinstall' ~/sys_typescripts/typescript_make_cortexA72_nodebug_clang_bootstrap-amd64-host-2019-03-26:01:34:22 | sort | more
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libavl/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libavl/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libctf/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libctf/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libdtrace/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libnvpair/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/cddl/lib/libnvpair/_libinstall
. . .
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_eh/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_eh/_libinstall
Building /usr/obj/cortexA72_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libgcc_eh/_libinstall
. .  .
Comment 9 Ed Maste freebsd_committer 2019-07-30 19:21:22 UTC
On Tue, 30 Jul 2019 at 13:38, <jenkins-admin@freebsd.org> wrote:
>
> FreeBSD-head-powerpcspe-build - Build #12186 (r350451) - Failure
>
> Build information: https://ci.freebsd.org/job/FreeBSD-head-powerpcspe-build/12186/
> Full change log: https://ci.freebsd.org/job/FreeBSD-head-powerpcspe-build/12186/changes
> Full build log: https://ci.freebsd.org/job/FreeBSD-head-powerpcspe-build/12186/console
>

13:37:03 /usr/obj/usr/src/powerpc.powerpcspe/tmp/usr/bin/ld: cannot find -lgcc_s
PR 233769 Possible build race: ld: error: unable to find library -lgcc_s
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233769
Comment 10 Bryan Drewery freebsd_committer 2019-07-30 20:05:13 UTC
This is due to tools/install.sh not supporting -S. If that is fixed
then this is easily fixed.
Comment 11 Bryan Drewery freebsd_committer 2019-07-30 20:46:59 UTC
Just passing through -S in tools/install.sh is not enough.
1. Maybe this tool is expected to work on Linux where -S does something different.
2. The symlink/hardlink support call ln(1) which are is atomic for symlinks. It does unlink(2) followed by symlink(2).

Reworking the build to avoid double installs here is possible but would be more hackish IMO. We would need to muck with the SUBDIR lists for gnu/lib to fix this specific case so that it only installs during the right phase of
buildworld (like we do for llvm kinda). But that's JUST gnu/lib. The same
problem could come up for other libraries eventually given enough install-to-worldtmp threads.

META_MODE+filemon may eventually learn how to avoid the double installs but it has not yet. DIRDEPS_BUILD learned how. It's not hard to implement just risky
and needs time devoted to testing. Of course this then requires filemon which
isn't the solution some people want to hear.

The double install/relink issue isn't causing a problem beyond this race which is why I think we just need to make the install atomic rather than wack-a-mole with SUBDIR hacks.
Comment 12 Bryan Drewery freebsd_committer 2019-07-30 20:47:51 UTC
Oh and it's actually the symlink causing the problem here not the library itself.