Bug 212877
| Summary: | `make installworld' from r302966 to r306043 fails (bsd.compiler.mk fails to call cc --version) | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | Raphael Kubo da Costa <rakuco> | ||||
| Component: | misc | Assignee: | Bryan Drewery <bdrewery> | ||||
| Status: | Closed FIXED | ||||||
| Severity: | Affects Only Me | CC: | bdrewery, dim, imp, sjg | ||||
| Priority: | --- | ||||||
| Version: | CURRENT | ||||||
| Hardware: | Any | ||||||
| OS: | Any | ||||||
| See Also: | https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215561 | ||||||
| Attachments: |
|
||||||
I see the problem, it's because installworld with WITH_SYSTEM_COMPILER is assuming it will be installed on the same system it was built on. The 11.0 release notes will need an entry stating that building on a system to install on another system using a shared src/obj tree should pass -DWITHOUT_SYSTEM_COMPILER to make when building, or -DWITHOUT_CROSS_COMPILER when installing. Or we need to disable WITH_SYSTEM_COMPILER by default on releng/11.0 in share/mk/src.opts.mk. A commit references this bug: Author: bdrewery Date: Wed Sep 21 21:23:10 UTC 2016 New revision: 306143 URL: https://svnweb.freebsd.org/changeset/base/306143 Log: Disable SYSTEM_COMPILER by default. This is a direct commit to releng/11.0. Having it enabled can lead to a situation where building on one system and installing on another will fail due to not finding cc in the OBJDIR. An actual fix will be made on head separately. PR: 212877 Relnotes: yes Sponsored by: Dell EMC Isilon Approved by: re (gjb) Changes: releng/11.0/share/mk/src.opts.mk I almost have a patch ready for testing. Do you actively do this process still? Yes, so I'd be glad to give it a try. (In reply to Raphael Kubo da Costa from comment #5) > Yes, so I'd be glad to give it a try. Awesome. https://people.freebsd.org/~bdrewery/patches/system-compiler-metadata-cookie.diff This will only fix the problem after a buildworld with the patch present. Then an installworld should be fine. Remember that we're interested in a buildworld where SYSTEM_COMPILER triggers and clang is not built. You can run 'make test-system-compiler' and look for USING_SYSTEM_COMPILER=yes to ensure that clang is not being built before running buildworld. Hmm, I wonder if there's an easy way to reproduce the buggy behavior when __FreeBSD_cc_version hasn't changed. I've just tried `make installworld' (r314877) without any additional parameters on my laptop (without your patch) and everything worked fine. (In reply to Raphael Kubo da Costa from comment #7) > I've just tried `make installworld' (r314877) Sorry, I forgot to say this is also the same revision I'm currently running. (In reply to Raphael Kubo da Costa from comment #7) > Hmm, I wonder if there's an easy way to reproduce the buggy behavior when > __FreeBSD_cc_version hasn't changed. I've just tried `make installworld' > (r314877) without any additional parameters on my laptop (without your > patch) and everything worked fine. Do you have a /usr/obj/usr/src/tmp/usr/bin/cc file? It so then clang built at some point and the installworld is falling back to that. Remove that and you may be able to recreate the initial bug. (In reply to Raphael Kubo da Costa from comment #8) > (In reply to Raphael Kubo da Costa from comment #7) > > I've just tried `make installworld' (r314877) > > Sorry, I forgot to say this is also the same revision I'm currently running. That's relevant too, the installworld needs to *not* trigger the SYSTEM_COMPILER logic. Try setting WITHOUT_SYSTEM_COMPILER=yes (in env) or MK_SYSTEM_COMPILER=no (as make argument) for the installworld. Thanks, I was able to reproduce the bug with MK_SYSTEM_COMPILER=no. Do I also need to set that in buildworld or is it relevant only when installworld'ing from my laptop? (In reply to Raphael Kubo da Costa from comment #11) > Thanks, I was able to reproduce the bug with MK_SYSTEM_COMPILER=no. > > Do I also need to set that in buildworld or is it relevant only when > installworld'ing from my laptop? Only for installworld. So pass that as well with my patch when installing. OK, I've rebuilt world here with your patch, but got the same error when installing it with MK_SYSTEM_COMPILER=no.
FWIW, when I launched `make buildworld' on my beefy machine, it output a message at the beginning:
> make[1]: "/usr/src/Makefile.inc1" line 146: SYSTEM_COMPILER: Determined that CC=cc matches the source tree. Not bootstrapping a cross-compiler.
And there's no /usr/obj/usr/src/tmp/usr/bin/cc in there.
(In reply to Raphael Kubo da Costa from comment #13) > OK, I've rebuilt world here with your patch, but got the same error when > installing it with MK_SYSTEM_COMPILER=no. > Do you have the full output for the installworld? Can you also cat /usr/obj/usr/src/compiler-metadata.mk to here? Created attachment 181065 [details]
make installworld (r314877)
As run from my laptop (which is not building world itself).
(In reply to Bryan Drewery from comment #14) > Can you also cat /usr/obj/usr/src/compiler-metadata.mk to here? I'm assuming you mean /usr/obj/usr/src/tmp/usr/src/compiler-metadata.mk: .info Using cached compiler metadata from build on bsdmonster at Wed Mar 22 17:27:26 CET 2017 COMPILER_VERSION=40000 COMPILER_TYPE=clang COMPILER_FREEBSD_VERSION=1200006 .export COMPILER_VERSION COMPILER_TYPE COMPILER_FREEBSD_VERSION (In reply to Raphael Kubo da Costa from comment #16) > (In reply to Bryan Drewery from comment #14) > > Can you also cat /usr/obj/usr/src/compiler-metadata.mk to here? > > I'm assuming you mean > /usr/obj/usr/src/tmp/usr/src/compiler-metadata.mk: There's the bug, it's in the wrong directory. New patch coming shortly... > > .info Using cached compiler metadata from build on bsdmonster at Wed Mar 22 > 17:27:26 CET 2017 > COMPILER_VERSION=40000 > COMPILER_TYPE=clang > COMPILER_FREEBSD_VERSION=1200006 > .export COMPILER_VERSION COMPILER_TYPE COMPILER_FREEBSD_VERSION Thanks for your testing. I found a way to recreate the original failure locally so can test from here. Ensuring there's no WORLDTMP/usr/bin/cc and passing MK_SYSTEM_COMPILER=no to installworld is enough. FWIW here's the patch but it may break cross-building and external toolchain support and other random things. I'll need to test quite a bit before committing. https://people.freebsd.org/~bdrewery/patches/system-compiler-metadata-cookie-2.diff (In reply to Bryan Drewery from comment #19) > FWIW here's the patch but it may break cross-building and external toolchain > support and other random things. I'll need to test quite a bit before > committing. > > https://people.freebsd.org/~bdrewery/patches/system-compiler-metadata-cookie- > 2.diff This patch isn't quite right, it only worked because I was manually running 'TARGET=amd64 TARGET_ARCH=amd64 make -f Makefile.inc1 compiler-metadata' first. It needs to include ${WORLDTMP}${.CURDIR}/compiler-metadata.mk to work. https://people.freebsd.org/~bdrewery/patches/system-compiler-metadata-cookie-3.diff works better. A commit references this bug: Author: bdrewery Date: Thu Apr 13 22:07:35 UTC 2017 New revision: 316794 URL: https://svnweb.freebsd.org/changeset/base/316794 Log: Cache compiler metadata and reuse it at installworld time. Right after cross-tools, a compiler-metadata.mk file is created that stores all of the bsd.compiler.mk metadata. It is then read in with a fail-safe during installworld time. The file is explicitly removed when invoking cross-tools to ensure that a stale file is not left around from odd manual 'make _cross-tools' -> 'make installworld' invocations. This fixes several issues: - With WITH_SYSTEM_COMPILER (default yes on head and no on releng/11.0): If you build on a system where the bootstrap compiler does not build due to the host compiler matching the in-tree one, but then installworld on another system where that logic fails (a bootstrap compiler is needed), the installworld immediately fails with: sh: cc: not found Note that fixing this logic may then hit a case where a rebuild is attempted in installworld. Normally cc would be ran with 'CFLAGS+=ERROR-tried-to-rebuild-during-make-install' to cause an error such as: cc: error: no such file or directory: 'ERROR-tried-to-rebuild-during-make-install' However, now it will just fail with the 'cc: not found' error. Inspection of the compile line will show 'ERROR-tried-to-rebuild-during-make-install'; It's not useful to set CC to anything other than 'cc' during install as it is more helpful to see the attempted compile rather than some other bogus error. - This now avoids running bsd.compiler.mk (cc executions) even more during installworld. There are compiler-dependent SUBDIR in the tree which required having a compiler during install. There is at least 1 case where CC is still executed in the install, such as from a LOOKUP!= in secure/lib/libcrypto/Makefile.inc checking for 'vzeroall' support. This is not significant for installworld as the lookup has a fallback (and hides its error) and only modifies CFLAGS, thus it's not worth fixing. PR: 212877 MFC after: 2 weeks Sponsored by: Dell EMC Isilon Changes: head/Makefile head/Makefile.inc1 Thanks for fixing this! A commit references this bug: Author: bdrewery Date: Fri Jun 23 16:38:46 UTC 2017 New revision: 320274 URL: https://svnweb.freebsd.org/changeset/base/320274 Log: Set compiler metadata for stageworld/distributeworld. This fixes LD errors during 'make packages' but also for the unlikely case of 'buildworld' on 1 system and 'packages' on another [1]. PR: 212877 [1] MFC after: 2 weeks Sponsored by: Dell EMC Isilon Changes: head/Makefile.inc1 A commit references this bug: Author: bdrewery Date: Wed Aug 8 19:21:09 UTC 2018 New revision: 337496 URL: https://svnweb.freebsd.org/changeset/base/337496 Log: MFC r320274: Set compiler metadata for stageworld/distributeworld. PR: 212877 Changes: _U stable/11/ stable/11/Makefile.inc1 |
I have a setup where I build world and kernel on one beefy machine and set up read-only NFS mounts for /usr/src and /usr/obj to install them into my laptop. This time, I was trying to upgrade from r302966 to r306043. `make installkernel` worked fine, but `make installworld` did not. The failure happens right at the beginning: progs=$(for prog in [ awk cap_mkdb cat chflags chmod chown cmp cp date echo egrep find grep id install ln make mkdir mtree mv pwd_mkdb rm sed services_mkdb sh strip sysctl test true uname wc zic tzsetup makewhatis; do if progpath=`which $prog`; then echo $progpath; else echo "Required tool $prog not found in PATH." >&2; exit 1; fi; done); libs=$(ldd -f "%o %p\n" -f "%o %p\n" $progs 2>/dev/null | sort -u | while read line; do set -- $line; if [ "$2 $3" != "not found" ]; then echo $2; else echo "Required library $1 not found." >&2; exit 1; fi; done); cp $libs $progs /tmp/install.3UpJSkkF cp -R ${PATH_LOCALE:-"/usr/share/locale"} /tmp/install.3UpJSkkF/locale cd /usr/src; MAKEOBJDIRPREFIX=/usr/obj MACHINE_ARCH=amd64 MACHINE=amd64 CPUTYPE=ivybridge GROFF_BIN_PATH=/usr/obj/usr/src/tmp/legacy/usr/bin GROFF_FONT_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/groff_font GROFF_TMAC_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/tmac CC="cc -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin" CXX="c++ -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin" CPP="cpp -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin" AS="as" AR="ar" LD="ld" NM=nm OBJCOPY="objcopy" RANLIB=ranlib STRINGS= SIZE="size" PATH=/usr/obj/usr/src/tmp/legacy/usr/sbin:/usr/obj/usr/src/tmp/legacy/usr/bin:/usr/obj/usr/src/tmp/legacy/bin:/usr/obj/usr/src/tmp/usr/sbin:/usr/obj/usr/src/tmp/usr/bin:/tmp/install.3UpJSkkF LD_LIBRARY_PATH=/tmp/install.3UpJSkkF PATH_LOCALE=/tmp/install.3UpJSkkF/locale /usr/obj/usr/src/make.amd64/bmake -f Makefile.inc1 __MAKE_SHELL=/tmp/install.3UpJSkkF/sh reinstall; MAKEOBJDIRPREFIX=/usr/obj MACHINE_ARCH=amd64 MACHINE=amd64 CPUTYPE=ivybridge GROFF_BIN_PATH=/usr/obj/usr/src/tmp/legacy/usr/bin GROFF_FONT_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/groff_font GROFF_TMAC_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/tmac CC="cc -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin" CXX="c++ -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin" CPP="cpp -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin" AS="as" AR="ar" LD="ld" NM=nm OBJCOPY="objcopy" RANLIB=ranlib STRINGS= SIZE="size" PATH=/usr/obj/usr/src/tmp/legacy/usr/sbin:/usr/obj/usr/src/tmp/legacy/usr/bin:/usr/obj/usr/src/tmp/legacy/bin:/usr/obj/usr/src/tmp/usr/sbin:/usr/obj/usr/src/tmp/usr/bin:/tmp/install.3UpJSkkF LD_LIBRARY_PATH=/tmp/install.3UpJSkkF PATH_LOCALE=/tmp/install.3UpJSkkF/locale rm -rf /tmp/install.3UpJSkkF sh: cc: not found bmake[2]: "/usr/src/share/mk/bsd.compiler.mk" line 145: Unable to determine compiler type for CC=cc -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin. Consider setting COMPILER_TYPE. Note the "sh: cc: not found" part above -- because I'm moving from __FreeBSD_cc_version 1200000 to 1200002, Makefile.inc1 is not setting MK_CLANG_BOOTSTRAP=no, so bsd.compiler.mk is trying to determine the compiler and its version again, but there's no `cc' in the PATH that's being passed, only the actual binary (`clang').