Bug 212877 - `make installworld' from r302966 to r306043 fails (bsd.compiler.mk fails to call cc --version)
Summary: `make installworld' from r302966 to r306043 fails (bsd.compiler.mk fails to c...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Bryan Drewery
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-09-21 09:16 UTC by Raphael Kubo da Costa
Modified: 2018-08-08 19:21 UTC (History)
4 users (show)

See Also:


Attachments
make installworld (r314877) (3.78 KB, text/plain)
2017-03-22 18:00 UTC, Raphael Kubo da Costa
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Raphael Kubo da Costa freebsd_committer freebsd_triage 2016-09-21 09:16:20 UTC
I have a setup where I build world and kernel on one beefy machine and set up read-only NFS mounts for /usr/src and /usr/obj to install them into my laptop.

This time, I was trying to upgrade from r302966 to r306043. `make installkernel` worked fine, but `make installworld` did not. The failure happens right at the beginning:

progs=$(for prog in [ awk cap_mkdb cat chflags chmod chown cmp cp  date echo egrep find grep id install   ln make mkdir mtree mv pwd_mkdb  rm sed services_mkdb sh strip sysctl test true uname wc zic tzsetup   makewhatis; do  if progpath=`which $prog`; then  echo $progpath;  else  echo "Required tool $prog not found in PATH." >&2;  exit 1;  fi;  done);  libs=$(ldd -f "%o %p\n" -f "%o %p\n" $progs 2>/dev/null | sort -u |  while read line; do  set -- $line;  if [ "$2 $3" != "not found" ]; then  echo $2;  else  echo "Required library $1 not found." >&2;  exit 1;  fi;  done);  cp $libs $progs /tmp/install.3UpJSkkF
cp -R ${PATH_LOCALE:-"/usr/share/locale"} /tmp/install.3UpJSkkF/locale
cd /usr/src; MAKEOBJDIRPREFIX=/usr/obj  MACHINE_ARCH=amd64  MACHINE=amd64  CPUTYPE=ivybridge GROFF_BIN_PATH=/usr/obj/usr/src/tmp/legacy/usr/bin  GROFF_FONT_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/groff_font  GROFF_TMAC_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/tmac CC="cc -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin" CXX="c++  -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin"  CPP="cpp -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin"  AS="as" AR="ar" LD="ld" NM=nm  OBJCOPY="objcopy"  RANLIB=ranlib STRINGS=  SIZE="size" PATH=/usr/obj/usr/src/tmp/legacy/usr/sbin:/usr/obj/usr/src/tmp/legacy/usr/bin:/usr/obj/usr/src/tmp/legacy/bin:/usr/obj/usr/src/tmp/usr/sbin:/usr/obj/usr/src/tmp/usr/bin:/tmp/install.3UpJSkkF  LD_LIBRARY_PATH=/tmp/install.3UpJSkkF  PATH_LOCALE=/tmp/install.3UpJSkkF/locale /usr/obj/usr/src/make.amd64/bmake -f Makefile.inc1    __MAKE_SHELL=/tmp/install.3UpJSkkF/sh reinstall;  MAKEOBJDIRPREFIX=/usr/obj  MACHINE_ARCH=amd64  MACHINE=amd64  CPUTYPE=ivybridge GROFF_BIN_PATH=/usr/obj/usr/src/tmp/legacy/usr/bin  GROFF_FONT_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/groff_font  GROFF_TMAC_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/tmac CC="cc -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin" CXX="c++  -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin"  CPP="cpp -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin"  AS="as" AR="ar" LD="ld" NM=nm  OBJCOPY="objcopy"  RANLIB=ranlib STRINGS=  SIZE="size" PATH=/usr/obj/usr/src/tmp/legacy/usr/sbin:/usr/obj/usr/src/tmp/legacy/usr/bin:/usr/obj/usr/src/tmp/legacy/bin:/usr/obj/usr/src/tmp/usr/sbin:/usr/obj/usr/src/tmp/usr/bin:/tmp/install.3UpJSkkF  LD_LIBRARY_PATH=/tmp/install.3UpJSkkF  PATH_LOCALE=/tmp/install.3UpJSkkF/locale rm -rf /tmp/install.3UpJSkkF
sh: cc: not found
bmake[2]: "/usr/src/share/mk/bsd.compiler.mk" line 145: Unable to determine compiler type for CC=cc -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/tmp -B/usr/obj/usr/src/tmp/usr/bin.  Consider setting COMPILER_TYPE.

Note the "sh: cc: not found" part above -- because I'm moving from __FreeBSD_cc_version 1200000 to 1200002, Makefile.inc1 is not setting MK_CLANG_BOOTSTRAP=no, so bsd.compiler.mk is trying to determine the compiler and its version again, but there's no `cc' in the PATH that's being passed, only the actual binary (`clang').
Comment 1 Bryan Drewery freebsd_committer freebsd_triage 2016-09-21 20:05:11 UTC
I see the problem, it's because installworld with WITH_SYSTEM_COMPILER is assuming
it will be installed on the same system it was built on.
Comment 2 Bryan Drewery freebsd_committer freebsd_triage 2016-09-21 20:09:47 UTC
The 11.0 release notes will need an entry stating that building on a system to
install on another system using a shared src/obj tree should pass -DWITHOUT_SYSTEM_COMPILER to make when building, or -DWITHOUT_CROSS_COMPILER
when installing.

Or we need to disable WITH_SYSTEM_COMPILER by default on releng/11.0 in share/mk/src.opts.mk.
Comment 3 commit-hook freebsd_committer freebsd_triage 2016-09-21 21:23:25 UTC
A commit references this bug:

Author: bdrewery
Date: Wed Sep 21 21:23:10 UTC 2016
New revision: 306143
URL: https://svnweb.freebsd.org/changeset/base/306143

Log:
  Disable SYSTEM_COMPILER by default.

  This is a direct commit to releng/11.0.

  Having it enabled can lead to a situation where building
  on one system and installing on another will fail due
  to not finding cc in the OBJDIR.

  An actual fix will be made on head separately.

  PR:		212877
  Relnotes:	yes
  Sponsored by:	Dell EMC Isilon
  Approved by:	re (gjb)

Changes:
  releng/11.0/share/mk/src.opts.mk
Comment 4 Bryan Drewery freebsd_committer freebsd_triage 2017-03-22 16:44:11 UTC
I almost have a patch ready for testing. Do you actively do this process still?
Comment 5 Raphael Kubo da Costa freebsd_committer freebsd_triage 2017-03-22 16:45:28 UTC
Yes, so I'd be glad to give it a try.
Comment 6 Bryan Drewery freebsd_committer freebsd_triage 2017-03-22 16:56:29 UTC
(In reply to Raphael Kubo da Costa from comment #5)
> Yes, so I'd be glad to give it a try.

Awesome.

https://people.freebsd.org/~bdrewery/patches/system-compiler-metadata-cookie.diff

This will only fix the problem after a buildworld with the patch present.  Then an installworld should be fine.

Remember that we're interested in a buildworld where SYSTEM_COMPILER triggers and clang is not built.  You can run 'make test-system-compiler' and look for USING_SYSTEM_COMPILER=yes to ensure that clang is not being built before running buildworld.
Comment 7 Raphael Kubo da Costa freebsd_committer freebsd_triage 2017-03-22 17:14:24 UTC
Hmm, I wonder if there's an easy way to reproduce the buggy behavior when __FreeBSD_cc_version hasn't changed. I've just tried `make installworld' (r314877) without any additional parameters on my laptop (without your patch) and everything worked fine.
Comment 8 Raphael Kubo da Costa freebsd_committer freebsd_triage 2017-03-22 17:15:11 UTC
(In reply to Raphael Kubo da Costa from comment #7)
> I've just tried `make installworld' (r314877)

Sorry, I forgot to say this is also the same revision I'm currently running.
Comment 9 Bryan Drewery freebsd_committer freebsd_triage 2017-03-22 17:15:41 UTC
(In reply to Raphael Kubo da Costa from comment #7)
> Hmm, I wonder if there's an easy way to reproduce the buggy behavior when
> __FreeBSD_cc_version hasn't changed. I've just tried `make installworld'
> (r314877) without any additional parameters on my laptop (without your
> patch) and everything worked fine.

Do you have a /usr/obj/usr/src/tmp/usr/bin/cc file?  It so then clang built at some point and the installworld is falling back to that.  Remove that and you may be able to recreate the initial bug.
Comment 10 Bryan Drewery freebsd_committer freebsd_triage 2017-03-22 17:16:39 UTC
(In reply to Raphael Kubo da Costa from comment #8)
> (In reply to Raphael Kubo da Costa from comment #7)
> > I've just tried `make installworld' (r314877)
> 
> Sorry, I forgot to say this is also the same revision I'm currently running.

That's relevant too, the installworld needs to *not* trigger the SYSTEM_COMPILER logic.  Try setting WITHOUT_SYSTEM_COMPILER=yes (in env) or MK_SYSTEM_COMPILER=no (as make argument) for the installworld.
Comment 11 Raphael Kubo da Costa freebsd_committer freebsd_triage 2017-03-22 17:22:32 UTC
Thanks, I was able to reproduce the bug with MK_SYSTEM_COMPILER=no.

Do I also need to set that in buildworld or is it relevant only when installworld'ing from my laptop?
Comment 12 Bryan Drewery freebsd_committer freebsd_triage 2017-03-22 17:25:41 UTC
(In reply to Raphael Kubo da Costa from comment #11)
> Thanks, I was able to reproduce the bug with MK_SYSTEM_COMPILER=no.
> 
> Do I also need to set that in buildworld or is it relevant only when
> installworld'ing from my laptop?

Only for installworld. So pass that as well with my patch when installing.
Comment 13 Raphael Kubo da Costa freebsd_committer freebsd_triage 2017-03-22 17:45:48 UTC
OK, I've rebuilt world here with your patch, but got the same error when installing it with MK_SYSTEM_COMPILER=no.

FWIW, when I launched `make buildworld' on my beefy machine, it output a message at the beginning:

> make[1]: "/usr/src/Makefile.inc1" line 146: SYSTEM_COMPILER: Determined that CC=cc matches the source tree.  Not bootstrapping a cross-compiler.

And there's no /usr/obj/usr/src/tmp/usr/bin/cc in there.
Comment 14 Bryan Drewery freebsd_committer freebsd_triage 2017-03-22 17:57:05 UTC
(In reply to Raphael Kubo da Costa from comment #13)
> OK, I've rebuilt world here with your patch, but got the same error when
> installing it with MK_SYSTEM_COMPILER=no.
> 

Do you have the full output for the installworld?

Can you also cat /usr/obj/usr/src/compiler-metadata.mk  to here?
Comment 15 Raphael Kubo da Costa freebsd_committer freebsd_triage 2017-03-22 18:00:26 UTC
Created attachment 181065 [details]
make installworld (r314877)

As run from my laptop (which is not building world itself).
Comment 16 Raphael Kubo da Costa freebsd_committer freebsd_triage 2017-03-22 18:01:12 UTC
(In reply to Bryan Drewery from comment #14)
> Can you also cat /usr/obj/usr/src/compiler-metadata.mk  to here?

I'm assuming you mean 
/usr/obj/usr/src/tmp/usr/src/compiler-metadata.mk:

.info Using cached compiler metadata from build on bsdmonster at Wed Mar 22 17:27:26 CET 2017
COMPILER_VERSION=40000
COMPILER_TYPE=clang
COMPILER_FREEBSD_VERSION=1200006
.export COMPILER_VERSION  COMPILER_TYPE  COMPILER_FREEBSD_VERSION
Comment 17 Bryan Drewery freebsd_committer freebsd_triage 2017-03-22 18:02:25 UTC
(In reply to Raphael Kubo da Costa from comment #16)
> (In reply to Bryan Drewery from comment #14)
> > Can you also cat /usr/obj/usr/src/compiler-metadata.mk  to here?
> 
> I'm assuming you mean 
> /usr/obj/usr/src/tmp/usr/src/compiler-metadata.mk:

There's the bug, it's in the wrong directory. New patch coming shortly...

> 
> .info Using cached compiler metadata from build on bsdmonster at Wed Mar 22
> 17:27:26 CET 2017
> COMPILER_VERSION=40000
> COMPILER_TYPE=clang
> COMPILER_FREEBSD_VERSION=1200006
> .export COMPILER_VERSION  COMPILER_TYPE  COMPILER_FREEBSD_VERSION
Comment 18 Bryan Drewery freebsd_committer freebsd_triage 2017-03-22 18:19:09 UTC
Thanks for your testing. I found a way to recreate the original failure locally so can test from here.

Ensuring there's no WORLDTMP/usr/bin/cc and passing MK_SYSTEM_COMPILER=no to installworld is enough.
Comment 19 Bryan Drewery freebsd_committer freebsd_triage 2017-03-22 18:21:05 UTC
FWIW here's the patch but it may break cross-building and external toolchain support and other random things.  I'll need to test quite a bit before committing.

https://people.freebsd.org/~bdrewery/patches/system-compiler-metadata-cookie-2.diff
Comment 20 Bryan Drewery freebsd_committer freebsd_triage 2017-04-11 04:35:55 UTC
(In reply to Bryan Drewery from comment #19)
> FWIW here's the patch but it may break cross-building and external toolchain
> support and other random things.  I'll need to test quite a bit before
> committing.
> 
> https://people.freebsd.org/~bdrewery/patches/system-compiler-metadata-cookie-
> 2.diff

This patch isn't quite right, it only worked because I was manually running
'TARGET=amd64 TARGET_ARCH=amd64 make -f Makefile.inc1 compiler-metadata' first.
It needs to include ${WORLDTMP}${.CURDIR}/compiler-metadata.mk to work.

https://people.freebsd.org/~bdrewery/patches/system-compiler-metadata-cookie-3.diff works better.
Comment 21 commit-hook freebsd_committer freebsd_triage 2017-04-13 22:07:52 UTC
A commit references this bug:

Author: bdrewery
Date: Thu Apr 13 22:07:35 UTC 2017
New revision: 316794
URL: https://svnweb.freebsd.org/changeset/base/316794

Log:
  Cache compiler metadata and reuse it at installworld time.

  Right after cross-tools, a compiler-metadata.mk file is created that
  stores all of the bsd.compiler.mk metadata.  It is then read in
  with a fail-safe during installworld time.

  The file is explicitly removed when invoking cross-tools to ensure that
  a stale file is not left around from odd manual 'make _cross-tools' ->
  'make installworld' invocations.

  This fixes several issues:
      - With WITH_SYSTEM_COMPILER (default yes on head and no on releng/11.0):
        If you build on a system where the bootstrap compiler does not
        build due to the host compiler matching the in-tree one, but then
        installworld on another system where that logic fails (a
        bootstrap compiler is needed), the installworld immediately fails
        with:
             sh: cc: not found
        Note that fixing this logic may then hit a case where a rebuild is
        attempted in installworld.  Normally cc would be ran with
        'CFLAGS+=ERROR-tried-to-rebuild-during-make-install' to cause an
        error such as:
            cc: error: no such file or directory: 'ERROR-tried-to-rebuild-during-make-install'
        However, now it will just fail with the 'cc: not found' error.
        Inspection of the compile line will show
        'ERROR-tried-to-rebuild-during-make-install';  It's not useful to
        set CC to anything other than 'cc' during install as it is more
        helpful to see the attempted compile rather than some other bogus
        error.
      - This now avoids running bsd.compiler.mk (cc executions) even more
        during installworld.  There are compiler-dependent SUBDIR in the
        tree which required having a compiler during install.

  There is at least 1 case where CC is still executed in the install,
  such as from a LOOKUP!= in secure/lib/libcrypto/Makefile.inc checking
  for 'vzeroall' support.  This is not significant for installworld
  as the lookup has a fallback (and hides its error) and only modifies CFLAGS,
  thus it's not worth fixing.

  PR:		212877
  MFC after:	2 weeks
  Sponsored by:	Dell EMC Isilon

Changes:
  head/Makefile
  head/Makefile.inc1
Comment 22 Raphael Kubo da Costa freebsd_committer freebsd_triage 2017-04-13 22:49:51 UTC
Thanks for fixing this!
Comment 23 commit-hook freebsd_committer freebsd_triage 2017-06-23 16:39:52 UTC
A commit references this bug:

Author: bdrewery
Date: Fri Jun 23 16:38:46 UTC 2017
New revision: 320274
URL: https://svnweb.freebsd.org/changeset/base/320274

Log:
  Set compiler metadata for stageworld/distributeworld.

  This fixes LD errors during 'make packages' but also for the unlikely case of
  'buildworld' on 1 system and 'packages' on another [1].

  PR:		212877 [1]
  MFC after:	2 weeks
  Sponsored by:	Dell EMC Isilon

Changes:
  head/Makefile.inc1
Comment 24 commit-hook freebsd_committer freebsd_triage 2018-08-08 19:21:53 UTC
A commit references this bug:

Author: bdrewery
Date: Wed Aug  8 19:21:09 UTC 2018
New revision: 337496
URL: https://svnweb.freebsd.org/changeset/base/337496

Log:
  MFC r320274:

    Set compiler metadata for stageworld/distributeworld.

  PR:	212877

Changes:
_U  stable/11/
  stable/11/Makefile.inc1