Bug 185480 - WORLDTMP first in PATH during installworld
Summary: WORLDTMP first in PATH during installworld
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: unspecified
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-01-05 03:20 UTC by na
Modified: 2017-12-31 22:27 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description na 2014-01-05 03:20:00 UTC
Disclaimer: I don't know much about this stuff, but after posting it here: http://lists.freebsd.org/pipermail/freebsd-mips/2014-January/003227.html
I was asked to submit a PR, so here it is.

This only causes failures if one tries to installworld on a different platform than buildworld was run on, but I suspect it may be silently hiding unintentional behavior in the other, more usual cases.

The various subdirs of WORLDTMP (bin, usr/bin, usr/sbin, and so on) are the first elements of PATH during installkernel/installworld. Is this intentional? A couple of things suggest maybe not.

The first is that WORLDTMP contains a toolchain for MACHINE_ARCH, even if you are cross-compiling by setting TARGET_ARCH. So, if you later try to install the crossbuilt kernel/world on the TARGET_ARCH machine, it will try to execute binaries built for the build machine, and fail:

    cc: Exec format error
    make[2]: "/usr/src/share/mk/bsd.compiler.mk" line 9: warning: "cc
--version" returned non-zero status

This is easily remedied by either deleting WORLDTMP between build and install, or changing its definition in Makefile.inc1 to include ${MACHINE_ARCH} in the name before buildworld, so it won't be found by an installworld with a different MACHINE_ARCH. I'm not sure this is the right fix, though, as it leaves installkernel/world with only the existing installed root (/bin, /usr/bin, ...) in its PATH.

The second one is a bit more glaringly obvious. After the above workaround is used, installworld fails a little later:

    install -s -o root -g wheel -m 444   -fschg -S  libc.so.7 /lib
    install: exec(strip): No such file or directory

The PATH here again begins with the root at WORLDTMP, but ends not with the system root but /tmp/install.3aqj2XwQ. I noticed that installworld had just created this directory, and simply neglected to put strip in it, easily fixed by adding strip to the list of ITOOLS= in Makefile.inc1.

However, under normal circumstances, nothing in this new directory is ever used, since everything is found in WORLDTMP first. Can that be right? We just went through the trouble of creating and populating a directory we don't intend to use? It seemed more likely that the intent is to use it, and that WORLDTMP is first in the PATH by mistake.

Fix: 

If it's intentional that WORLDTMP is first in the PATH, then:
  1. why are we bothering to build IROOT?
  2. can it at least be redefined to include ${MACHINE_ARCH} in the name, so it won't break later installs run on a different arch?
  3. if this is to be fully supported, then I guess a version of WORLDTMP needs to be compiled for the TARGET_ARCH as well when cross-building.

If it isn't intentional... well. I guess it's a lot simpler then. :)
How-To-Repeat: server# uname -m
amd64
server# cd /usr/src && svn diff && svn info | egrep '^URL|Revision'
URL: svn://svn0.us-east.freebsd.org/base/releng/10.0
Revision: 260251
server# export MAKEOBJDIRPREFIX=/mnt/obj
server# rm -rf $MAKEOBJDIRPREFIX
server# TARGET=mips TARGET=mips64 make KERNCONF=ERL buildworld buildkernel

client# uname -m
mips
client# mount | grep /usr
server:/usr/src on /usr/src (nfs, read-only)
server:/mnt/obj/mips.mips64 on /usr/obj (nfs, read-only)
client# cd /usr/src && make KERNCONF=ERL installkernel installworld
Comment 1 Brooks Davis freebsd_committer 2014-01-05 19:30:04 UTC
I believe that WORLDTMP is first the path to allow new versions of tools
to be used in the install process.  It's critical that we do this or we
could only use new tool features after multiple major releases.

It is not supported to build on one system and install on another.  It
could be, but it isn't now.  Apparently it's never been a high enough
priority for anyone, probably because there are plenty of workaround.

The simplest workaround is to just do an installworld to some arbitrary
DESTDIR, tar up the result, remove schg flags on the target with
"chflags -R noschg /", and extracting the tarball.  With the -DNO_ROOT
feature I added to the install targets a while back this is easily
accomplished even without root access on the build system.  Just do
installworld with -DNO_ROOT and then use ${DESTDIR}/METALOG as the input
to tar.

-- Brooks
Comment 2 na 2014-01-05 21:03:24 UTC
Thanks for the explanation. That just leaves one question: why are we
bothering to create and populate ${INSTALLTMP}? Under normal
circumstances, where ${WORLDTMP} exists, it doesn't seem to be used.
In fact, it's missing the 'strip' binary, but it doesn't make a
difference until I come along and try to run without ${WORLDTMP}...

As for the rest, if the process is unsupported, then I guess I will
stop doing it :) ... but I'd just like to state for the record that it
seems to work perfectly fine here, after these workarounds. I think
it's also possible to have a "correct" solution, by using the freshly
built world instead of ${WORLDTMP} when ${MACHINE_ARCH} says so. If
there is any interest at all, I can get that working and submit a
patch.

On Sun, Jan 5, 2014 at 2:30 PM, Brooks Davis <brooks@freebsd.org> wrote:
> I believe that WORLDTMP is first the path to allow new versions of tools
> to be used in the install process.  It's critical that we do this or we
> could only use new tool features after multiple major releases.
>
> It is not supported to build on one system and install on another.  It
> could be, but it isn't now.  Apparently it's never been a high enough
> priority for anyone, probably because there are plenty of workaround.
>
> The simplest workaround is to just do an installworld to some arbitrary
> DESTDIR, tar up the result, remove schg flags on the target with
> "chflags -R noschg /", and extracting the tarball.  With the -DNO_ROOT
> feature I added to the install targets a while back this is easily
> accomplished even without root access on the build system.  Just do
> installworld with -DNO_ROOT and then use ${DESTDIR}/METALOG as the input
> to tar.
>
> -- Brooks
Comment 3 Brooks Davis freebsd_committer 2014-01-05 21:27:01 UTC
On Sun, Jan 05, 2014 at 04:03:24PM -0500, Nathan Dorfman wrote:
> Thanks for the explanation. That just leaves one question: why are we
> bothering to create and populate ${INSTALLTMP}? Under normal
> circumstances, where ${WORLDTMP} exists, it doesn't seem to be used.
> In fact, it's missing the 'strip' binary, but it doesn't make a
> difference until I come along and try to run without ${WORLDTMP}...

Hmm, right, I'd forgotten about INSTALLTMP.  With that in mind I think
you may be correct.  We shouldn't need WORLDTMP at all.

> As for the rest, if the process is unsupported, then I guess I will
> stop doing it :) ... but I'd just like to state for the record that it
> seems to work perfectly fine here, after these workarounds. I think
> it's also possible to have a "correct" solution, by using the freshly
> built world instead of ${WORLDTMP} when ${MACHINE_ARCH} says so. If
> there is any interest at all, I can get that working and submit a
> patch.

If it can be done without adding much more complexity it's probably
worth doing.  I'm a bit skeptical that it will be easy to get the built
world to actually work in the general case where things in INSTALLTMP
depend on a new libc but it should be possible.

-- Brooks
Comment 4 Eitan Adler freebsd_committer freebsd_triage 2017-12-31 07:59:59 UTC
For bugs matching the following criteria:

Status: In Progress Changed: (is less than) 2014-06-01

Reset to default assignee and clear in-progress tags.

Mail being skipped