Disclaimer: I don't know much about this stuff, but after posting it here: http://lists.freebsd.org/pipermail/freebsd-mips/2014-January/003227.html I was asked to submit a PR, so here it is. This only causes failures if one tries to installworld on a different platform than buildworld was run on, but I suspect it may be silently hiding unintentional behavior in the other, more usual cases. The various subdirs of WORLDTMP (bin, usr/bin, usr/sbin, and so on) are the first elements of PATH during installkernel/installworld. Is this intentional? A couple of things suggest maybe not. The first is that WORLDTMP contains a toolchain for MACHINE_ARCH, even if you are cross-compiling by setting TARGET_ARCH. So, if you later try to install the crossbuilt kernel/world on the TARGET_ARCH machine, it will try to execute binaries built for the build machine, and fail: cc: Exec format error make[2]: "/usr/src/share/mk/bsd.compiler.mk" line 9: warning: "cc --version" returned non-zero status This is easily remedied by either deleting WORLDTMP between build and install, or changing its definition in Makefile.inc1 to include ${MACHINE_ARCH} in the name before buildworld, so it won't be found by an installworld with a different MACHINE_ARCH. I'm not sure this is the right fix, though, as it leaves installkernel/world with only the existing installed root (/bin, /usr/bin, ...) in its PATH. The second one is a bit more glaringly obvious. After the above workaround is used, installworld fails a little later: install -s -o root -g wheel -m 444 -fschg -S libc.so.7 /lib install: exec(strip): No such file or directory The PATH here again begins with the root at WORLDTMP, but ends not with the system root but /tmp/install.3aqj2XwQ. I noticed that installworld had just created this directory, and simply neglected to put strip in it, easily fixed by adding strip to the list of ITOOLS= in Makefile.inc1. However, under normal circumstances, nothing in this new directory is ever used, since everything is found in WORLDTMP first. Can that be right? We just went through the trouble of creating and populating a directory we don't intend to use? It seemed more likely that the intent is to use it, and that WORLDTMP is first in the PATH by mistake. Fix: If it's intentional that WORLDTMP is first in the PATH, then: 1. why are we bothering to build IROOT? 2. can it at least be redefined to include ${MACHINE_ARCH} in the name, so it won't break later installs run on a different arch? 3. if this is to be fully supported, then I guess a version of WORLDTMP needs to be compiled for the TARGET_ARCH as well when cross-building. If it isn't intentional... well. I guess it's a lot simpler then. :) How-To-Repeat: server# uname -m amd64 server# cd /usr/src && svn diff && svn info | egrep '^URL|Revision' URL: svn://svn0.us-east.freebsd.org/base/releng/10.0 Revision: 260251 server# export MAKEOBJDIRPREFIX=/mnt/obj server# rm -rf $MAKEOBJDIRPREFIX server# TARGET=mips TARGET=mips64 make KERNCONF=ERL buildworld buildkernel client# uname -m mips client# mount | grep /usr server:/usr/src on /usr/src (nfs, read-only) server:/mnt/obj/mips.mips64 on /usr/obj (nfs, read-only) client# cd /usr/src && make KERNCONF=ERL installkernel installworld
I believe that WORLDTMP is first the path to allow new versions of tools to be used in the install process. It's critical that we do this or we could only use new tool features after multiple major releases. It is not supported to build on one system and install on another. It could be, but it isn't now. Apparently it's never been a high enough priority for anyone, probably because there are plenty of workaround. The simplest workaround is to just do an installworld to some arbitrary DESTDIR, tar up the result, remove schg flags on the target with "chflags -R noschg /", and extracting the tarball. With the -DNO_ROOT feature I added to the install targets a while back this is easily accomplished even without root access on the build system. Just do installworld with -DNO_ROOT and then use ${DESTDIR}/METALOG as the input to tar. -- Brooks
Thanks for the explanation. That just leaves one question: why are we bothering to create and populate ${INSTALLTMP}? Under normal circumstances, where ${WORLDTMP} exists, it doesn't seem to be used. In fact, it's missing the 'strip' binary, but it doesn't make a difference until I come along and try to run without ${WORLDTMP}... As for the rest, if the process is unsupported, then I guess I will stop doing it :) ... but I'd just like to state for the record that it seems to work perfectly fine here, after these workarounds. I think it's also possible to have a "correct" solution, by using the freshly built world instead of ${WORLDTMP} when ${MACHINE_ARCH} says so. If there is any interest at all, I can get that working and submit a patch. On Sun, Jan 5, 2014 at 2:30 PM, Brooks Davis <brooks@freebsd.org> wrote: > I believe that WORLDTMP is first the path to allow new versions of tools > to be used in the install process. It's critical that we do this or we > could only use new tool features after multiple major releases. > > It is not supported to build on one system and install on another. It > could be, but it isn't now. Apparently it's never been a high enough > priority for anyone, probably because there are plenty of workaround. > > The simplest workaround is to just do an installworld to some arbitrary > DESTDIR, tar up the result, remove schg flags on the target with > "chflags -R noschg /", and extracting the tarball. With the -DNO_ROOT > feature I added to the install targets a while back this is easily > accomplished even without root access on the build system. Just do > installworld with -DNO_ROOT and then use ${DESTDIR}/METALOG as the input > to tar. > > -- Brooks
On Sun, Jan 05, 2014 at 04:03:24PM -0500, Nathan Dorfman wrote: > Thanks for the explanation. That just leaves one question: why are we > bothering to create and populate ${INSTALLTMP}? Under normal > circumstances, where ${WORLDTMP} exists, it doesn't seem to be used. > In fact, it's missing the 'strip' binary, but it doesn't make a > difference until I come along and try to run without ${WORLDTMP}... Hmm, right, I'd forgotten about INSTALLTMP. With that in mind I think you may be correct. We shouldn't need WORLDTMP at all. > As for the rest, if the process is unsupported, then I guess I will > stop doing it :) ... but I'd just like to state for the record that it > seems to work perfectly fine here, after these workarounds. I think > it's also possible to have a "correct" solution, by using the freshly > built world instead of ${WORLDTMP} when ${MACHINE_ARCH} says so. If > there is any interest at all, I can get that working and submit a > patch. If it can be done without adding much more complexity it's probably worth doing. I'm a bit skeptical that it will be easy to get the built world to actually work in the general case where things in INSTALLTMP depend on a new libc but it should be possible. -- Brooks
For bugs matching the following criteria: Status: In Progress Changed: (is less than) 2014-06-01 Reset to default assignee and clear in-progress tags. Mail being skipped