When building with -j16 on a 32 core system I get install failures in the tests tree. They look something like this (with different files failing different times): --- _SCRIPTSINS_pkill-i_test --- install -N /home/bed22/freebsd/src/etc -U -M /home/bed22/freebsd/root///METALOG -D /home/bed22/freebsd/root -o root -g wheel -m 555 pkill-i_test /home/bed22/ freebsd/root//usr/tests/bin/pkill/pkill-i_test ... --- _SCRIPTSINS_pkill-i_test --- install: /home/bed22/freebsd/root//usr/tests/bin/pkill/pkill-i_test: Permission denied The make command was: make DESTDIR=/home/bed22/freebsd/root/ -DNO_ROOT -DDB_FROM_SRC -j16 installworld
Indeed, this appears to be reproducible with racing non-root invocations of install. % sh -c "while true; do install -m 555 foo bar; done" & % sh -c "while true; do install -m 555 foo bar; done" & install: bar: Permission denied install: bar: Permission denied install: bar: Permission denied install: bar: No such file or directory ... No errors are reported when those are run as root. Looking at the METALOG from my 8 core system (which hasn't reproduced the issue) shows: feynman% grep -c './usr/tests/bin/pkill/pkill-i_test '/tank/emaste/obj/root/freebsd/METALOG 29 i.e., this (and other pkill tests) are installed 29 times. Of 1571 duplicated installs I see in METALOG, 1546 are in usr/tests
Created attachment 146876 [details] Failing install log a -j16 Add a log showing the failure at Ed's request.
Root cause of this failure is bug 191055
*** This bug has been marked as a duplicate of bug 191055 ***
A commit references this bug: Author: ngie Date: Wed Sep 24 04:45:36 UTC 2014 New revision: 272055 URL: http://svnweb.freebsd.org/changeset/base/272055 Log: Improve bsd.progs.mk compatibility with bsd.prog.mk 1. Do not install FILES/SCRIPTS multiple times if PROGS is specified; this is already handled via bsd.prog.mk when it's called recursively (PR: 191055, 191955). 2. Some variables, like BINDIR and PROGNAME, default to a value if unset whereas others get appended to, like CFLAGS. Add support for the former case (PR: 191056) 3. Make "checkdpadd" and "clean" available targets for recursive execution. Reviewed by: marcel, sjg Phabric: D822 PR: 191055, 191056, 191955 MFC after: 1 week Sponsored by: EMC / Isilon Storage Division Changes: head/share/mk/bsd.progs.mk
Reopening this bug to deal with the layering violation that needs to be fixed in bsd.test.mk with how it deals with bsd.progs.mk/bsd.files.mk/bsd.obj.mk (it's installing FILES twice). This is a day 0 bug with bsd.test.mk from the initial test import that causes FILES to be installed twice instead of once. I have an initial patch that I'm running through a make tinderbox -- once it passes some unit tests and the tinderbox I'll put it out for review.
A commit references this bug: Author: ngie Date: Thu Oct 23 00:54:53 UTC 2014 New revision: 273501 URL: https://svnweb.freebsd.org/changeset/base/273501 Log: MFC r272055: Improve bsd.progs.mk compatibility with bsd.prog.mk 1. Do not install FILES/SCRIPTS multiple times if PROGS is specified; this is already handled via bsd.prog.mk when it's called recursively (PR: 191055, 191955). 2. Some variables, like BINDIR and PROGNAME, default to a value if unset whereas others get appended to, like CFLAGS. Add support for the former case (PR: 191056) 3. Make "checkdpadd" and "clean" available targets for recursive execution. Reviewed by: marcel, sjg Phabric: D822 PR: 191055, 191056, 191955 Sponsored by: EMC / Isilon Storage Division Changes: _U stable/10/ stable/10/share/mk/bsd.progs.mk
A commit references this bug: Author: bdrewery Date: Wed Sep 23 23:20:51 UTC 2015 New revision: 288158 URL: https://svnweb.freebsd.org/changeset/base/288158 Log: Fix most cases of bsd.progs.mk running duplicate or missing commands. This mostly fixes an interaction with bsd.test.mk with PROGS and SCRIPTS. This was most notable with 'make clean' and 'make install', which r281055 and r272055 attempted to address but were inadequate. It also addresses similar issues in bsd.progs.mk when not using bsd.test.mk. This also fixes cases of NOT running commands in the parent when using bsd.progs.mk: - 'make clean' was not run for the main process for Makefiles which had both FILES and SUBDIR but no PROGS or SCRIPTS. This usually was just a leftover Kyuafile.auto. One such example is usr.bin/bmake/tests/sysmk/t1/2. - 'make obj' was not running in the current directory with bsd.test.mk due to early inclusion of bsd.subdir.mk. This was not really a problem due to the SUBDIRS using 'mkdir -p' for their objdirs. There were subtle bugs causing this wrong behavior: 1. bsd.progs.mk needs to set SCRIPTS to empty when recursing to avoid the sub-makes from installing, cleaning or building the SCRIPTS; only the parent make should be doing this. r281055 effectively did the same but wasn't enough. 2. CLEANFILES may contain (especially from *.test.mk) files which only the parent should clean, such as from FILES and SCRIPTS. To resolve sub-makes also cleaning these, reset CLEANFILES and CLEANDIRS in the children before including bsd.prog.mk. A tempting alternative would be to only handle CLEANFILES in the parent but then the child bsd.prog.mk CLEANFILES of per-PROGS wouldn't be setup. 3. bsd.subdir.mk was included too soon in bsd.test.mk. It needs to be included after bsd.prog.mk as the SCRIPTS logic is short-circuitted if 'install:' is already defined (which bsd.subdir.mk does). There is actually no need to include bsd.subdir.mk from bsd.test.mk as bsd.prog.mk and bsd.obj.mk will do so in the proper order. The description in r257095 covers this for FILES and was fixed differently, though changing the handling of target(install) in bsd.prog.mk may make sense after more research. 4. bsd.progs.mk had extra logic to handle recursing SCRIPTS if PROGS was empty, which isn't its business to be doing. SCRIPTS is handled fine by bsd.prog.mk. This mostly reverts and reworks the fix in r259209 and partially reverts r272055. 5. bsd.progs.mk has no need to depend 'all:' on SCRIPTS and FILES. These are handled by bsd.prog.mk/bsd.files.mk fine. This also partially reverts r272055. 6. bsd.progs.mk was not drop-in safe for bsd.prog.mk. Move the PROGS check from r273186 to allow it to be used safely. Specific tested cases: SCRIPTS:no PROGS:no FILES:yes SUBDIR:yes usr.bin/bmake/tests/sysmk/t1/2 SCRIPTS:yes PROGS:no FILES:yes SUBDIR:no usr.bin/bmake/tests/sysmk/t1/2/1 SCRIPTS:yes PROGS:yes FILES:yes SUBDIR:yes lib/libthr/tests SCRIPTS:yes PROGS:no FILES:yes SUBDIR:no usr.bin/yacc/tests libexec/atf/atf-sh/tests A full buildworld/installworld/clean comparison with mtree was also done. The only relevant difference was the new fixed behavior of removing Kyuafile.auto from the objdir in 'clean'. Converting SCRIPTS to be a special case FILES group will make this less fragile and is being explored. One known remaining issue is 'cleandepend' removing the tags files for every recursive call. Note that the 'make clean' command runs for the CURDIR last, which can make it appear to run multiple times when cleaning in tests/, but each command is for a SUBDIR returning up the chain. This is purely bsd.subdir.mk behavior. PR: 191055 PR: 191955 MFC after: 2 weeks Sponsored by: EMC / Isilon Storage Division Changes: head/share/mk/bsd.prog.mk head/share/mk/bsd.progs.mk head/share/mk/bsd.test.mk