Bug 233735 - Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/endian.h: No such file or directory
Summary: Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/end...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-12-02 21:31 UTC by Li-Wen Hsu
Modified: 2021-04-30 05:40 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Li-Wen Hsu freebsd_committer 2018-12-02 21:31:35 UTC
Failed runs:

https://ci.freebsd.org/job/FreeBSD-head-powerpc-build/8632/console
https://ci.freebsd.org/job/FreeBSD-stable-12-powerpc64-build/190/console
https://ci.freebsd.org/job/FreeBSD-head-riscv64-build/11547/console
https://ci.freebsd.org/job/FreeBSD-stable-12-aarch64-build/109/console
https://ci.freebsd.org/job/FreeBSD-head-armv6-build/2069/console
https://ci.freebsd.org/job/FreeBSD-head-armv7-build/1668/console

From console output, it looks that machine/ is read before or very closed of the time it is created:

00:07:35.870 --- genoffset.o ---
00:07:35.870 cc --sysroot=/usr/obj/usr/src/powerpc.powerpc/tmp -B/usr/obj/usr/src/powerpc.powerpc/tmp/usr/bin -c -O -pipe -g -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -msoft-float -fPIC -fno-omit-frame-pointer -MD -MF.depend.genoffset.o -MTgenoffset.o -mno-altivec -msoft-float -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-uninitialized -fms-extensions -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fms-extensions -Wa,-many -std=iso9899:1999 /usr/src/sys/kern/genoffset.c
00:07:35.870 In file included from /usr/src/sys/sys/param.h:97,
00:07:35.870                  from /usr/src/sys/kern/genoffset.c:35:
00:07:35.871 /usr/src/sys/sys/types.h:46:28: error: machine/endian.h: No such file or directory
00:07:35.871 In file included from /usr/src/sys/sys/types.h:47,
00:07:35.871                  from /usr/src/sys/sys/param.h:97,
00:07:35.871                  from /usr/src/sys/kern/genoffset.c:35:
00:07:35.871 /usr/src/sys/sys/_types.h:35:28: error: machine/_types.h: No such file or directory
00:07:35.871 In file included from /usr/src/sys/sys/types.h:47,
00:07:35.871                  from /usr/src/sys/sys/param.h:97,
00:07:35.871                  from /usr/src/sys/kern/genoffset.c:35:

...

00:07:35.871 --- machine ---
00:07:35.871 machine -> /usr/src/sys/powerpc/include
Comment 1 Li-Wen Hsu freebsd_committer 2018-12-05 18:13:27 UTC
https://ci.freebsd.org/job/FreeBSD-stable-12-i386-build/209/console

00:24:29.399 --- genoffset.o ---
00:24:29.399 cc -target i386-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/i386.i386/tmp -B/usr/obj/usr/src/i386.i386/tmp/usr/bin -c -O -pipe -g -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -MD -MF.depend.genoffset.o -MTgenoffset.o -mno-mmx -mno-sse -msoft-float -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-address-of-packed-member -mno-aes -mno-avx -std=iso9899:1999 /usr/src/sys/kern/genoffset.c
00:24:29.399 In file included from /usr/src/sys/kern/genoffset.c:35:
00:24:29.399 In file included from /usr/src/sys/sys/param.h:96:
00:24:29.399 In file included from /usr/src/sys/sys/types.h:46:
00:24:29.399 ./machine/endian.h:6:10: fatal error: 'x86/endian.h' file not found
00:24:29.399 #include <x86/endian.h>
00:24:29.399          ^~~~~~~~~~~~~~
00:24:29.399 1 error generated.
00:24:29.404 *** [genoffset.o] Error code 1
00:24:29.404 
00:24:29.404 make[2]: stopped in /usr/obj/usr/src/i386.i386/sys/GENERIC
00:24:29.404 --- x86 ---
00:24:29.404 x86 -> /usr/src/sys/x86/include
00:24:29.407 --- machine ---
00:24:29.407 machine -> /usr/src/sys/i386/include
Comment 2 Mark Millard 2018-12-11 21:12:01 UTC
There was a list exchange tied to this back on 2018-Aug-31: Re: svn commit: r338404 - head/sys/conf . Ed Maste and Glen Baber were involved.

Note: -r338404 or anything near it was not likely to be the cause of the problem.

I reported then that:

https://ci.freebsd.org/job/FreeBSD-head-powerpc-build/7454/consoleText

showed the problem for -r339319 but the the prior and later builds
of -r339318 and -r338320 for powerpc did not show the problem. -r339319
was unlikely to be involved in the problem based on its content.

But I vaguely remember seeing this on a rare occasion for a much longer
time. (But I've no specific, public evidence to reference.)
Comment 7 Li-Wen Hsu freebsd_committer 2020-06-04 12:43:30 UTC
Close this for now since this has not happened for more than a year.
Comment 8 Mark Millard 2021-04-30 02:35:55 UTC
(In reply to Li-Wen Hsu from comment #7)

I just got this on release/13.0.0 on arm64 cross building
to target armv7, where buildworld worked fine but buildkernel
then got the problem. (I'll note that I've had other successful
buildworld buildkernel makes in recent days.) I was building
release/13.0.0 from release/13.0.0 (again).

However, I was able to repeat the failure by manually repeating
just the cc command that the .o.meta file reported (from the
directory it also reported). This eliminates any potential for
a race. Yet, before and after doing the cc show the file and
links and such as in place and working and having existed for
some time:

# date
Thu Apr 29 15:55:36 PDT 2021

# ls -PTld /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h
lrwxr-xr-x  1 root  wheel  33 Apr 29 12:38:12 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h -> ../../../sys/arm/include/endian.h

# ls -PTldU /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h
lrwxr-xr-x  1 root  wheel  33 Apr 29 12:38:12 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h -> ../../../sys/arm/include/endian.h

# find / -name endian.h -print | grep sys/arm/include/endian.h 
/usr/13_0R-src/sys/arm/include/endian.h
/usr/13S-src/sys/arm/include/endian.h
/usr/main-src/sys/arm/include/endian.h

# ls -Tld /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/*
-rw-r--r--   1 root  wheel   0 Apr 29 12:32:35 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/_worldtmp
drwxr-xr-x   2 root  wheel   2 Apr 20 21:51:31 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/bin
drwxr-xr-x   4 root  wheel   4 Apr 21 02:26:20 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/etc
drwxr-xr-x   5 root  wheel   6 Apr 29 12:32:36 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/legacy
drwxr-xr-x   4 root  wheel  65 Apr 29 15:10:19 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/lib
drwxrwxr-x   2 root  wheel   2 Apr 21 03:39:07 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/obj-kernel-tools
drwxrwxr-x   9 root  wheel   9 Apr 20 21:52:00 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/obj-tools
lrwxr-xr-x   1 root  wheel  18 Apr 29 12:37:44 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/sys -> /usr/13_0R-src/sys
drwxr-xr-x  12 root  wheel  12 Apr 20 21:51:31 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr

# ls -LTld /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h
-rw-r--r--  1 root  wheel  4311 Apr 28 02:03:34 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h

# ls -LTldU /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h
-rw-r--r--  1 root  wheel  4311 Apr 28 02:03:34 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h

For reference:

# more /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/genoffset.o.meta 
# Meta data file /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/genoffset.o.meta
CMD cc -mcpu=cortex-a7 -target armv7-gnueabihf-freebsd13.0 --sysroot=/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp -B/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/bin -c -O2 -pipe -fno-strict-aliasing -g -nostdinc -I. -I/usr/13_0R-src/sys -I/usr/13_0R-src/sys/contrib/ck/include -I/usr/13_0R-src/sys/contrib/libfdt -I/usr/13_0R-src/sys/contrib/device-tree/include -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -march=armv7a -DLINUX_DTS_VERSION=\""5.9"\" -funwind-tables -fdebug-prefix-map=./machine=/usr/13_0R-src/sys/arm/include -ffreestanding -fwrapv -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-address-of-packed-member -Wno-format-zero-length -mfpu=none -std=iso9899:1999 -fcommon /usr/13_0R-src/sys/kern/genoffset.c
CWD /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7
TARGET genoffset.o
-- command output --
In file included from /usr/13_0R-src/sys/kern/genoffset.c:35:
In file included from /usr/13_0R-src/sys/sys/param.h:99:
/usr/13_0R-src/sys/sys/types.h:46:10: fatal error: 'machine/endian.h' file not found
#include <machine/endian.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.

*** Error code 1

-- filemon acquired metadata --
# filemon version 5
# Target pid 53002
# Start 1619736690.618151
V 5
E 53040 /bin/sh
R 53040 /etc/libmap.conf
R 53040 /var/run/ld-elf.so.hints
R 53040 /lib/libedit.so.8
R 53040 /lib/libc.so.7
R 53040 /lib/libncursesw.so.9
R 53040 /usr/share/locale/C.UTF-8/LC_CTYPE
F 53040 53042
E 53042 /usr/bin/cc
R 53042 /etc/libmap.conf
R 53042 /var/run/ld-elf.so.hints
R 53042 /lib/libz.so.6
R 53042 /usr/lib/libexecinfo.so.1
R 53042 /lib/libncursesw.so.9
R 53042 /lib/libthr.so.3
R 53042 /usr/lib/libc++.so.1
R 53042 /lib/libcxxrt.so.1
R 53042 /lib/libm.so.5
R 53042 /lib/libc.so.7
R 53042 /lib/libelf.so.2
R 53042 /lib/libgcc_s.so.1
R 53042 /usr/13_0R-src/sys/kern/genoffset.c
R 53042 genoffset-2c08fcfd.o.tmp
W 53042 genoffset-2c08fcfd.o.tmp
R 53042 ./opt_global.h
R 53042 /usr/13_0R-src/sys/sys/cdefs.h
R 53042 /usr/13_0R-src/sys/sys/param.h
R 53042 /usr/13_0R-src/sys/sys/_null.h
R 53042 /usr/13_0R-src/sys/sys/types.h
R 53042 /usr/13_0R-src/sys/sys/_types.h
R 53042 /usr/13_0R-src/sys/sys/_pthreadtypes.h
R 53042 /usr/13_0R-src/sys/sys/_stdint.h
R 53042 /usr/13_0R-src/sys/sys/select.h
R 53042 /usr/13_0R-src/sys/sys/_sigset.h
R 53042 /usr/13_0R-src/sys/sys/_timeval.h
R 53042 /usr/13_0R-src/sys/sys/timespec.h
R 53042 /usr/13_0R-src/sys/sys/_timespec.h
R 53042 /usr/13_0R-src/sys/sys/syslimits.h
R 53042 /usr/13_0R-src/sys/sys/errno.h
R 53042 /usr/13_0R-src/sys/sys/time.h
R 53042 /usr/13_0R-src/sys/sys/priority.h
R 53042 /usr/13_0R-src/sys/sys/assym.h
R 53042 /usr/13_0R-src/sys/sys/proc.h
R 53042 /usr/13_0R-src/sys/sys/callout.h
R 53042 /usr/13_0R-src/sys/sys/_callout.h
R 53042 /usr/13_0R-src/sys/sys/queue.h
R 53042 /usr/13_0R-src/sys/sys/event.h
R 53042 /usr/13_0R-src/sys/sys/_eventhandler.h
R 53042 /usr/13_0R-src/sys/sys/condvar.h
R 53042 /usr/13_0R-src/sys/sys/_lock.h
R 53042 /usr/13_0R-src/sys/sys/lock_profile.h
R 53042 /usr/13_0R-src/sys/sys/_mutex.h
R 53042 /usr/13_0R-src/sys/sys/osd.h
R 53042 /usr/13_0R-src/sys/sys/rtprio.h
R 53042 /usr/13_0R-src/sys/sys/runq.h
R 53042 /usr/13_0R-src/sys/sys/resource.h
R 53042 /usr/13_0R-src/sys/sys/sigio.h
R 53042 /usr/13_0R-src/sys/sys/signal.h
R 53042 /usr/13_0R-src/sys/sys/signalvar.h
R 53042 /usr/13_0R-src/sys/sys/pcpu.h
R 53042 /usr/13_0R-src/sys/sys/_cpuset.h
R 53042 /usr/13_0R-src/sys/sys/_bitset.h
R 53042 /usr/13_0R-src/sys/sys/_sx.h
R 53042 /usr/13_0R-src/sys/sys/_rmlock.h
R 53042 /usr/13_0R-src/sys/sys/systm.h
R 53042 /usr/13_0R-src/sys/sys/stdint.h
R 53042 /usr/13_0R-src/sys/sys/kpilite.h
R 53042 /usr/13_0R-src/sys/sys/libkern.h
R 53042 /usr/13_0R-src/sys/sys/ucontext.h
R 53042 /usr/13_0R-src/sys/sys/_ucontext.h
R 53042 /usr/13_0R-src/sys/sys/ucred.h
R 53042 /usr/13_0R-src/sys/bsm/audit.h
R 53042 /usr/13_0R-src/sys/sys/_domainset.h
D 53042 genoffset-2c08fcfd.o.tmp
D 53042 genoffset.o
X 53042 1 0
X 53040 1 0
# Stop 1619736690.746151
# Bye bye

Stage 3.1 reported:

. . .
Building /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/genoffset.o
Building /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/vnode_if_newproto.h
Building /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/vnode_if_typedef.h
Building /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/usbdevs_data.h
Building /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/sdma-imx6q.c
Building /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/vnode_if.c
--- genoffset.o ---
In file included from /usr/13_0R-src/sys/kern/genoffset.c:35:
In file included from /usr/13_0R-src/sys/sys/param.h:99:
/usr/13_0R-src/sys/sys/types.h:46:10: fatal error: 'machine/endian.h' file not found
#include <machine/endian.h>
         ^~~~~~~~~~~~~~~~~~


My build activity here is from experimenting with using zfs
for the first time in many years. So that aspect is unlike
my prior examples of this failure. But it also explains why
I'd rebuild what I'd already built previously.
Comment 9 Mark Millard 2021-04-30 02:45:55 UTC
(In reply to Mark Millard from comment #8)

Fixing some poor wording on my part:

"This eliminates any potential for a race."

I meant: ". . . with other processes in the build".

I was not trying to claim a wider context then
that.
Comment 10 Mark Millard 2021-04-30 03:28:41 UTC
(In reply to Mark Millard from comment #9)

Some experiments with a command adjusted to be tolerable
to gcc10 (just to find the include handling behavior) and
using -v that shows the include paths shows:

#include "..." search starts here:
#include <...> search starts here:
 .
 /usr/13_0R-src/sys
 /usr/13_0R-src/sys/contrib/ck/include
 /usr/13_0R-src/sys/contrib/libfdt
 /usr/13_0R-src/sys/contrib/device-tree/include

and gets:

In file included from /usr/13_0R-src/sys/sys/param.h:99,
                 from /usr/13_0R-src/sys/kern/genoffset.c:35:
/usr/13_0R-src/sys/sys/types.h:46:10: fatal error: machine/endian.h: No such file or directory
   46 | #include <machine/endian.h>
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.

Given the search path listed, the failure makes sense.

It appears that:

--sysroot=/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp

is not ending up contributing to the path list. Same
when I added a -isysroot for the path.

Notably I have:

# zfs list 
NAME                                  USED  AVAIL     REFER  MOUNTPOINT
zroot                                56.2G   137G       96K  /zroot
zroot/BUILDs                         25.3G   137G      120K  /usr/obj/BUILDs
. . .
zroot/BUILDs/13_0R-CA7-nodbg-clang   5.04G   137G     5.04G  /usr/obj/BUILDs/13_0R-CA7-nodbg-clang
. . .

but there is also:

. . .
zroot/usr/13_0R-src                  2.02G   137G     2.02G  /usr/13_0R-src
. . .

and paths with a /usr/13_0R-src prefix are showing up.
(But that does not involve two levels of zfs file systems,
however.)

It does appear that both system-clang and gcc10 are ending
up with similar behavior for the sysroot include file
handling during this failure. (I've not rebooted yet.)
So either I've got an operator-error problem that I've not
noticed or the problem is not specific to system-clang.

(I used gcc10 because I'm not aware of a way to have
clang report its include search path sequence, just
its library search path sequence.)
Comment 11 Mark Millard 2021-04-30 04:20:19 UTC
(In reply to Mark Millard from comment #10)

[Back to system-clang.]

clang may not report its search path sequence but
truss reports the following involving "machine":

52747: fstatat(AT_FDCWD,"./machine",0xffffffff9190,0x0) ERR#2 'No such file or directory'
52747: fstatat(AT_FDCWD,"/usr/13_0R-src/sys/machine",0xffffffff9190,0x0) ERR#2 'No such file or directory'
52747: fstatat(AT_FDCWD,"/usr/13_0R-src/sys/contrib/ck/include/machine",0xffffffff9190,0x0) ERR#2 'No such file or directory'
52747: fstatat(AT_FDCWD,"/usr/13_0R-src/sys/contrib/libfdt/machine",0xffffffff9190,0x0) ERR#2 'No such file or directory'   
52747: fstatat(AT_FDCWD,"/usr/13_0R-src/sys/contrib/device-tree/include/machine",0xffffffff9190,0x0) ERR#2 'No such file or directory'
52747: fstatat(AT_FDCWD,"/usr/13_0R-src/sys/sys/machine",0xffffffff9ab0,0x0) ERR#2 'No such file or directory'

It is not even trying the sysroot as a path prefix for
the include involving "machine".

The usage involving "BUILDs" was:

# grep BUILDs truss.txt
52747: fstatat(AT_FDCWD,"/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/local/cuda",0xffffffffc430,0x0) ERR#2 'No such file or directory'
52747: fstatat(AT_FDCWD,"/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/local/cuda-8.0",0xffffffffc430,0x0) ERR#2 'No such file or directory'
52747: fstatat(AT_FDCWD,"/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/local/cuda-7.5",0xffffffffc430,0x0) ERR#2 'No such file or directory'
52747: fstatat(AT_FDCWD,"/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/local/cuda-7.0",0xffffffffc430,0x0) ERR#2 'No such file or directory'
52747: fstatat(AT_FDCWD,"/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/opt/rocm",0xffffffffc4e0,0x0) ERR#2 'No such file or directory'
52747: fstatat(AT_FDCWD,"/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7",{ mode=drwxr-xr-x ,inode=6,size=3187,blksize=16384 },0x0) = 0 (0x0)
52747: __realpathat(AT_FDCWD,"./opt_global.h","/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/opt_global.h",1024,0) = 0 (0x0)

There is no evidence of any attempted use of sysroot has
a general include path prefix.

A "cc -###" with the other parameters as in the .o.meta
does report:

"-isysroot" "/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp"

So the following would seem to apply if otherwise
/usr/include/ usage would be expected:

# ls -PTldU /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h
lrwxr-xr-x  1 root  wheel  33 Apr 29 12:38:12 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h -> ../../../sys/arm/include/endian.h

# ls -LTld /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h
-rw-r--r--  1 root  wheel  4311 Apr 28 02:03:34 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h

# ls -LTld /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/sys/arm/include/endian.h
-rw-r--r--  1 root  wheel  4311 Apr 28 02:03:34 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/sys/arm/include/endian.h

But I see no evidence that -isysroot ( or any /usr/include )
is being put to use.
Comment 12 Mark Millard 2021-04-30 04:51:21 UTC
(In reply to Mark Millard from comment #11)

Well, I replicated the issue for my "CA72" builds
(instead of the prior "CA7" ones): targeting
cortex-a72 (aarch64). But I experimented in a
different direction after that:

rm -fr /usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/sys/GENERIC-NODBG-CA72

then attempting the buildkernel. That is building
fine. Somehow the older build's content is involved
in the new builds include file handling failure.
I will note that for the older build, the part of
the path-text "13_0R-src" was different because
I'd changed naming conventions.

My builds are META_MODE builds, not that I know such
matters for why the include file is not found.

For CA7, instead of deleting GENERIC-NODBG-CA7, I'll
rename it before starting another kernel build. This
might let me find something interesting by
comparison/contrast of the older and newer material.
Comment 13 Mark Millard 2021-04-30 05:40:23 UTC
(In reply to Mark Millard from comment #12)

[You can skip to the # pwd and later lines to just
see what the below lead to discovering.  (- is new
from-scratch rebuild and + is older rebuild failure
that was not for a from scratch rebuild)]

Well, the prefix differences in the .o.meta file
look like:

 CWD /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7
 TARGET genoffset.o
-OODATE /usr/13_0R-src/sys/kern/genoffset.c opt_global.h machine
+OODATE /usr/13_0R-src/sys/kern/genoffset.c opt_global.h
 -- command output --
+In file included from /usr/13_0R-src/sys/kern/genoffset.c:35:
+In file included from /usr/13_0R-src/sys/sys/param.h:99:
+/usr/13_0R-src/sys/sys/types.h:46:10: fatal error: 'machine/endian.h' file not found
+#include <machine/endian.h>
+         ^~~~~~~~~~~~~~~~~~
+1 error generated.
 
+*** Error code 1
+
. . .

Both new and old show the likes of:

-R 94818 ./opt_global.h
and:
+R 53671 ./opt_global.h

but only new shows lines with "machine":

-R 94818 /usr/13_0R-src/sys/sys/types.h
-R 94818 ./machine/endian.h
-R 94818 /usr/13_0R-src/sys/sys/_types.h
-R 94818 ./machine/_types.h
. . .
-R 94818 /usr/13_0R-src/sys/sys/priority.h
-R 94818 ./machine/param.h
-R 94818 ./machine/_align.h
-R 94818 /usr/13_0R-src/sys/sys/assym.h
. . .
-R 94818 /usr/13_0R-src/sys/sys/runq.h
-R 94818 ./machine/runq.h
-R 94818 /usr/13_0R-src/sys/sys/resource.h
-R 94818 /usr/13_0R-src/sys/sys/sigio.h
-R 94818 /usr/13_0R-src/sys/sys/signal.h
-R 94818 ./machine/_limits.h
-R 94818 ./machine/signal.h
-R 94818 /usr/13_0R-src/sys/sys/signalvar.h
. . .
-R 94818 /usr/13_0R-src/sys/sys/_rmlock.h
-R 94818 ./machine/pcpu.h
-R 94818 ./machine/pcpu_aux.h
-R 94818 /usr/13_0R-src/sys/sys/systm.h
-R 94818 ./machine/atomic.h
-R 94818 /usr/13_0R-src/sys/sys/atomic_common.h
-R 94818 ./machine/armreg.h
-R 94818 ./machine/atomic-v6.h
-R 94818 /usr/13_0R-src/sys/sys/_atomic_subword.h
-R 94818 ./machine/cpufunc.h
-R 94818 /usr/13_0R-src/sys/sys/stdint.h
-R 94818 ./machine/_stdint.h
-R 94818 /usr/13_0R-src/sys/sys/kpilite.h
-R 94818 /usr/13_0R-src/sys/sys/libkern.h
-R 94818 /usr/13_0R-src/sys/sys/ucontext.h
-R 94818 ./machine/ucontext.h
-R 94818 /usr/13_0R-src/sys/sys/_ucontext.h
. . .
-R 94818 /usr/13_0R-src/sys/sys/_domainset.h
-R 94818 ./machine/proc.h
-R 94818 ./machine/utrap.h
-R 94818 ./machine/cpu.h
-R 94818 ./machine/frame.h
-R 94818 ./machine/cpu-v6.h
-R 94818 ./machine/cpuinfo.h
-R 94818 ./machine/sysreg.h
-M 94818 'genoffset-ee1f8ac4.o.tmp' 'genoffset.o'
-X 94818 0 0
-X 94815 0 0
-# Stop 1619758457.627651

The paths with "machine" show ./ prefixes, not
the explicit path I had expected. And that leads
to my incorrect expectations and to the explanation
for the behavior:

# pwd
/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys

# ls -Tld */machine
lrwxr-xr-x  1 root  wheel  24 Apr 21 03:39:09 2021 GENERIC-NODBG-CA7-older/machine -> /usr/src/sys/arm/include
lrwxr-xr-x  1 root  wheel  30 Apr 29 21:54:15 2021 GENERIC-NODBG-CA7/machine -> /usr/13_0R-src/sys/arm/include

The older tree links to an absolute path that starts with /usr/src/
(from before my rename). So "machine" was present locally but pointed
to a path that had nothing to reference. That is enough to apparently
block the compiler from looking via --sysroot's path.

That probably also gets back to the .o.meta file difference:

-OODATE /usr/13_0R-src/sys/kern/genoffset.c opt_global.h machine
+OODATE /usr/13_0R-src/sys/kern/genoffset.c opt_global.h

where the older not-from-scratch rebuild does not list "machine"
but the from-scratch build's one does.

buildworld, by contrast, does not produce such absolute paths that
reach outside a fairly localized sub tree (by comparison). So it
survived the rename in a way that did not mess up the rebuild.

I've no clue if the original intermittent problem was with creating
or using or replacing such symbolic links (whatever the content of
the links). Back then I never even thought to look for this type of
"machine" usage.