Failed runs: https://ci.freebsd.org/job/FreeBSD-head-powerpc-build/8632/console https://ci.freebsd.org/job/FreeBSD-stable-12-powerpc64-build/190/console https://ci.freebsd.org/job/FreeBSD-head-riscv64-build/11547/console https://ci.freebsd.org/job/FreeBSD-stable-12-aarch64-build/109/console https://ci.freebsd.org/job/FreeBSD-head-armv6-build/2069/console https://ci.freebsd.org/job/FreeBSD-head-armv7-build/1668/console From console output, it looks that machine/ is read before or very closed of the time it is created: 00:07:35.870 --- genoffset.o --- 00:07:35.870 cc --sysroot=/usr/obj/usr/src/powerpc.powerpc/tmp -B/usr/obj/usr/src/powerpc.powerpc/tmp/usr/bin -c -O -pipe -g -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -msoft-float -fPIC -fno-omit-frame-pointer -MD -MF.depend.genoffset.o -MTgenoffset.o -mno-altivec -msoft-float -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-uninitialized -fms-extensions -finline-limit=15000 --param inline-unit-growth=100 --param large-function-growth=1000 -fms-extensions -Wa,-many -std=iso9899:1999 /usr/src/sys/kern/genoffset.c 00:07:35.870 In file included from /usr/src/sys/sys/param.h:97, 00:07:35.870 from /usr/src/sys/kern/genoffset.c:35: 00:07:35.871 /usr/src/sys/sys/types.h:46:28: error: machine/endian.h: No such file or directory 00:07:35.871 In file included from /usr/src/sys/sys/types.h:47, 00:07:35.871 from /usr/src/sys/sys/param.h:97, 00:07:35.871 from /usr/src/sys/kern/genoffset.c:35: 00:07:35.871 /usr/src/sys/sys/_types.h:35:28: error: machine/_types.h: No such file or directory 00:07:35.871 In file included from /usr/src/sys/sys/types.h:47, 00:07:35.871 from /usr/src/sys/sys/param.h:97, 00:07:35.871 from /usr/src/sys/kern/genoffset.c:35: ... 00:07:35.871 --- machine --- 00:07:35.871 machine -> /usr/src/sys/powerpc/include
https://ci.freebsd.org/job/FreeBSD-stable-12-i386-build/209/console 00:24:29.399 --- genoffset.o --- 00:24:29.399 cc -target i386-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/i386.i386/tmp -B/usr/obj/usr/src/i386.i386/tmp/usr/bin -c -O -pipe -g -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -MD -MF.depend.genoffset.o -MTgenoffset.o -mno-mmx -mno-sse -msoft-float -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-address-of-packed-member -mno-aes -mno-avx -std=iso9899:1999 /usr/src/sys/kern/genoffset.c 00:24:29.399 In file included from /usr/src/sys/kern/genoffset.c:35: 00:24:29.399 In file included from /usr/src/sys/sys/param.h:96: 00:24:29.399 In file included from /usr/src/sys/sys/types.h:46: 00:24:29.399 ./machine/endian.h:6:10: fatal error: 'x86/endian.h' file not found 00:24:29.399 #include <x86/endian.h> 00:24:29.399 ^~~~~~~~~~~~~~ 00:24:29.399 1 error generated. 00:24:29.404 *** [genoffset.o] Error code 1 00:24:29.404 00:24:29.404 make[2]: stopped in /usr/obj/usr/src/i386.i386/sys/GENERIC 00:24:29.404 --- x86 --- 00:24:29.404 x86 -> /usr/src/sys/x86/include 00:24:29.407 --- machine --- 00:24:29.407 machine -> /usr/src/sys/i386/include
There was a list exchange tied to this back on 2018-Aug-31: Re: svn commit: r338404 - head/sys/conf . Ed Maste and Glen Baber were involved. Note: -r338404 or anything near it was not likely to be the cause of the problem. I reported then that: https://ci.freebsd.org/job/FreeBSD-head-powerpc-build/7454/consoleText showed the problem for -r339319 but the the prior and later builds of -r339318 and -r338320 for powerpc did not show the problem. -r339319 was unlikely to be involved in the problem based on its content. But I vaguely remember seeing this on a rare occasion for a much longer time. (But I've no specific, public evidence to reference.)
https://ci.freebsd.org/job/FreeBSD-head-sparc64-build/11283/console
https://ci.freebsd.org/job/FreeBSD-head-amd64-build/11587/console
https://ci.freebsd.org/job/FreeBSD-head-i386-build/10814/console
https://ci.freebsd.org/job/FreeBSD-stable-12-armv7-build/430/console https://ci.freebsd.org/job/FreeBSD-stable-12-armv7-build/464/console
Close this for now since this has not happened for more than a year.
(In reply to Li-Wen Hsu from comment #7) I just got this on release/13.0.0 on arm64 cross building to target armv7, where buildworld worked fine but buildkernel then got the problem. (I'll note that I've had other successful buildworld buildkernel makes in recent days.) I was building release/13.0.0 from release/13.0.0 (again). However, I was able to repeat the failure by manually repeating just the cc command that the .o.meta file reported (from the directory it also reported). This eliminates any potential for a race. Yet, before and after doing the cc show the file and links and such as in place and working and having existed for some time: # date Thu Apr 29 15:55:36 PDT 2021 # ls -PTld /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h lrwxr-xr-x 1 root wheel 33 Apr 29 12:38:12 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h -> ../../../sys/arm/include/endian.h # ls -PTldU /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h lrwxr-xr-x 1 root wheel 33 Apr 29 12:38:12 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h -> ../../../sys/arm/include/endian.h # find / -name endian.h -print | grep sys/arm/include/endian.h /usr/13_0R-src/sys/arm/include/endian.h /usr/13S-src/sys/arm/include/endian.h /usr/main-src/sys/arm/include/endian.h # ls -Tld /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/* -rw-r--r-- 1 root wheel 0 Apr 29 12:32:35 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/_worldtmp drwxr-xr-x 2 root wheel 2 Apr 20 21:51:31 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/bin drwxr-xr-x 4 root wheel 4 Apr 21 02:26:20 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/etc drwxr-xr-x 5 root wheel 6 Apr 29 12:32:36 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/legacy drwxr-xr-x 4 root wheel 65 Apr 29 15:10:19 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/lib drwxrwxr-x 2 root wheel 2 Apr 21 03:39:07 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/obj-kernel-tools drwxrwxr-x 9 root wheel 9 Apr 20 21:52:00 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/obj-tools lrwxr-xr-x 1 root wheel 18 Apr 29 12:37:44 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/sys -> /usr/13_0R-src/sys drwxr-xr-x 12 root wheel 12 Apr 20 21:51:31 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr # ls -LTld /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h -rw-r--r-- 1 root wheel 4311 Apr 28 02:03:34 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h # ls -LTldU /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h -rw-r--r-- 1 root wheel 4311 Apr 28 02:03:34 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h For reference: # more /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/genoffset.o.meta # Meta data file /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/genoffset.o.meta CMD cc -mcpu=cortex-a7 -target armv7-gnueabihf-freebsd13.0 --sysroot=/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp -B/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/bin -c -O2 -pipe -fno-strict-aliasing -g -nostdinc -I. -I/usr/13_0R-src/sys -I/usr/13_0R-src/sys/contrib/ck/include -I/usr/13_0R-src/sys/contrib/libfdt -I/usr/13_0R-src/sys/contrib/device-tree/include -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -march=armv7a -DLINUX_DTS_VERSION=\""5.9"\" -funwind-tables -fdebug-prefix-map=./machine=/usr/13_0R-src/sys/arm/include -ffreestanding -fwrapv -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-address-of-packed-member -Wno-format-zero-length -mfpu=none -std=iso9899:1999 -fcommon /usr/13_0R-src/sys/kern/genoffset.c CWD /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7 TARGET genoffset.o -- command output -- In file included from /usr/13_0R-src/sys/kern/genoffset.c:35: In file included from /usr/13_0R-src/sys/sys/param.h:99: /usr/13_0R-src/sys/sys/types.h:46:10: fatal error: 'machine/endian.h' file not found #include <machine/endian.h> ^~~~~~~~~~~~~~~~~~ 1 error generated. *** Error code 1 -- filemon acquired metadata -- # filemon version 5 # Target pid 53002 # Start 1619736690.618151 V 5 E 53040 /bin/sh R 53040 /etc/libmap.conf R 53040 /var/run/ld-elf.so.hints R 53040 /lib/libedit.so.8 R 53040 /lib/libc.so.7 R 53040 /lib/libncursesw.so.9 R 53040 /usr/share/locale/C.UTF-8/LC_CTYPE F 53040 53042 E 53042 /usr/bin/cc R 53042 /etc/libmap.conf R 53042 /var/run/ld-elf.so.hints R 53042 /lib/libz.so.6 R 53042 /usr/lib/libexecinfo.so.1 R 53042 /lib/libncursesw.so.9 R 53042 /lib/libthr.so.3 R 53042 /usr/lib/libc++.so.1 R 53042 /lib/libcxxrt.so.1 R 53042 /lib/libm.so.5 R 53042 /lib/libc.so.7 R 53042 /lib/libelf.so.2 R 53042 /lib/libgcc_s.so.1 R 53042 /usr/13_0R-src/sys/kern/genoffset.c R 53042 genoffset-2c08fcfd.o.tmp W 53042 genoffset-2c08fcfd.o.tmp R 53042 ./opt_global.h R 53042 /usr/13_0R-src/sys/sys/cdefs.h R 53042 /usr/13_0R-src/sys/sys/param.h R 53042 /usr/13_0R-src/sys/sys/_null.h R 53042 /usr/13_0R-src/sys/sys/types.h R 53042 /usr/13_0R-src/sys/sys/_types.h R 53042 /usr/13_0R-src/sys/sys/_pthreadtypes.h R 53042 /usr/13_0R-src/sys/sys/_stdint.h R 53042 /usr/13_0R-src/sys/sys/select.h R 53042 /usr/13_0R-src/sys/sys/_sigset.h R 53042 /usr/13_0R-src/sys/sys/_timeval.h R 53042 /usr/13_0R-src/sys/sys/timespec.h R 53042 /usr/13_0R-src/sys/sys/_timespec.h R 53042 /usr/13_0R-src/sys/sys/syslimits.h R 53042 /usr/13_0R-src/sys/sys/errno.h R 53042 /usr/13_0R-src/sys/sys/time.h R 53042 /usr/13_0R-src/sys/sys/priority.h R 53042 /usr/13_0R-src/sys/sys/assym.h R 53042 /usr/13_0R-src/sys/sys/proc.h R 53042 /usr/13_0R-src/sys/sys/callout.h R 53042 /usr/13_0R-src/sys/sys/_callout.h R 53042 /usr/13_0R-src/sys/sys/queue.h R 53042 /usr/13_0R-src/sys/sys/event.h R 53042 /usr/13_0R-src/sys/sys/_eventhandler.h R 53042 /usr/13_0R-src/sys/sys/condvar.h R 53042 /usr/13_0R-src/sys/sys/_lock.h R 53042 /usr/13_0R-src/sys/sys/lock_profile.h R 53042 /usr/13_0R-src/sys/sys/_mutex.h R 53042 /usr/13_0R-src/sys/sys/osd.h R 53042 /usr/13_0R-src/sys/sys/rtprio.h R 53042 /usr/13_0R-src/sys/sys/runq.h R 53042 /usr/13_0R-src/sys/sys/resource.h R 53042 /usr/13_0R-src/sys/sys/sigio.h R 53042 /usr/13_0R-src/sys/sys/signal.h R 53042 /usr/13_0R-src/sys/sys/signalvar.h R 53042 /usr/13_0R-src/sys/sys/pcpu.h R 53042 /usr/13_0R-src/sys/sys/_cpuset.h R 53042 /usr/13_0R-src/sys/sys/_bitset.h R 53042 /usr/13_0R-src/sys/sys/_sx.h R 53042 /usr/13_0R-src/sys/sys/_rmlock.h R 53042 /usr/13_0R-src/sys/sys/systm.h R 53042 /usr/13_0R-src/sys/sys/stdint.h R 53042 /usr/13_0R-src/sys/sys/kpilite.h R 53042 /usr/13_0R-src/sys/sys/libkern.h R 53042 /usr/13_0R-src/sys/sys/ucontext.h R 53042 /usr/13_0R-src/sys/sys/_ucontext.h R 53042 /usr/13_0R-src/sys/sys/ucred.h R 53042 /usr/13_0R-src/sys/bsm/audit.h R 53042 /usr/13_0R-src/sys/sys/_domainset.h D 53042 genoffset-2c08fcfd.o.tmp D 53042 genoffset.o X 53042 1 0 X 53040 1 0 # Stop 1619736690.746151 # Bye bye Stage 3.1 reported: . . . Building /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/genoffset.o Building /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/vnode_if_newproto.h Building /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/vnode_if_typedef.h Building /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/usbdevs_data.h Building /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/sdma-imx6q.c Building /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/vnode_if.c --- genoffset.o --- In file included from /usr/13_0R-src/sys/kern/genoffset.c:35: In file included from /usr/13_0R-src/sys/sys/param.h:99: /usr/13_0R-src/sys/sys/types.h:46:10: fatal error: 'machine/endian.h' file not found #include <machine/endian.h> ^~~~~~~~~~~~~~~~~~ My build activity here is from experimenting with using zfs for the first time in many years. So that aspect is unlike my prior examples of this failure. But it also explains why I'd rebuild what I'd already built previously.
(In reply to Mark Millard from comment #8) Fixing some poor wording on my part: "This eliminates any potential for a race." I meant: ". . . with other processes in the build". I was not trying to claim a wider context then that.
(In reply to Mark Millard from comment #9) Some experiments with a command adjusted to be tolerable to gcc10 (just to find the include handling behavior) and using -v that shows the include paths shows: #include "..." search starts here: #include <...> search starts here: . /usr/13_0R-src/sys /usr/13_0R-src/sys/contrib/ck/include /usr/13_0R-src/sys/contrib/libfdt /usr/13_0R-src/sys/contrib/device-tree/include and gets: In file included from /usr/13_0R-src/sys/sys/param.h:99, from /usr/13_0R-src/sys/kern/genoffset.c:35: /usr/13_0R-src/sys/sys/types.h:46:10: fatal error: machine/endian.h: No such file or directory 46 | #include <machine/endian.h> | ^~~~~~~~~~~~~~~~~~ compilation terminated. Given the search path listed, the failure makes sense. It appears that: --sysroot=/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp is not ending up contributing to the path list. Same when I added a -isysroot for the path. Notably I have: # zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 56.2G 137G 96K /zroot zroot/BUILDs 25.3G 137G 120K /usr/obj/BUILDs . . . zroot/BUILDs/13_0R-CA7-nodbg-clang 5.04G 137G 5.04G /usr/obj/BUILDs/13_0R-CA7-nodbg-clang . . . but there is also: . . . zroot/usr/13_0R-src 2.02G 137G 2.02G /usr/13_0R-src . . . and paths with a /usr/13_0R-src prefix are showing up. (But that does not involve two levels of zfs file systems, however.) It does appear that both system-clang and gcc10 are ending up with similar behavior for the sysroot include file handling during this failure. (I've not rebooted yet.) So either I've got an operator-error problem that I've not noticed or the problem is not specific to system-clang. (I used gcc10 because I'm not aware of a way to have clang report its include search path sequence, just its library search path sequence.)
(In reply to Mark Millard from comment #10) [Back to system-clang.] clang may not report its search path sequence but truss reports the following involving "machine": 52747: fstatat(AT_FDCWD,"./machine",0xffffffff9190,0x0) ERR#2 'No such file or directory' 52747: fstatat(AT_FDCWD,"/usr/13_0R-src/sys/machine",0xffffffff9190,0x0) ERR#2 'No such file or directory' 52747: fstatat(AT_FDCWD,"/usr/13_0R-src/sys/contrib/ck/include/machine",0xffffffff9190,0x0) ERR#2 'No such file or directory' 52747: fstatat(AT_FDCWD,"/usr/13_0R-src/sys/contrib/libfdt/machine",0xffffffff9190,0x0) ERR#2 'No such file or directory' 52747: fstatat(AT_FDCWD,"/usr/13_0R-src/sys/contrib/device-tree/include/machine",0xffffffff9190,0x0) ERR#2 'No such file or directory' 52747: fstatat(AT_FDCWD,"/usr/13_0R-src/sys/sys/machine",0xffffffff9ab0,0x0) ERR#2 'No such file or directory' It is not even trying the sysroot as a path prefix for the include involving "machine". The usage involving "BUILDs" was: # grep BUILDs truss.txt 52747: fstatat(AT_FDCWD,"/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/local/cuda",0xffffffffc430,0x0) ERR#2 'No such file or directory' 52747: fstatat(AT_FDCWD,"/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/local/cuda-8.0",0xffffffffc430,0x0) ERR#2 'No such file or directory' 52747: fstatat(AT_FDCWD,"/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/local/cuda-7.5",0xffffffffc430,0x0) ERR#2 'No such file or directory' 52747: fstatat(AT_FDCWD,"/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/local/cuda-7.0",0xffffffffc430,0x0) ERR#2 'No such file or directory' 52747: fstatat(AT_FDCWD,"/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/opt/rocm",0xffffffffc4e0,0x0) ERR#2 'No such file or directory' 52747: fstatat(AT_FDCWD,"/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7",{ mode=drwxr-xr-x ,inode=6,size=3187,blksize=16384 },0x0) = 0 (0x0) 52747: __realpathat(AT_FDCWD,"./opt_global.h","/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7/opt_global.h",1024,0) = 0 (0x0) There is no evidence of any attempted use of sysroot has a general include path prefix. A "cc -###" with the other parameters as in the .o.meta does report: "-isysroot" "/usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp" So the following would seem to apply if otherwise /usr/include/ usage would be expected: # ls -PTldU /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h lrwxr-xr-x 1 root wheel 33 Apr 29 12:38:12 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h -> ../../../sys/arm/include/endian.h # ls -LTld /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h -rw-r--r-- 1 root wheel 4311 Apr 28 02:03:34 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/usr/include/machine/endian.h # ls -LTld /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/sys/arm/include/endian.h -rw-r--r-- 1 root wheel 4311 Apr 28 02:03:34 2021 /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/tmp/sys/arm/include/endian.h But I see no evidence that -isysroot ( or any /usr/include ) is being put to use.
(In reply to Mark Millard from comment #11) Well, I replicated the issue for my "CA72" builds (instead of the prior "CA7" ones): targeting cortex-a72 (aarch64). But I experimented in a different direction after that: rm -fr /usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/sys/GENERIC-NODBG-CA72 then attempting the buildkernel. That is building fine. Somehow the older build's content is involved in the new builds include file handling failure. I will note that for the older build, the part of the path-text "13_0R-src" was different because I'd changed naming conventions. My builds are META_MODE builds, not that I know such matters for why the include file is not found. For CA7, instead of deleting GENERIC-NODBG-CA7, I'll rename it before starting another kernel build. This might let me find something interesting by comparison/contrast of the older and newer material.
(In reply to Mark Millard from comment #12) [You can skip to the # pwd and later lines to just see what the below lead to discovering. (- is new from-scratch rebuild and + is older rebuild failure that was not for a from scratch rebuild)] Well, the prefix differences in the .o.meta file look like: CWD /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys/GENERIC-NODBG-CA7 TARGET genoffset.o -OODATE /usr/13_0R-src/sys/kern/genoffset.c opt_global.h machine +OODATE /usr/13_0R-src/sys/kern/genoffset.c opt_global.h -- command output -- +In file included from /usr/13_0R-src/sys/kern/genoffset.c:35: +In file included from /usr/13_0R-src/sys/sys/param.h:99: +/usr/13_0R-src/sys/sys/types.h:46:10: fatal error: 'machine/endian.h' file not found +#include <machine/endian.h> + ^~~~~~~~~~~~~~~~~~ +1 error generated. +*** Error code 1 + . . . Both new and old show the likes of: -R 94818 ./opt_global.h and: +R 53671 ./opt_global.h but only new shows lines with "machine": -R 94818 /usr/13_0R-src/sys/sys/types.h -R 94818 ./machine/endian.h -R 94818 /usr/13_0R-src/sys/sys/_types.h -R 94818 ./machine/_types.h . . . -R 94818 /usr/13_0R-src/sys/sys/priority.h -R 94818 ./machine/param.h -R 94818 ./machine/_align.h -R 94818 /usr/13_0R-src/sys/sys/assym.h . . . -R 94818 /usr/13_0R-src/sys/sys/runq.h -R 94818 ./machine/runq.h -R 94818 /usr/13_0R-src/sys/sys/resource.h -R 94818 /usr/13_0R-src/sys/sys/sigio.h -R 94818 /usr/13_0R-src/sys/sys/signal.h -R 94818 ./machine/_limits.h -R 94818 ./machine/signal.h -R 94818 /usr/13_0R-src/sys/sys/signalvar.h . . . -R 94818 /usr/13_0R-src/sys/sys/_rmlock.h -R 94818 ./machine/pcpu.h -R 94818 ./machine/pcpu_aux.h -R 94818 /usr/13_0R-src/sys/sys/systm.h -R 94818 ./machine/atomic.h -R 94818 /usr/13_0R-src/sys/sys/atomic_common.h -R 94818 ./machine/armreg.h -R 94818 ./machine/atomic-v6.h -R 94818 /usr/13_0R-src/sys/sys/_atomic_subword.h -R 94818 ./machine/cpufunc.h -R 94818 /usr/13_0R-src/sys/sys/stdint.h -R 94818 ./machine/_stdint.h -R 94818 /usr/13_0R-src/sys/sys/kpilite.h -R 94818 /usr/13_0R-src/sys/sys/libkern.h -R 94818 /usr/13_0R-src/sys/sys/ucontext.h -R 94818 ./machine/ucontext.h -R 94818 /usr/13_0R-src/sys/sys/_ucontext.h . . . -R 94818 /usr/13_0R-src/sys/sys/_domainset.h -R 94818 ./machine/proc.h -R 94818 ./machine/utrap.h -R 94818 ./machine/cpu.h -R 94818 ./machine/frame.h -R 94818 ./machine/cpu-v6.h -R 94818 ./machine/cpuinfo.h -R 94818 ./machine/sysreg.h -M 94818 'genoffset-ee1f8ac4.o.tmp' 'genoffset.o' -X 94818 0 0 -X 94815 0 0 -# Stop 1619758457.627651 The paths with "machine" show ./ prefixes, not the explicit path I had expected. And that leads to my incorrect expectations and to the explanation for the behavior: # pwd /usr/obj/BUILDs/13_0R-CA7-nodbg-clang/usr/13_0R-src/arm.armv7/sys # ls -Tld */machine lrwxr-xr-x 1 root wheel 24 Apr 21 03:39:09 2021 GENERIC-NODBG-CA7-older/machine -> /usr/src/sys/arm/include lrwxr-xr-x 1 root wheel 30 Apr 29 21:54:15 2021 GENERIC-NODBG-CA7/machine -> /usr/13_0R-src/sys/arm/include The older tree links to an absolute path that starts with /usr/src/ (from before my rename). So "machine" was present locally but pointed to a path that had nothing to reference. That is enough to apparently block the compiler from looking via --sysroot's path. That probably also gets back to the .o.meta file difference: -OODATE /usr/13_0R-src/sys/kern/genoffset.c opt_global.h machine +OODATE /usr/13_0R-src/sys/kern/genoffset.c opt_global.h where the older not-from-scratch rebuild does not list "machine" but the from-scratch build's one does. buildworld, by contrast, does not produce such absolute paths that reach outside a fairly localized sub tree (by comparison). So it survived the rename in a way that did not mess up the rebuild. I've no clue if the original intermittent problem was with creating or using or replacing such symbolic links (whatever the content of the links). Back then I never even thought to look for this type of "machine" usage.
^Triage: Re-open given (recent) comments post close and a new issue (see also)
(In reply to Kubilay Kocak from comment #14) What is the see also ticket #?
(In reply to Li-Wen Hsu from comment #15) Whoops, added issue ID. Also comments post original close