Created attachment 237151 [details] src.conf that will reproduce the issue Context: FreeBSD buildworld will succeed with all WITHOUT_* build options enable, minus WITHOUT_AUTO_OBJ WITHOUT_UNIFIED_OBJDIR WITHOUT_INSTALLLIB. This provides a far more efficient exercise of the the available build options than the /usr/src/tools/tools/build_option_survey This commit and review causes the WITHOUT_MACHDEP_OPTIMIZATION build option to fail on "error: undefined symbol: __stpcpy": https://reviews.freebsd.org/D36463 https://cgit.freebsd.org/src/commit/?id=c65e42dbde4198ce46aef7ddac061390abe672dc Xin LI is aware of this issue and has kindly offered this clarification and requested this ticket: "The build broke because some MD files were used, and some were not. For strcpy()'s case, on amd64, it's a wrapper around the MD stpcpy() and naturally it calls __stpcpy() which itself gets (correctly) excluded. The MD C source was used because this line which is common for Makefile.inc's in the subdirectories in libc: .PATH: ${LIBC_SRCTOP}/${LIBC_ARCH}/string ${LIBC_SRCTOP}/string which basically means, if you have foo.c in the MD directory, it's used in favor of the MI version. So these need to be fixed for ${MK_MACHDEP_OPTIMIZATIONS} == no case too. I think this could have been avoided if the initial testing was done with two clean buildworld builds with and without the new option; for build infrastructure changes like this, I'd recommend doing full universe runs locally or on universe14{a,b} at cluster in the future before pushing." Thank you Xin for looking into this! Attachments: a src.conf that reproduces the issue on main and stable/13
Created attachment 237152 [details] Build output with the resulting error
Building with WITHOUT_MACHDEP_OPTIMIZATION=yes only works fine. Building with your src.conf results in ld: error: /usr/obj/usr/src/amd64.amd64/tmp/usr/lib/libgcc_s.so: undefined reference to logAPIs [--no-allow-shlib-undefined] ld: error: /usr/obj/usr/src/amd64.amd64/tmp/usr/lib/libgcc_s.so: undefined reference to logUnwinding [--no-allow-shlib-undefined] ld: error: /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to yp_next [--no-allow-shlib-undefined] ld: error: /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to yp_match [--no-allow-shlib-undefined] ld: error: /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to yp_first [--no-allow-shlib-undefined] How do I reproduce the problem?
(In reply to Gleb Popov from comment #2) What branch are you building? I see this on main and stable/13: git -C /usr/src branch * stable/13 cat /etc/src.conf WITHOUT_MACHDEP_OPTIMIZATIONS=YES git -C /usr/src log|head commit e70ce8d3ae1ba3d73e8efe833c941cda6a3bcf5a Author: Konstantin Belousov <kib@FreeBSD.org> Date: Sat Oct 8 11:33:11 2022 +0300 ... cd /usr/src && make -j4 buildworld > ../output.log tail ../output.log ... --- init_test.full --- cc -target x86_64-unknown-freebsd13.1 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -O2 -pipe -fno-common -I/usr/src/lib/csu/amd64 -g -std=gnu99 -Wno-format-zero-length -fstack-protector-strong -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-error=unused-but-set-variable -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef -Wno-address-of-packed-member -Qunused-arguments -o init_test.full init_test.o -lprivateatf-c ld: error: /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to __stpcpy [--no-allow-shlib-undefined] cc: error: linker command failed with exit code 1 (use -v to see invocation) --- all_subdir_cddl --- make[2]: stopped in /usr/src --- all_subdir_bin --- ... I will pull main and build again.
(In reply to Michael Dexter from comment #3) I'm building 14-CURRENT. Should I use src.conf you provided or just a single WITHOUT_MACHDEP_OPTIMIZATION=YES line?
(In reply to Gleb Popov from comment #4) Start with just the one entry to src.conf but I will not rule out there being more than one issue or a race condition. I updated my stable/13 and main branches and built again with only the one entry: # cat /etc/src.conf WITHOUT_MACHDEP_OPTIMIZATIONS=YES * stable/13 commit 67d863df0833cdb1ca054a7c4e7d551b4ad28d95 Author: Jose Luis Duran <jlduran@gmail.com> Date: Tue Oct 11 12:39:26 2022 -0400 le -Wno-error=unused-but-set-variable -Qunused-arguments -pie -o cat.full cat.o ld: error: /usr/obj/b/stable/13/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to __stpcpy [--no-allow-shlib-undefined] cc: error: linker command failed with exit code 1 (use -v to see invocation) *** [cat.full] Error code 1 make[4]: stopped in /b/stable/13/src/bin/cat * main commit 2e0e27392732a811b52c002a707ac848bad01b30 Author: Gleb Smirnoff <glebius@FreeBSD.org> Date: Thu Oct 13 09:03:38 2022 -0700 --- all_subdir_lib --- ld: error: /usr/obj/b/main/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to __stpcpy [--no-allow-shlib-undefined] cc: error: linker command failed with exit code 1 (use -v to see invocation) *** [init_test.full] Error code 1 make[7]: stopped in /b/main/src/lib/csu/tests/dynamic While it stops in different places (I've seen that with race conditions) this is common to the builds regardless of tree: ld: error: /usr/obj/b/main/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to __stpcpy [--no-allow-shlib-undefined] I can test on an RE VM snapshot if desired. These builds have been on 13.1R for consistency.
I made the following change but it didn't fix the issue diff --git a/lib/libc/string/Makefile.inc b/lib/libc/string/Makefile.inc index afc113eeb867..6ae567a7a31a 100644 --- a/lib/libc/string/Makefile.inc +++ b/lib/libc/string/Makefile.inc @@ -1,7 +1,10 @@ # @(#)Makefile.inc 8.1 (Berkeley) 6/4/93 # $FreeBSD$ -.PATH: ${LIBC_SRCTOP}/${LIBC_ARCH}/string ${LIBC_SRCTOP}/string +.if ${MK_MACHDEP_OPTIMIZATIONS} == no +.PATH: ${LIBC_SRCTOP}/${LIBC_ARCH}/string +.endif +.PATH: ${LIBC_SRCTOP}/string .PATH: ${SRCTOP}/sys/libkern CFLAGS+= -I${LIBC_SRCTOP}/locale I even verified that the build is using generic version of strcpy.c from /usr/src/lib/libc/string/ At the same time, the __stpcpy symbol is only used in amd64/string/strcpy.c file. I have no idea what's causing this error.
The problem seems to be gone now. make -C lib/libc works fine for me on 14-CURRENT. Michael, can you verify?
(In reply to Gleb Popov from comment #7) Do you have a link to a commit? Using the 2022-11-10 Release Engineering snapshot with included sources: uname -a FreeBSD current 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n259127-689a9368eb6: Thu Nov 10 10:09:36 UTC 2022 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 # cat /etc/src.conf WITHOUT_MACHDEP_OPTIMIZATIONS=YES # pwd /usr/src # make -j8 buildworld ... --- all_subdir_cddl --- --- all_subdir_cddl/lib/libtpool --- ===> cddl/lib/libtpool (all) --- all_subdir_bin --- --- all_subdir_bin/cat --- ld: error: /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to __stpcpy [--no-allow-shlib-undefined] --- all_subdir_kerberos5 --- --- kadm5_pwcheck.3.gz --- gzip -cn /usr/src/crypto/heimdal/lib/kadm5/kadm5_pwcheck.3 > kadm5_pwcheck.3.gz --- all_subdir_bin --- cc: error: linker command failed with exit code 1 (use -v to see invocation) *** [cat.full] Error code 1 I am also testing MAIN as of "Thu Nov 10 16:39:47 2022 +0100" on 13.1R. For me the problem still exists. What is your host and source combination?
% git log -1 commit 443e6eccbd8f46ee60ef48db85d4bb909529efac (HEAD -> main, freebsd/main, freebsd/HEAD)
# uname -a FreeBSD freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n259385-fd5882e1f2ca: Wed Nov 23 07:55:32 UTC 2022 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 # cat /etc/src.conf WITHOUT_MACHDEP_OPTIMIZATIONS=YES # sysctl hw.ncpu hw.ncpu: 8 # make -j8 buildworld -C /usr/src > 2022-11-23-WITHOUT_MACHDEP_OPTIMIZATIONS.log ===> kerberos5/lib/libkrb5 (all) --- all_subdir_bin --- --- all_subdir_bin/cat --- ld: error: /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to __stpcpy [--no-allow-shlib-undefined] cc: error: linker command failed with exit code 1 (use -v to see invocation) 0.75 real 0.33 user 0.08 sys Other suggestions?
I can reproduce it now with `make buildworld`, but `make -C lib/libc -DNO_CLEAN` right after that results in different error. ld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC >>> defined in /usr/obj/usr/src/amd64.amd64/lib/libc/libc.a(__getosreldate.o) >>> referenced by __getosreldate.c:58 (/usr/src/lib/libc/gen/__getosreldate.c:58) >>> __getosreldate.o:(__getosreldate) in archive /usr/obj/usr/src/amd64.amd64/lib/libc/libc.a ld: error: cannot preempt symbol: _DYNAMIC >>> referenced by auxv.c:64 (/usr/src/lib/libc/gen/auxv.c:64) >>> auxv.o:(__init_elf_aux_vector) in archive /usr/obj/usr/src/amd64.amd64/lib/libc/libc.a ld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC >>> defined in /usr/obj/usr/src/amd64.amd64/lib/libc/libc.a(auxv.o) >>> referenced by auxv.c:66 (/usr/src/lib/libc/gen/auxv.c:66) >>> auxv.o:(__init_elf_aux_vector) in archive /usr/obj/usr/src/amd64.amd64/lib/libc/libc.a Michael, can you please help me to find a way to reproduce the issue without having to do buildworld?
Gleb, Unfortunately, my only only use case is a buildworld and I am not qualified to comment on building libc. Is this build option no longer intended to work independently? If that is the case, an early abort would be greatly appreciated. I prepared a build option regression test for Li-Wen while at EuroBSDcon but this is blocking it, unless it can no longer be used independently by design and that simply needs to be documented.
I honestly tried making another stab on this problem, but can't figure anything. The problem appears only when doing make buildworld, but simply re-making the failed target makes it go away, despite the same command line being used. Doing a buildworld after each change to see if it worked is unrealistic. I need help from somebody who understands FreeBSD build system to narrow down exact make invocations that reproduce the problem.
The following patch seems to fix the problem for me. Michael, can you please verify? diff --git a/lib/libc/string/Makefile.inc b/lib/libc/string/Makefile.inc index afc113eeb867..3ecc3bc5b334 100644 --- a/lib/libc/string/Makefile.inc +++ b/lib/libc/string/Makefile.inc @@ -1,7 +1,10 @@ # @(#)Makefile.inc 8.1 (Berkeley) 6/4/93 # $FreeBSD$ -.PATH: ${LIBC_SRCTOP}/${LIBC_ARCH}/string ${LIBC_SRCTOP}/string +.if ${MK_MACHDEP_OPTIMIZATIONS} != "no" +.PATH: ${LIBC_SRCTOP}/${LIBC_ARCH}/string +.endif +.PATH: ${LIBC_SRCTOP}/string .PATH: ${SRCTOP}/sys/libkern CFLAGS+= -I${LIBC_SRCTOP}/locale @@ -39,8 +42,10 @@ CFLAGS.memchr.c+= -fno-sanitize=address .endif +.if ${MK_MACHDEP_OPTIMIZATIONS} != "no" # machine-dependent string sources .sinclude "${LIBC_SRCTOP}/${LIBC_ARCH}/string/Makefile.inc" +.endif MAN+= bcmp.3 bcopy.3 bstring.3 bzero.3 ffs.3 index.3 memccpy.3 memchr.3 \ memcmp.3 memcpy.3 memmem.3 memmove.3 memset.3 strcasecmp.3 strcat.3 \
(In reply to Gleb Popov from comment #14) The patch looks reasonable to me
(In reply to Gleb Popov from comment #14) ОЧЕНЬ ХОРОШО! # git branch * main # git log | head commit 11b5b9e8a520e6ebc5c10c808566e1d08d801990 Author: Søren Schmidt <sos@FreeBSD.org> Date: Sat Jan 7 07:42:44 2023 +0000 # cat /etc/src.conf WITHOUT_MACHDEP_OPTIMIZATIONS=YES I manually added the changes and the buildworld completed successfully! >>> World build completed on Mon Jan 9 20:21:42 PST 2023 >>> World built in 10472 seconds, ncpu: 4, make -j4 This was on 13.1R but I concur with Ed, the syntax looks good. Great work! I will test it again the moment it hits the tree.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=016e46fd869ebf9891ca4b2cf1d22b337717a8c8 commit 016e46fd869ebf9891ca4b2cf1d22b337717a8c8 Author: Gleb Popov <arrowd@FreeBSD.org> AuthorDate: 2023-01-10 10:17:01 +0000 Commit: Gleb Popov <arrowd@FreeBSD.org> CommitDate: 2023-01-10 14:00:41 +0000 libc: Fix build with WITHOUT_MACHDEP_OPTIMIZATIONS=YES set. Test Plan: `make buildword WITHOUT_MACHDEP_OPTIMIZATIONS=YES` on 14-CURRENT and 13-STABLE Reviewed by: emaste Differential Revision: https://reviews.freebsd.org/D38017 PR: 266900 lib/libc/string/Makefile.inc | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
I am testing this for compatibility with other build options used simultaneously and let you know what I find.
For example: (patched) CURRENT on CURRENT builds fine, patched CURRENT on 13.1R threw an error. I will try it from the tree now that it's in.
Update: My standard test is passing, building CURRENT on 13.1: https://callfortesting.org/results/bos-ci/ This test enables all WITHOUT_ options and steps through the requested branches. I look forward to the MFC's! I have provided Li-Wen a variation on this test for inclusion in the official CI system. It's quick to run as it builds a minimum system.
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b83b87f53e73f4d29e094eb89141b251c3272ddd commit b83b87f53e73f4d29e094eb89141b251c3272ddd Author: Gleb Popov <arrowd@FreeBSD.org> AuthorDate: 2023-01-10 10:17:01 +0000 Commit: Gleb Popov <arrowd@FreeBSD.org> CommitDate: 2023-01-12 19:49:59 +0000 libc: Fix build with WITHOUT_MACHDEP_OPTIMIZATIONS=YES set. Test Plan: `make buildword WITHOUT_MACHDEP_OPTIMIZATIONS=YES` on 14-CURRENT and 13-STABLE Reviewed by: emaste Differential Revision: https://reviews.freebsd.org/D38017 PR: 266900 (cherry picked from commit 016e46fd869ebf9891ca4b2cf1d22b337717a8c8) lib/libc/string/Makefile.inc | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
Stable is passing! Thanks! https://callfortesting.org/results/bos-ci/