Bug 266900

Summary: WITHOUT_MACHDEP_OPTIMIZATIONS build option fails on error: undefined symbol: __stpcpy
Product: Base System Reporter: Michael Dexter <editor>
Component: binAssignee: Gleb Popov <arrowd>
Status: Closed FIXED    
Severity: Affects Some People CC: delphij, emaste, imp, kevans
Priority: ---    
Version: CURRENT   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
src.conf that will reproduce the issue
none
Build output with the resulting error none

Description Michael Dexter 2022-10-08 07:39:21 UTC 
Created attachment 237151 [details]
src.conf that will reproduce the issue

Context: FreeBSD buildworld will succeed with all WITHOUT_* build options enable, minus WITHOUT_AUTO_OBJ WITHOUT_UNIFIED_OBJDIR WITHOUT_INSTALLLIB. This provides a far more efficient exercise of the the available build options than the /usr/src/tools/tools/build_option_survey

This commit and review causes the WITHOUT_MACHDEP_OPTIMIZATION build option to fail on "error: undefined symbol: __stpcpy":

https://reviews.freebsd.org/D36463

https://cgit.freebsd.org/src/commit/?id=c65e42dbde4198ce46aef7ddac061390abe672dc

Xin LI is aware of this issue and has kindly offered this clarification and requested this ticket:

"The build broke because some MD files were used, and some were not.  For strcpy()'s case, on amd64, it's a wrapper around the MD stpcpy() and naturally it calls __stpcpy() which itself gets (correctly) excluded.

The MD C source was used because this line which is common for Makefile.inc's in the subdirectories in libc:

.PATH: ${LIBC_SRCTOP}/${LIBC_ARCH}/string ${LIBC_SRCTOP}/string

which basically means, if you have foo.c in the MD directory, it's used in favor of the MI version.

So these need to be fixed for ${MK_MACHDEP_OPTIMIZATIONS} == no case too.

I think this could have been avoided if the initial testing was done with two clean buildworld builds with and without the new option; for build infrastructure changes like this, I'd recommend doing full universe runs locally or on universe14{a,b} at cluster in the future before pushing."

Thank you Xin for looking into this!

Attachments: a src.conf that reproduces the issue on main and stable/13
Comment 1 Michael Dexter 2022-10-08 07:44:48 UTC 
Created attachment 237152 [details]
Build output with the resulting error
Comment 2 Gleb Popov freebsd_committer freebsd_triage 2022-10-10 07:01:13 UTC 
Building with WITHOUT_MACHDEP_OPTIMIZATION=yes only works fine.
Building with your src.conf results in

ld: error: /usr/obj/usr/src/amd64.amd64/tmp/usr/lib/libgcc_s.so: undefined reference to logAPIs [--no-allow-shlib-undefined]
ld: error: /usr/obj/usr/src/amd64.amd64/tmp/usr/lib/libgcc_s.so: undefined reference to logUnwinding [--no-allow-shlib-undefined]
ld: error: /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to yp_next [--no-allow-shlib-undefined]
ld: error: /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to yp_match [--no-allow-shlib-undefined]
ld: error: /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to yp_first [--no-allow-shlib-undefined]

How do I reproduce the problem?
Comment 3 Michael Dexter 2022-10-14 02:22:16 UTC 
(In reply to Gleb Popov from comment #2)
What branch are you building?

I see this on main and stable/13:

git -C /usr/src branch
* stable/13

cat /etc/src.conf
WITHOUT_MACHDEP_OPTIMIZATIONS=YES

git -C /usr/src log|head
commit e70ce8d3ae1ba3d73e8efe833c941cda6a3bcf5a
Author: Konstantin Belousov <kib@FreeBSD.org>
Date:   Sat Oct 8 11:33:11 2022 +0300
...

cd /usr/src && make -j4 buildworld > ../output.log

tail ../output.log

...
--- init_test.full ---
cc -target x86_64-unknown-freebsd13.1 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -O2 -pipe -fno-common -I/usr/src/lib/csu/amd64 -g -std=gnu99 -Wno-format-zero-length -fstack-protector-strong -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-error=unused-but-set-variable -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef -Wno-address-of-packed-member -Qunused-arguments   -o init_test.full init_test.o  -lprivateatf-c
ld: error: /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to __stpcpy [--no-allow-shlib-undefined]
cc: error: linker command failed with exit code 1 (use -v to see invocation)
--- all_subdir_cddl ---

make[2]: stopped in /usr/src
--- all_subdir_bin ---
...

I will pull main and build again.
Comment 4 Gleb Popov freebsd_committer freebsd_triage 2022-10-14 06:45:19 UTC 
(In reply to Michael Dexter from comment #3)
I'm building 14-CURRENT.

Should I use src.conf you provided or just a single WITHOUT_MACHDEP_OPTIMIZATION=YES line?
Comment 5 Michael Dexter 2022-10-15 04:17:01 UTC 
(In reply to Gleb Popov from comment #4)
Start with just the one entry to src.conf but I will not rule out there being more than one issue or a race condition.

I updated my stable/13 and main branches and built again with only the one entry:

# cat /etc/src.conf
WITHOUT_MACHDEP_OPTIMIZATIONS=YES

* stable/13
commit 67d863df0833cdb1ca054a7c4e7d551b4ad28d95
Author: Jose Luis Duran <jlduran@gmail.com>
Date:   Tue Oct 11 12:39:26 2022 -0400

le -Wno-error=unused-but-set-variable -Qunused-arguments  -pie   -o cat.full cat.o
ld: error: /usr/obj/b/stable/13/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to __stpcpy [--no-allow-shlib-undefined]
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** [cat.full] Error code 1

make[4]: stopped in /b/stable/13/src/bin/cat


* main
commit 2e0e27392732a811b52c002a707ac848bad01b30
Author: Gleb Smirnoff <glebius@FreeBSD.org>
Date:   Thu Oct 13 09:03:38 2022 -0700

--- all_subdir_lib ---
ld: error: /usr/obj/b/main/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to __stpcpy [--no-allow-shlib-undefined]
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** [init_test.full] Error code 1

make[7]: stopped in /b/main/src/lib/csu/tests/dynamic


While it stops in different places (I've seen that with race conditions) this is common to the builds regardless of tree:

ld: error: /usr/obj/b/main/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to __stpcpy [--no-allow-shlib-undefined]

I can test on an RE VM snapshot if desired. These builds have been on 13.1R for consistency.
Comment 6 Gleb Popov freebsd_committer freebsd_triage 2022-10-17 13:08:08 UTC 
I made the following change but it didn't fix the issue

diff --git a/lib/libc/string/Makefile.inc b/lib/libc/string/Makefile.inc
index afc113eeb867..6ae567a7a31a 100644
--- a/lib/libc/string/Makefile.inc
+++ b/lib/libc/string/Makefile.inc
@@ -1,7 +1,10 @@
 #      @(#)Makefile.inc        8.1 (Berkeley) 6/4/93
 # $FreeBSD$
 
-.PATH: ${LIBC_SRCTOP}/${LIBC_ARCH}/string ${LIBC_SRCTOP}/string
+.if ${MK_MACHDEP_OPTIMIZATIONS} == no
+.PATH: ${LIBC_SRCTOP}/${LIBC_ARCH}/string
+.endif
+.PATH: ${LIBC_SRCTOP}/string
 .PATH: ${SRCTOP}/sys/libkern
 
 CFLAGS+= -I${LIBC_SRCTOP}/locale

I even verified that the build is using generic version of strcpy.c from /usr/src/lib/libc/string/

At the same time, the __stpcpy symbol is only used in amd64/string/strcpy.c file. I have no idea what's causing this error.
Comment 7 Gleb Popov freebsd_committer freebsd_triage 2022-11-08 11:41:50 UTC 
The problem seems to be gone now. make -C lib/libc works fine for me on 14-CURRENT.

Michael, can you verify?
Comment 8 Michael Dexter 2022-11-11 09:54:19 UTC 

(In reply to Gleb Popov from comment #7)

Do you have a link to a commit?

Using the 2022-11-10 Release Engineering snapshot with included sources:

uname -a
FreeBSD current 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n259127-689a9368eb6: Thu Nov 10 10:09:36 UTC 2022     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

# cat /etc/src.conf
WITHOUT_MACHDEP_OPTIMIZATIONS=YES

# pwd
/usr/src

# make -j8 buildworld
...

--- all_subdir_cddl ---
--- all_subdir_cddl/lib/libtpool ---
===> cddl/lib/libtpool (all)
--- all_subdir_bin ---
--- all_subdir_bin/cat ---
ld: error: /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to __stpcpy [--no-allow-shlib-undefined]
--- all_subdir_kerberos5 ---
--- kadm5_pwcheck.3.gz ---
gzip -cn /usr/src/crypto/heimdal/lib/kadm5/kadm5_pwcheck.3 > kadm5_pwcheck.3.gz
--- all_subdir_bin ---
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** [cat.full] Error code 1

I am also testing MAIN as of "Thu Nov 10 16:39:47 2022 +0100" on 13.1R.

For me the problem still exists. What is your host and source combination?
Comment 9 Gleb Popov freebsd_committer freebsd_triage 2022-11-11 10:00:05 UTC 
% git log -1
commit 443e6eccbd8f46ee60ef48db85d4bb909529efac (HEAD -> main, freebsd/main, freebsd/HEAD)
Comment 10 Michael Dexter 2022-11-28 22:33:08 UTC 
# uname -a
FreeBSD freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n259385-fd5882e1f2ca: Wed Nov 23 07:55:32 UTC 2022     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

# cat /etc/src.conf
WITHOUT_MACHDEP_OPTIMIZATIONS=YES

# sysctl hw.ncpu
hw.ncpu: 8

# make -j8 buildworld -C /usr/src > 2022-11-23-WITHOUT_MACHDEP_OPTIMIZATIONS.log

===> kerberos5/lib/libkrb5 (all)
--- all_subdir_bin ---
--- all_subdir_bin/cat ---
ld: error: /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7: undefined reference to __stpcpy [--no-allow-shlib-undefined]
cc: error: linker command failed with exit code 1 (use -v to see invocation)
        0.75 real         0.33 user         0.08 sys

Other suggestions?
Comment 11 Gleb Popov freebsd_committer freebsd_triage 2022-11-29 11:54:00 UTC 
I can reproduce it now with `make buildworld`, but `make -C lib/libc -DNO_CLEAN` right after that results in different error.


ld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /usr/obj/usr/src/amd64.amd64/lib/libc/libc.a(__getosreldate.o)
>>> referenced by __getosreldate.c:58 (/usr/src/lib/libc/gen/__getosreldate.c:58)
>>>               __getosreldate.o:(__getosreldate) in archive /usr/obj/usr/src/amd64.amd64/lib/libc/libc.a

ld: error: cannot preempt symbol: _DYNAMIC
>>> referenced by auxv.c:64 (/usr/src/lib/libc/gen/auxv.c:64)
>>>               auxv.o:(__init_elf_aux_vector) in archive /usr/obj/usr/src/amd64.amd64/lib/libc/libc.a

ld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /usr/obj/usr/src/amd64.amd64/lib/libc/libc.a(auxv.o)
>>> referenced by auxv.c:66 (/usr/src/lib/libc/gen/auxv.c:66)
>>>               auxv.o:(__init_elf_aux_vector) in archive /usr/obj/usr/src/amd64.amd64/lib/libc/libc.a



Michael, can you please help me to find a way to reproduce the issue without having to do buildworld?
Comment 12 Michael Dexter 2023-01-05 19:39:24 UTC 
Gleb,

Unfortunately, my only only use case is a buildworld and I am not qualified to comment on building libc.

Is this build option no longer intended to work independently?

If that is the case, an early abort would be greatly appreciated.

I prepared a build option regression test for Li-Wen while at EuroBSDcon but this is blocking it, unless it can no longer be used independently by design and that simply needs to be documented.
Comment 13 Gleb Popov freebsd_committer freebsd_triage 2023-01-06 21:22:47 UTC 
I honestly tried making another stab on this problem, but can't figure anything.

The problem appears only when doing make buildworld, but simply re-making the failed target makes it go away, despite the same command line being used.

Doing a buildworld after each change to see if it worked is unrealistic. I need help from somebody who understands FreeBSD build system to narrow down exact make invocations that reproduce the problem.
Comment 14 Gleb Popov freebsd_committer freebsd_triage 2023-01-08 14:41:40 UTC 
The following patch seems to fix the problem for me. Michael, can you please verify?


diff --git a/lib/libc/string/Makefile.inc b/lib/libc/string/Makefile.inc
index afc113eeb867..3ecc3bc5b334 100644
--- a/lib/libc/string/Makefile.inc
+++ b/lib/libc/string/Makefile.inc
@@ -1,7 +1,10 @@
 #      @(#)Makefile.inc        8.1 (Berkeley) 6/4/93
 # $FreeBSD$
 
-.PATH: ${LIBC_SRCTOP}/${LIBC_ARCH}/string ${LIBC_SRCTOP}/string
+.if ${MK_MACHDEP_OPTIMIZATIONS} != "no"
+.PATH: ${LIBC_SRCTOP}/${LIBC_ARCH}/string
+.endif
+.PATH: ${LIBC_SRCTOP}/string
 .PATH: ${SRCTOP}/sys/libkern
 
 CFLAGS+= -I${LIBC_SRCTOP}/locale
@@ -39,8 +42,10 @@ CFLAGS.memchr.c+=            -fno-sanitize=address
 .endif
 
 
+.if ${MK_MACHDEP_OPTIMIZATIONS} != "no"
 # machine-dependent string sources
 .sinclude "${LIBC_SRCTOP}/${LIBC_ARCH}/string/Makefile.inc"
+.endif
 
 MAN+=  bcmp.3 bcopy.3 bstring.3 bzero.3 ffs.3 index.3 memccpy.3 memchr.3 \
        memcmp.3 memcpy.3 memmem.3 memmove.3 memset.3 strcasecmp.3 strcat.3 \
Comment 15 Ed Maste freebsd_committer freebsd_triage 2023-01-09 18:57:46 UTC 
(In reply to Gleb Popov from comment #14)
The patch looks reasonable to me
Comment 16 Michael Dexter 2023-01-10 04:46:20 UTC 
(In reply to Gleb Popov from comment #14)
ОЧЕНЬ ХОРОШО!

# git branch
* main

# git log | head
commit 11b5b9e8a520e6ebc5c10c808566e1d08d801990
Author: Søren Schmidt <sos@FreeBSD.org>
Date:   Sat Jan 7 07:42:44 2023 +0000

# cat /etc/src.conf
WITHOUT_MACHDEP_OPTIMIZATIONS=YES

I manually added the changes and the buildworld completed successfully!

>>> World build completed on Mon Jan  9 20:21:42 PST 2023
>>> World built in 10472 seconds, ncpu: 4, make -j4

This was on 13.1R but I concur with Ed, the syntax looks good.

Great work! I will test it again the moment it hits the tree.
Comment 17 commit-hook freebsd_committer freebsd_triage 2023-01-10 14:02:06 UTC 
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=016e46fd869ebf9891ca4b2cf1d22b337717a8c8

commit 016e46fd869ebf9891ca4b2cf1d22b337717a8c8
Author:     Gleb Popov <arrowd@FreeBSD.org>
AuthorDate: 2023-01-10 10:17:01 +0000
Commit:     Gleb Popov <arrowd@FreeBSD.org>
CommitDate: 2023-01-10 14:00:41 +0000

    libc: Fix build with WITHOUT_MACHDEP_OPTIMIZATIONS=YES set.

    Test Plan: `make buildword WITHOUT_MACHDEP_OPTIMIZATIONS=YES` on 14-CURRENT and 13-STABLE

    Reviewed by: emaste

    Differential Revision: https://reviews.freebsd.org/D38017

    PR:             266900

 lib/libc/string/Makefile.inc | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
Comment 18 Michael Dexter 2023-01-10 20:27:50 UTC 
I am testing this for compatibility with other build options used simultaneously and let you know what I find.
Comment 19 Michael Dexter 2023-01-10 23:36:11 UTC 
For example: (patched) CURRENT on CURRENT builds fine, patched CURRENT on 13.1R threw an error. I will try it from the tree now that it's in.
Comment 20 Michael Dexter 2023-01-11 00:31:17 UTC 
Update: My standard test is passing, building CURRENT on 13.1:

https://callfortesting.org/results/bos-ci/

This test enables all WITHOUT_ options and steps through the requested branches.

I look forward to the MFC's!

I have provided Li-Wen a variation on this test for inclusion in the official CI system. It's quick to run as it builds a minimum system.
Comment 21 commit-hook freebsd_committer freebsd_triage 2023-01-12 19:51:13 UTC 
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=b83b87f53e73f4d29e094eb89141b251c3272ddd

commit b83b87f53e73f4d29e094eb89141b251c3272ddd
Author:     Gleb Popov <arrowd@FreeBSD.org>
AuthorDate: 2023-01-10 10:17:01 +0000
Commit:     Gleb Popov <arrowd@FreeBSD.org>
CommitDate: 2023-01-12 19:49:59 +0000

    libc: Fix build with WITHOUT_MACHDEP_OPTIMIZATIONS=YES set.

    Test Plan: `make buildword WITHOUT_MACHDEP_OPTIMIZATIONS=YES` on 14-CURRENT and 13-STABLE

    Reviewed by: emaste

    Differential Revision: https://reviews.freebsd.org/D38017

    PR:             266900
    (cherry picked from commit 016e46fd869ebf9891ca4b2cf1d22b337717a8c8)

 lib/libc/string/Makefile.inc | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
Comment 22 Michael Dexter 2023-01-12 21:05:15 UTC 
Stable is passing! Thanks!

https://callfortesting.org/results/bos-ci/