Bug 206123 - devel/powerpc64-gcc based buildworld installed (or other modern gcc's): powerpc64 lib32 use fails because of lib/csu/powerpc/crti.S mismatch with how (e.g.) devel/powerpc64-gcc/work/gcc-6.3.0/libgcc/crtstuff.c works
Summary: devel/powerpc64-gcc based buildworld installed (or other modern gcc's): power...
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: powerpc Any
: --- Affects Only Me
Assignee: freebsd-toolchain (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-01-11 03:25 UTC by Mark Millard
Modified: 2020-06-16 22:58 UTC (History)
2 users (show)

See Also:
bugzilla: maintainer-feedback? (bapt)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mark Millard 2016-01-11 03:25:08 UTC
powerpc64 context: devel/powerpc64-gcc used for buildworld produces a -m32 -mcpu=powerpc crtbeginS.o (WITH_LIB32= for powerpc64 or for TARGET_ARCH=powerpc) that is not suitable. This leads to _init in /usr/lib32/libc.so.7 crashing, preventing lib32's use.

(If I avoid use of lib32 I can use powerpc64-gcc to build and install kernel and world, including WITH_BOOT= . WITH_LIB32= does build and install --it just does not work.)

The lib32/crtbeginS.o ends up with .init code that looks like:

# /usr/local/powerpc64-freebsd/bin/objdump -d /usr/lib32/crtbeginS.o | tail -20
 178:	7d 61 5b 78 	mr      r1,r11
 17c:	4e 80 00 20 	blr

Disassembly of section .fini:

00000000 <.fini>:
   0:	48 00 00 01 	bl      0 <.fini>

Disassembly of section .init:

00000000 <.init>:
   0:	80 7e 00 00 	lwz     r3,0(r30)
   4:	81 23 00 00 	lwz     r9,0(r3)
   8:	2f 89 00 00 	cmpwi   cr7,r9,0
   c:	41 9e 00 18 	beq     cr7,24 <__do_global_dtors_aux+0x24>
  10:	81 3e 00 00 	lwz     r9,0(r30)
  14:	2f 89 00 00 	cmpwi   cr7,r9,0
  18:	41 9e 00 0c 	beq     cr7,24 <__do_global_dtors_aux+0x24>
  1c:	7d 29 03 a6 	mtctr   r9
  20:	4e 80 04 21 	bctrl

which depends on r30's preexisting value (and the offset ends up adjusted from zero when put to use in /usr/lib32/libc.so.7).

For comparison/contrast: From a buildworld that uses gcc4.2.1's toolchain (src.conf empty):

# /usr/local/powerpc64-freebsd/bin/objdump -d /usr/obj/lib32/usr/src/gnu/lib/csu/crtbeginS.o | tail

Disassembly of section .fini:

00000000 <.fini>:
   0:	48 00 00 01 	bl      0 <.fini>

Disassembly of section .init:

00000000 <.init>:
   0:	48 00 00 01 	bl      0 <.init>


Back to how the powerpc64-gcc's produced crtbeginS.o behaves in use:

For compiling with -m32 via powerpc64-gcc lib32/libc.so.7 dies in its _init:

# more main.c
int main(void)
{
    return 0;
}
# /usr/local/bin/powerpc64-portbld-freebsd11.0-gcc --version
powerpc64-portbld-freebsd11.0-gcc (FreeBSD Ports Collection for powerpc64) 5.2.0
. . .
# /usr/local/bin/powerpc64-portbld-freebsd11.0-gcc -m32 -mcpu=powerpc main.c
# file a.out
a.out: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, FreeBSD-style, for FreeBSD 11.0 (1100093), not stripped
# /usr/local/bin/gdb a.out
GNU gdb (GDB) 7.10 [GDB v7.10 for FreeBSD]
. . .
Reading symbols from a.out...(no debugging symbols found)...done.
(gdb) run
Starting program: /root/c_tests/a.out 
warning: `/libexec/ld-elf.so.1': Shared library architecture powerpc:common64 is not compatible with target architecture powerpc:common.
warning: `/libexec/ld-elf.so.1': Shared library architecture powerpc:common64 is not compatible with target architecture powerpc:common.

Program received signal SIGSEGV, Segmentation fault.
0x41867ebc in _init () from /usr/lib32/libc.so.7
(gdb) x/20i 0x41867ea0
   0x41867ea0:	.long 0xc3b15
   0x41867ea4:	.long 0x0
   0x41867ea8 <_init>:	stwu    r1,-16(r1)
   0x41867eac <_init+4>:	mflr    r0
   0x41867eb0 <_init+8>:	stw     r31,12(r1)
   0x41867eb4 <_init+12>:	stw     r0,20(r1)
   0x41867eb8 <_init+16>:	mr      r31,r1
=> 0x41867ebc <_init+20>:	lwz     r3,-11432(r30)
   0x41867ec0 <_init+24>:	lwz     r9,0(r3)
   0x41867ec4 <_init+28>:	cmpwi   cr7,r9,0
   0x41867ec8 <_init+32>:	beq     cr7,0x41867ee0 <_init+56>
   0x41867ecc <_init+36>:	lwz     r9,-856(r30)
   0x41867ed0 <_init+40>:	cmpwi   cr7,r9,0
   0x41867ed4 <_init+44>:	beq     cr7,0x41867ee0 <_init+56>
   0x41867ed8 <_init+48>:	mtctr   r9
   0x41867edc <_init+52>:	bctrl
   0x41867ee0 <_init+56>:	lwz     r29,-1140(r30)
   0x41867ee4 <_init+60>:	lwzu    r9,-4(r29)
   0x41867ee8 <_init+64>:	cmpwi   cr7,r9,-1
   0x41867eec <_init+68>:	beq     cr7,0x41867f04 <_init+92>
(gdb) info registers
. . .
r30            0x4183bb8c	1099152268
r31            0xffffd730	4294956848
pc             0x41867ebc	0x41867ebc <_init+20>
msr            <unavailable>
cr             0x28000482	671089794
lr             0x41814d9c	0x41814d9c <objlist_call_init+296>
. . .
(gdb) x/150i objlist_call_init
   0x41814c74 <objlist_call_init>:	stwu    r1,-64(r1)
   0x41814c78 <objlist_call_init+4>:	mflr    r0
   0x41814c7c <objlist_call_init+8>:	bl      0x4183bb88
   0x41814c80 <objlist_call_init+12>:	li      r8,0
   0x41814c84 <objlist_call_init+16>:	stw     r30,56(r1)
   0x41814c88 <objlist_call_init+20>:	mflr    r30
. . .
   0x41814d88 <objlist_call_init+276>:	bl      0x418131b0 <ld_utrace_log>
   0x41814d8c <objlist_call_init+280>:	lwz     r9,4(r28)
   0x41814d90 <objlist_call_init+284>:	lwz     r5,256(r9)
   0x41814d94 <objlist_call_init+288>:	mtctr   r5
   0x41814d98 <objlist_call_init+292>:	bctrl
   0x41814d9c <objlist_call_init+296>:	lwz     r10,4(r28)
   0x41814da0 <objlist_call_init+300>:	lwz     r29,268(r10)
. . .
(gdb) info registers
. . .
r30            0x4183bb8c	1099152268
. . .
(gdb) x/20i 0x4183bb8c - 0x10
   0x4183bb7c:	bso     0x4183778c
   0x4183bb80:	bso     0x41837788
   0x4183bb84:	bso     0x4183773c
   0x4183bb88:	blrl
   0x4183bb8c <_SDA_BASE_>:	.long 0x2b428
   0x4183bb90 <_SDA_BASE_+4>:	.long 0x0
   0x4183bb94 <_SDA_BASE_+8>:	.long 0x0
   0x4183bb98 <_thread_autoinit_dummy_decl>:	.long 0x1
   0x4183bb9c <ld_env_prefix>:	beq     0x41835b80
   0x4183bba0 <ld_standard_library_path>:	beq     0x41835b8c
   0x4183bba4 <ld_path_rtld>:	beq     0x41835ba4
   0x4183bba8 <ld_path_libmap_conf>:	beq     0x41835bc0
   0x4183bbac <ld_elf_hints_default>:	beq     0x41835bd8
   0x4183bbb0 <tls_max_index>:	.long 0x2
   0x4183bbb4 <tls_dtv_generation>:	.long 0x2
   0x4183bbb8 <rtld_phdr_lock>:	bso     0x4183822c
   0x4183bbbc <rtld_libc_lock>:	bso     0x41838228
   0x4183bbc0 <rtld_bind_lock>:	bso     0x41838224
   0x4183bbc4 <realloc_srchlen>:	.long 0x4
   0x4183bbc8 <lmp_head>:	beq     0x41838be8
(gdb) x/i 0x4183bb8c - 11432
   0x41838ee4:	Cannot access memory at address 0x41838ee4

So the "lwz r3,-11432(r30)" ends up referencing <_SDA_BASE_-11432>, which is not accessible.


So far I have not managed to track down why the r30 based .init code shows up --or even what call standard(s) use r30 in a way that would appear to match. But the code's behavior seems to be just wrong for the FreeBSD powerpc context.
Comment 1 Mark Millard 2016-01-11 03:47:55 UTC
I should have explicitly said that the buildworld/buildkernel used powerpc64-gcc to do the build and included the src.conf. I should have shown the FreeBSD version for the latest example as well:

# freebsd-version -ku; uname -aKU
11.0-CURRENT
11.0-CURRENT
FreeBSD FBSDG5C0 11.0-CURRENT FreeBSD 11.0-CURRENT #9 r293632M: Sun Jan 10 12:47:56 PST 2016     markmi@FBSDG5C0:/usr/obj/xtoolchain/powerpc.powerpc64/usr/src/sys/GENERIC64vtsc-NODEBUG  powerpc 1100093 1100093

make.conf empty.

src.conf:

TO_TYPE=powerpc64
TOOLS_TO_TYPE=${TO_TYPE}
FROM_TYPE=powerpc64
TOOLS_FROM_TYPE=${FROM_TYPE}
VERSION_CONTEXT=11.0
#
KERNCONF=GENERIC64vtsc-NODEBUG
TARGET=powerpc
.if ${.MAKE.LEVEL} == 0
TARGET_ARCH=${TO_TYPE}
.export TARGET_ARCH
.endif
#
WITHOUT_CROSS_COMPILER=
#
# 1 thing that fails to build if attempted via gcc variants:
WITHOUT_CLANG_EXTRAS=
#
WITH_FAST_DEPEND=
WITH_LIBCPLUSPLUS=
WITH_LIB32=
WITH_BOOT=
WITH_CLANG=
WITH_CLANG_IS_CC=
WITH_CLANG_FULL=
WITH_LLDB=
#
WITHOUT_GCC=
WITHOUT_GNUCXX=
#
NO_WERROR=
MALLOC_PRODUCTION=
#
WITH_DEBUG=
WITH_DEBUG_FILES=
#
# TOOLS_TO_TYPE based on ${TO_TYPE}-xtoolchain-gcc related bintutils...
#
CROSS_TOOLCHAIN=${TO_TYPE}-gcc
X_COMPILER_TYPE=gcc
CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/
.if ${.MAKE.LEVEL} == 0
XCC=/usr/local/bin/${TOOLS_TO_TYPE}-portbld-freebsd${VERSION_CONTEXT}-gcc
XCXX=/usr/local/bin/${TOOLS_TO_TYPE}-portbld-freebsd${VERSION_CONTEXT}-g++
XCPP=/usr/local/bin/${TOOLS_TO_TYPE}-portbld-freebsd${VERSION_CONTEXT}-cpp
.export XCC
.export XCXX
.export XCPP
XAS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as
XAR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar
XLD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld
XNM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm
XOBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy
XOBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump
XRANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib
XSIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size
#NO-SUCH: XSTRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings
XSTRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings
.export XAS
.export XAR
.export XLD
.export XNM
.export XOBJCOPY
.export XOBJDUMP
.export XRANLIB
.export XSIZE
.export XSTRINGS
.endif
.if ${.MAKE.LEVEL} == 0
CC=/usr/local/bin/gcc49
CXX=/usr/local/bin/g++49
CPP=/usr/local/bin/cpp49
.export CC
.export CXX
.export CPP
.endif
.if ${.MAKE.LEVEL} == 0
AS=/usr/local/${TOOLS_FROM_TYPE}-portbld-freebsd${VERSION_CONTEXT}/bin/as
AR=/usr/local/${TOOLS_FROM_TYPE}-portbld-freebsd${VERSION_CONTEXT}/bin/ar
LD=/usr/local/${TOOLS_FROM_TYPE}-portbld-freebsd${VERSION_CONTEXT}/bin/ld
NM=/usr/local/${TOOLS_FROM_TYPE}-portbld-freebsd${VERSION_CONTEXT}/bin/nm
OBJCOPY=/usr/local/${TOOLS_FROM_TYPE}-portbld-freebsd${VERSION_CONTEXT}/bin/objcopy
OBJDUMP=/usr/local/${TOOLS_FROM_TYPE}-portbld-freebsd${VERSION_CONTEXT}/bin/objdump
RANLIB=/usr/local/${TOOLS_FROM_TYPE}-portbld-freebsd${VERSION_CONTEXT}/bin/ranlib
SIZE=/usr/local/${TOOLS_FROM_TYPE}-portbld-freebsd${VERSION_CONTEXT}/bin/size
#NO-SUCH: STRINGS=/usr/local/${TOOLS_FROM_TYPE}-portbld-freebsd${VERSION_CONTEXT}/bin/strings
STRINGS=/usr/local/bin/strings
.export AS
.export AR
.export LD
.export NM
.export OBJCOPY
.export OBJDUMP
.export RANLIB
.export SIZE
.export STRINGS
.endif
Comment 2 Mark Millard 2017-10-29 00:55:50 UTC
From a list exchange:

On 2017-Oct-28, at 5:03 PM, Justin Hibbits <chmeeedalf at gmail.com> wrote:

On Oct 28, 2017 17:08, "Mark Millard" <markmi at dsl-only.net> wrote:
powerpc64 and powerpc have very different stack handling
rules for FreeBSD. As an example, powerpc does not require
red-zones for signal handling in the kernel but powerpc64
does.

For lib32 support, what ABI is the powerpc code supposed
to follow in the powerpc64 environment? What style of
stack handling (and related register usage) is supposed
to be in use? If it is distinct from powerpc native's
ABI, what documentation should be looked at for the ABI?
END MILLARD

PowerPC via lib32 should be using the 32-bit svr4 ABI. If you see any discrepancy in that, it's a bug that needs fixed.
END HIBBITS

MILLARD again:

Then I expect that the reason that devel/powerpc64-gcc
based buildworld's make a lib32 that fails in code that
is from /usr/lib32/crtbeginS.o is because
/usr/lib32/crtbeginS.o ends up  not having the right
ABI involved and, so, misuses at least one register.

(I've not worked out if there is any special transition from
one ABI to the other (and later back) that is needed vs. not.
But the red-zoning for powerpc64 should be more than sufficient
for powerpc code that does not require it.)

An interesting point is:
(all on a powerpc64 FreeBSD head -r324071 system)

# clang -dumpmachine -m32
powerpc-unknown-freebsd12.0

# /usr/local/bin/powerpc64-unknown-freebsd12.0-gcc -dumpmachine -m32
powerpc64-unknown-freebsd12.0

# gcc7 -dumpmachine -m32
powerpc64-portbld-freebsd12.0

# clang -dumpmachine
powerpc64-unknown-freebsd12.0

# /usr/local/bin/powerpc64-unknown-freebsd12.0-gcc -dumpmachine
powerpc64-unknown-freebsd12.0

# gcc7 -dumpmachine
powerpc64-portbld-freebsd12.0

(But I'm not sure that these always reflect
ABI variations in code generated.)

I wonder if for gcc it takes a separate compiler
to have powerpc-unknown-freebsd12.0 (intending
to imply the FreeBSD 32-bit ABI is in use).


bugzilla 206123 should probably have notes added about the
probable ABI mismatch in /usr/lib32/crtbeginS.o. I may add
this whole message but someone with better background
information may able to submit more specific material.

At this point I do not know how to control what ABI code
is generated by devel/powerpc64-gcc in what becomes
/usr/lib32/crtbeginS.o (or how other code interfaces with
crtbeginS.o code).

Its been a long time since I tried other gcc variations
but all of them had the issue when I did try.
Comment 3 Mark Millard 2017-10-29 01:11:57 UTC
(In reply to Mark Millard from comment #2)

I wrote:

I wonder if for gcc it takes a separate compiler
to have powerpc-unknown-freebsd12.0 (intending
to imply the FreeBSD 32-bit ABI is in use).

END

(It may be that I should have stuck a "modern"
in the wording, given that gcc 4.2.1 supported
building a lib32 that worked.)

If it does take a separate gcc compiler to have
powerpc-unknown-freebsd12.0 with the FreeBSD
32-bit ABI, then in order to support building
lib32 might require a devel/powerpc-gcc port
if a pre-defined way to build a gcc-based lib32
is to be supported via ports mixed with the
FreeBSD build system.

Otherwise buidlworld via devel/powerpc64-gcc
may just want to always use WITHOUT_LIB32= .

But I'm not aware of the FreeBSD build environment
being set up to have a distinct compiler to target
lib32 code.
Comment 4 Mark Millard 2017-10-29 04:47:16 UTC
Based on some amd64 -> powerpc64 and amd64 -> powerpc
buildworld buildkernel directory trees. . .

Note the <.init. .  .> material and its use of r30
when devel/powerpc64-gcc builds lib32's crtbeginS.o :

# /usr/obj/powerpcvtsc_clang_gcc421/powerpc.powerpc/usr/src/tmp/usr/bin/objdump -d --prefix-address /usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/lib32/usr/lib32/crtbeginS.o | grep r30
0000000c <__do_global_dtors_aux+0xc> stw     r30,24(r1)
00000010 <__do_global_dtors_aux+0x10> mflr    r30
0000001c <__do_global_dtors_aux+0x1c> lwz     r28,0(r30)
00000038 <__do_global_dtors_aux+0x38> lwz     r9,0(r30)
00000044 <__do_global_dtors_aux+0x44> lwz     r9,0(r30)
00000050 <__do_global_dtors_aux+0x50> lwz     r29,0(r30)
000000a4 <__do_global_dtors_aux+0xa4> lwz     r30,-8(r11)
000000c4 <call___do_global_dtors_aux+0x14> stw     r30,8(r1)
000000d8 <call___do_global_dtors_aux+0x28> lwz     r30,-8(r11)
000000f0 <frame_dummy+0xc> stw     r30,8(r1)
000000f4 <frame_dummy+0x10> mflr    r30
00000104 <frame_dummy+0x20> lwz     r3,0(r30)
00000114 <frame_dummy+0x30> lwz     r9,0(r30)
00000124 <frame_dummy+0x40> lwz     r30,-8(r11)
0000013c <frame_dummy+0x58> lwz     r30,-8(r11)
00000158 <call_frame_dummy+0xc> stw     r30,8(r1)
0000015c <call_frame_dummy+0x10> mflr    r30
0000017c <call_frame_dummy+0x30> lwz     r30,-8(r11)
00000000 <.init> lwz     r3,0(r30)
00000010 <.init+0x10> lwz     r9,0(r30)

It is the .init code and its use of r30 that is
directly crashing. (There could be more wrong.)

By contrast there is no such <.init. . .> r30 material
when gcc 4.2.1 does the lib32 build (clang built gcc
4.2.1 here):

# /usr/obj/powerpcvtsc_clang_gcc421/powerpc.powerpc/usr/src/tmp/usr/bin/objdump -d --prefix-address /usr/obj/powerpcvtsc_clang_gcc421/powerpc.powerpc/usr/src/tmp/usr/lib/crtbeginS.o | grep r30
0000000c <__do_global_dtors_aux+0xc> stw     r30,24(r1)
00000010 <__do_global_dtors_aux+0x10> mflr    r30
0000001c <__do_global_dtors_aux+0x1c> lwz     r28,0(r30)
00000038 <__do_global_dtors_aux+0x38> lwz     r0,0(r30)
00000044 <__do_global_dtors_aux+0x44> lwz     r9,0(r30)
00000050 <__do_global_dtors_aux+0x50> lwz     r29,0(r30)
000000a0 <__do_global_dtors_aux+0xa0> lwz     r30,-8(r11)
000000e8 <frame_dummy+0xc> stw     r30,8(r1)
000000ec <frame_dummy+0x10> mflr    r30
000000fc <frame_dummy+0x20> lwz     r3,0(r30)
0000010c <frame_dummy+0x30> lwz     r0,0(r30)
00000128 <frame_dummy+0x4c> lwz     r30,-8(r11)

And the .init is very simple:

# /usr/obj/powerpcvtsc_clang_gcc421/powerpc.powerpc/usr/src/tmp/usr/bin/objdump -d --prefix-address /usr/obj/powerpcvtsc_clang_gcc421/powerpc.powerpc/usr/src/tmp/usr/lib/crtbeginS.o | grep init
Disassembly of section .init:
00000000 <.init> bl      00000000 <.init>

compared to the .init that devel/powerpc64-gcc
generated:

# /usr/obj/powerpcvtsc_clang_gcc421/powerpc.powerpc/usr/src/tmp/usr/bin/objdump -d --prefix-address /usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/lib32/usr/lib32/crtbeginS.o | grep init
Disassembly of section .init:
00000000 <.init> lwz     r3,0(r30)
00000004 <.init+0x4> lwz     r9,0(r3)
00000008 <.init+0x8> cmpwi   cr7,r9,0
0000000c <.init+0xc> beq-    cr7,00000024 <__do_global_dtors_aux+0x24>
00000010 <.init+0x10> lwz     r9,0(r30)
00000014 <.init+0x14> cmpwi   cr7,r9,0
00000018 <.init+0x18> beq-    cr7,00000024 <__do_global_dtors_aux+0x24>
0000001c <.init+0x1c> mtctr   r9
00000020 <.init+0x20> bctrl


Compared to the devel/powerpc64-gcc lib32's crtbeginS.o
below all the following have no r30 use in the gcc 4.2.1
output (but the routines do exist for the gcc 4.2.1
output):

000000c4 <call___do_global_dtors_aux+0x14> stw     r30,8(r1)
000000d8 <call___do_global_dtors_aux+0x28> lwz     r30,-8(r11)
00000158 <call_frame_dummy+0xc> stw     r30,8(r1)
0000015c <call_frame_dummy+0x10> mflr    r30
0000017c <call_frame_dummy+0x30> lwz     r30,-8(r11)
00000000 <.init> lwz     r3,0(r30)
00000010 <.init+0x10> lwz     r9,0(r30)
Comment 5 Mark Millard 2017-10-29 13:56:24 UTC
I've figured out the mismatch involved and, so,
why/how lib32 fails for devel/powerpc64-gcc based
builds: the .init code generation for
devel/powerpc64-gcc is tied to the glibc crti.S
for powerpc and that does not match what FreeBSD
has for the interface between the two parts.

As of 6 years ago or so glibc has code like (I'm
only dealing with the init side of things as an
example):

.section .init,"ax",@progbits
stwu r1, -16(r1)
. . .
bcl 29,31,.LMAGIC_LABEL
.LMAGIC_LABEL:
mflr r30
addis r30, r30, _GLOBAL_OFFSET_TABLE_-.LMAGIC_LABEL@ha
addi  r30, r30, _GLOBAL_OFFSET_TABLE_-.LMAGIC_LABEL@l
. . . (some pre-init function code) . . .

that comes before code that is from frame_dummy
and __do_global_ctors_aux (that are from
/wrkdirs/usr/ports/devel/powerpc64-gcc/work/gcc-6.3.0/libgcc/crtstuff.c ).

The code generated for frame_dummy and
__do_global_ctors_aux expects r30 to already
be set up for _GLOBAL_OFFSET_TABLE_ related
use by the kind of code that I showed above.


Instead FreeBSD has for powerpc just:
(things are configured for devel/powerpc64-gcc
to use it)

#include <machine/asm.h>
__FBSDID("$FreeBSD: head/lib/csu/powerpc/crti.S 217399 2011-01-14 11:34:58Z kib $");

        .section .init,"ax",@progbits
        .align  2
        .globl  _init
        .type   _init,@function
_init:
        stwu 1,-16(1)
        mflr 0
        stw 31,12(1)
        stw 0,20(1)
        mr 31,1

The overall result ends up being (from
an example .so):

0000214c <_init> stwu    r1,-16(r1)
00002150 <_init+0x4> mflr    r0
00002154 <_init+0x8> stw     r31,12(r1)
00002158 <_init+0xc> stw     r0,20(r1)
0000215c <_init+0x10> mr      r31,r1
(The above is the FreeBSD crti.S code.)
(Note the lack of initialization of r30
to the _GLOBAL_OFFSET_TABLE_ related value.)

(The below is the crtstuff.c frame_dummy code
inlined in a way that the function prolog code
is not present here.)
(Note the dependence on r30 having already been
initialized.)
00002160 <_init+0x14> lwz     r3,-712(r30)
00002164 <_init+0x18> lwz     r9,0(r3)
00002168 <_init+0x1c> cmpwi   cr7,r9,0
0000216c <_init+0x20> beq-    cr7,00002184 <_init+0x38>
00002170 <_init+0x24> lwz     r9,-16(r30)
00002174 <_init+0x28> cmpwi   cr7,r9,0
00002178 <_init+0x2c> beq-    cr7,00002184 <_init+0x38>
0000217c <_init+0x30> mtctr   r9
00002180 <_init+0x34> bctrl

(The below is the crtstuff.c __do_global_ctors_aux
loop code but inlined. . .)
00002184 <_init+0x38> lwz     r29,-36(r30)
00002188 <_init+0x3c> lwzu    r9,-4(r29)
0000218c <_init+0x40> cmpwi   cr7,r9,-1
00002190 <_init+0x44> beq-    cr7,000021a8 <_init+0x5c>
00002194 <_init+0x48> mtctr   r9
00002198 <_init+0x4c> bctrl
0000219c <_init+0x50> lwzu    r9,-4(r29)
000021a0 <_init+0x54> cmpwi   cr7,r9,-1
000021a4 <_init+0x58> bne+    cr7,00002194 <_init+0x48>

(The rest of the .init code follows.)
000021a8 <_init+0x5c> lwz     r11,0(r1)
000021ac <_init+0x60> lwz     r0,4(r11)
000021b0 <_init+0x64> mtlr    r0
000021b4 <_init+0x68> lwz     r31,-4(r11)
000021b8 <_init+0x6c> mr      r1,r11
000021bc <_init+0x70> blr

The way the compiler's source code is structured
and works it looks to me like the crti.S used needs
to have the initialization code for r30: it must
provide that part of the function prolog code as
well.
Comment 6 Mark Millard 2017-10-29 14:12:17 UTC
For reference,

devel/powerpc64-gcc/work/gcc-6.3.0/libgcc/crtstuff.c

uses globals that drive the r30 use in the
code being put in the .init area. . .

In frame_dummy:

__JCR_LIST__
_Jv_RegisterClasses

(with some testing for weak references).

In __do_global_ctors_aux:

__CTOR_END__


Also: CRT_CALL_STATIC_FUNCTION use
for frame_dummy and for __do_global_ctors_aux
avoids the function prolog code of either
being put in the .init code.
Comment 7 Mark Millard 2017-10-29 22:39:43 UTC
I've looked an a copy of the 1995 Sun Microsystems
PPC SYSVR4 ABI document and its table 3-3 about
processor register usage is not explicit about
a specific register being used relative to
_GLOBAL_OFFSET_TABLE_ handling.

There are examples that say things like:

Assumes GOT pointer in r31

but as far as I can tell no place requires that.
In fact there is wording like:

Combining the offset with the global offset table address in a general register (for example, r31 loaded in the sample prologue in Figure 3-33) gives the absolute address of the table entry holding the desired address.

which explicitly indicates "in a general register".

It appears that crti.S type code for share
libraries requires a local convention for
the choice of register that the compiler and
library must agree on.

So r30 looks to be a valid choice but possibly
compiler specific for a FreeBSD context.

But there may be a reason to stick with r30. . .

Looking in Power-Arch-32-bit-ABI-supp-1.0-Linux.pdf
reports that:

Under the Secure-PLT ABI, when using the Position-Independent Code (PIC) addressing model, register r30 is used (by convention between compiler & link editor) in nonleaf functions to hold the Global Offset Table (GOT) pointer.

. . .

Using r30 to hold the address of the _GLOBAL_OFFSET_TABLE_ symbol is the current convention used
by the compiler and link-editor and is only required for nonleaf routines which use the PIC addressing model. Leaf routines or code not using the PIC addressing model may use any available unreserved general-purpose register to hold the address of the _GLOBAL_OFFSET_TABLE_ symbol.

. . .

The PIC call stub sequence requires that the compiler ensure that the register used to hold the _GLOBAL_OFFSET_TABLE_ pointer is set before any calls are made from the PLT. The current convention between the compiler and link editor is that r30 be used for this purpose. This is a change from the BSS-PLT ABI which only required GOT addressing to access static storage.

. . .

For non-PIC code, r30 will not hold the GOT pointer; so the stubs must be different, as shown in the following implementation.

. . .

Note: This ABI does not require a fixed GOT register, or even one register used throughout a binary. Non-PIC code does not set the _GLOBAL_OFFSET_TABLE_ pointer and does not need to reserve a register for that purpose. Code under the PIC addressing model that accesses static storage or calls nonlocal functions will need a register to hold the _GLOBAL_OFFSET_TABLE_ pointer. However, leaf functions or functions that only call other functions which are static (@local) may use any general-purpose register within the constraints for the existing ABI.



By contrast Power-Arch-32-bit-ABI-supp-1.0-Embedded.pdf
does not say much about r30 use, including not specifying
the Secure-PLT ABI material.

The referenced documents are examples of:

Power Architecture® 32-bit Application Binary Interface Supplement 1.0

documents published in, for example, 2011 (Power.org
copyright for that date but various other copyrights
for various earlier dates).
Comment 8 Mark Millard 2017-10-29 23:34:19 UTC
(In reply to Mark Millard from comment #7)

I was not explicit but it appears that the crti.S
like code (but targeting devel/powerpc64-gcc code
generation conventions relative to r30 use) and
the .init material need to together treat r30 as
non-volatile overall in order to match the ABI
requirements for such.
Comment 9 Mark Millard 2017-10-30 02:13:39 UTC
--print-file-name seems to show some
odd differences among clang ,
powerpc64-unknown-freebsd12.0-gcc ,
and gcc7 for the likes of crti.o
and crtbeginS.o . For cross builds
powerpc64-unknown-freebsd12.0-gcc
seems to be broken in this area.

These first ones are from on
my amd64 -> powerpc64 cross build
environment:

# which clang
/usr/bin/clang

# clang --print-file-name=crti.o
/usr/lib/crti.o
# clang -m32 --print-file-name=crti.o
/usr/lib32/crti.o

# clang --print-file-name=crtbeginS.o
/usr/lib/crtbeginS.o
# clang -m32 --print-file-name=crtbeginS.o
/usr/lib32/crtbeginS.o

# powerpc64-unknown-freebsd12.0-gcc --print-file-name=crti.o
crti.o
# powerpc64-unknown-freebsd12.0-gcc -m32 --print-file-name=crti.o
crti.o

# powerpc64-unknown-freebsd12.0-gcc --print-file-name=crtbeginS.o
crtbeginS.o
# powerpc64-unknown-freebsd12.0-gcc -m32 --print-file-name=crtbeginS.o                                                                                                                     crtbeginS.o

(Note the lack of path prefixes above: they are not
explicitly from /usr/local/lib/. . . for some reason.
System ones would not be appropriate. Note that nothing
differentiates -m32 use.)

# gcc7 --print-file-name=crti.o
/usr/lib/crti.o
# gcc7 -m32 --print-file-name=crti.o
/usr/lib/crti.o

# gcc7 --print-file-name=crtbeginS.o
/usr/local/lib/gcc7/gcc/x86_64-portbld-freebsd12.0/7.2.0/crtbeginS.o
# gcc7 -m32 --print-file-name=crtbeginS.o                                                                                                                                                  /usr/local/lib/gcc7/gcc/x86_64-portbld-freebsd12.0/7.2.0/crtbeginS.o

(Odd mix of system and gcc7 materials above?)

On my (clang built) powerpc64 environment
powerpc64-unknown-freebsd12.0-gcc ends
up with different results:

# clang --print-file-name=crti.o
/usr/lib/crti.o
# clang -m32 --print-file-name=crti.o
/usr/lib32/crti.o

# clang --print-file-name=crtbeginS.o
/usr/lib/crtbeginS.o
# clang -m32 --print-file-name=crtbeginS.o
/usr/lib32/crtbeginS.o

# powerpc64-unknown-freebsd12.0-gcc --print-file-name=crti.o
/usr/lib/crti.o
# powerpc64-unknown-freebsd12.0-gcc -m32 --print-file-name=crti.o
/usr/lib/../lib32/crti.o

# powerpc64-unknown-freebsd12.0-gcc --print-file-name=crtbeginS.o
/usr/lib/crtbeginS.o
# powerpc64-unknown-freebsd12.0-gcc -m32 --print-file-name=crtbeginS.o
/usr/lib/../lib32/crtbeginS.o


(Path prefixes present above. Using system ones would
not be likely to match using /usr/local/lib/. . . ones
from a cross build environment. But to have builds
from both hosts be equivalent would require the hosts
to use the same files [by content].)

# gcc7 --print-file-name=crti.o
/usr/lib/crti.o
# gcc7 -m32 --print-file-name=crti.o
/usr/lib/../lib32/crti.o

# gcc7 --print-file-name=crtbeginS.o
/usr/local/lib/gcc7/gcc/powerpc64-portbld-freebsd12.0/7.2.0/crtbeginS.o
# gcc7 -m32 --print-file-name=crtbeginS.o
/usr/local/lib/gcc7/gcc/powerpc64-portbld-freebsd12.0/7.2.0/32/crtbeginS.o


(Again the odd(?) system vs. gcc mix for gcc7.)

Does devel/powerpc64-gcc for cross builds require
taking control of the crti.o and crtbeginS.o (and
. . .) generation to force a match to what
self-hosted would have on the target architecture?
Comment 10 Mark Millard 2017-10-30 02:54:07 UTC
(In reply to Mark Millard from comment #9)

Looking around shows that for the cross
build context devel/powerpc64-gcc does
not even have external files for crti.o
or crtbeginS.o or the like to copy from.

Instead it either has internal copies
or is doing on the fly generation of the
content.

# find /usr/local/ -name "crtbeginS.o" -print | more                                                                       
/usr/local/lib/gcc7/gcc/x86_64-portbld-freebsd12.0/7.2.0/crtbeginS.o

(This also confirms that lack of a . . ./32/crtbeginS.o
for the amd64 native gcc7 compiler. The same path
repeating for -m32 was no accident on amd64.)

# find /usr/local/ -name "crti.o" -print | more                                                                                                                                       

(nothing found.) So looking for *crt*.o:

# find /usr/local/ -name "*crt*.o" -print | more
/usr/local/lib/gcc7/gcc/x86_64-portbld-freebsd12.0/7.2.0/crtend.o
/usr/local/lib/gcc7/gcc/x86_64-portbld-freebsd12.0/7.2.0/crtendS.o
/usr/local/lib/gcc7/gcc/x86_64-portbld-freebsd12.0/7.2.0/crtbeginS.o
/usr/local/lib/gcc7/gcc/x86_64-portbld-freebsd12.0/7.2.0/crtbegin.o

devel/powerpc64-gcc has none.

Things only work when the on-the-fly/internal code
it generates/uses happens to match well with its
environment. The glibc tie for r30 use for GOT
is not a match for FreeBSD.
Comment 11 Mark Millard 2017-10-30 03:47:46 UTC
(In reply to Mark Millard from comment #9)

FYI: despite the lack of a path prefix
from --print-file-name . . .

It appears to me that what appears in the
lib32 .so.* files for .init is from:

/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/world32/usr/src/gnu/lib/csu/crt*.o

content.

Apparently without the path prefix it is looking
based on other path specifications to find something.
It just can not report later what it actually found
and used for the file requested.


So, not as bad as I was thinking earlier.
Comment 12 Mark Millard 2017-10-30 05:07:51 UTC
The problem is based on processing FreeBSD
source code, including /usr/src/contrib/gcc/crtstuff.c
and /usr/src/lib/csu/powerpc64/crt[in].S :

# find /usr/src/* /usr/obj/powerpc64vtsc_xtoolchain-gcc/ -name "*crt[in]*.*" -print | more
. . .
/usr/src/lib/csu/powerpc64/crti.S
/usr/src/lib/csu/powerpc64/crtn.S
. . .
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/lib/csu/powerpc64/crtn.o.meta
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/lib/csu/powerpc64/crtn.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/lib/csu/powerpc64/crti.o.meta
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/lib/csu/powerpc64/crti.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/world32/usr/src/lib/csu/powerpc/crtn.o.meta
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/world32/usr/src/lib/csu/powerpc/crtn.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/world32/usr/src/lib/csu/powerpc/crti.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/world32/usr/src/lib/csu/powerpc/crti.o.meta
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/tmp/usr/lib/crti.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/tmp/usr/lib/crtn.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/lib32/usr/lib32/crti.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/lib32/usr/lib32/crtn.o


# find /usr/src/* /usr/obj/powerpc64vtsc_xtoolchain-gcc/ -name "*crt[be]*S.*" -print | more
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/gnu/lib/csu/crtbeginS.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/gnu/lib/csu/crtendS.o.meta
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/gnu/lib/csu/crtendS.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/gnu/lib/csu/crtbeginS.o.meta
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/world32/usr/src/gnu/lib/csu/crtbeginS.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/world32/usr/src/gnu/lib/csu/crtendS.o.meta
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/world32/usr/src/gnu/lib/csu/crtbeginS.o.meta
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/world32/usr/src/gnu/lib/csu/crtendS.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/tmp/usr/lib/crtbeginS.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/tmp/usr/lib/crtendS.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/lib32/usr/lib32/crtendS.o
/usr/obj/powerpc64vtsc_xtoolchain-gcc/powerpc.powerpc64/usr/src/lib32/usr/lib32/crtbeginS.o

Showing -m32 examples:

-DCOMPAT_32BIT -mcpu=powerpc -m32
-DIN_GCC -DHAVE_LD_EH_FRAME_HDR -DDT_CONFIG -D__GLIBC__=3
-g0 -DCRT_BEGIN -DCRTSTUFFS_O
-DSHARED -fpic
-c -o crtbeginS.o
/usr/src/contrib/gcc/crtstuff.c


-DCOMPAT_32BIT -mcpu=powerpc -m32
-DIN_GCC -DHAVE_LD_EH_FRAME_HDR -DDT_CONFIG -D__GLIBC__=3
-g0 -DCRT_END -DCRTSTUFFS_O
-DSHARED -fpic
-o crtendS.o
/usr/src/contrib/gcc/crtstuff.c

(I omitted many command line options.)

It just appears that for devel/powerpc64-gcc and the
like crt[in].S analogous content but that is matched
to the category of compiler's register usage specifics
is needed (for issues the ABI does not make unique
across compilers but that still have conventions).
Comment 13 Mark Millard 2020-03-10 23:01:53 UTC
Other issues associated with the ABI updates for the
powerpc families prevents getting as far as testing
this again (based on modern devel/freebsd-gcc9@powerpc64
usage). Builds abort or hang (gnu ld stuck in a loop).

There is also the -msvr4-struct-return vs.
-maix-struct-return issue, where FreeBSD and its clang
(addicentally?) changed but gcc* is the old way,
leading to an ABI mismatch even with lib32 not
involved.

It may be some time before things are back to the
point of allowing lib32 handling to be checked for
devel/freebsd-gcc*@powerpc64 based buildworld's.