Bug 241728 - linuxkpi: panics after loading radeonkms kernel module (data storage interrupt trap)
Summary: linuxkpi: panics after loading radeonkms kernel module (data storage interrup...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: powerpc Any
: --- Affects Some People
Assignee: freebsd-emulation mailing list
URL:
Keywords: crash, needs-qa, regression
Depends on:
Blocks:
 
Reported: 2019-11-05 08:37 UTC by Piotr Kubaj
Modified: 2019-11-12 06:19 UTC (History)
9 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Piotr Kubaj freebsd_committer 2019-11-05 08:37:57 UTC
I'm on head on powerpc64 elfv2 (r353588) and experience kernel panic after kldload radeonkms. I found it's due to linuxkpi.
root@talos:~ # kldload radeonkms.ko

fatal kernel trap:

   exception       = 0x300 (data storage interrupt)
   virtual address = 0xcaa24580
   dsisr           = 0x42000000
   srr0            = 0xe0000000ce3e5dc4 (0x20000000cbf95dc4)
   srr1            = 0x9000000000009032
   current msr     = 0x9000000000009032
   lr              = 0xc000000002cced44 (0x87ed44)
   frame           = 0xe0000000bf08df00
   curthread       = 0xc00000002c43a560
          pid = 47271, comm = kldload

panic: data storage interrupt trap
cpuid = 16
time = 1572942809
KDB: stack backtrace:
0xe0000000bf08da80: at kdb_backtrace+0x60
0xe0000000bf08db90: at vpanic+0x1d8
0xe0000000bf08dc40: at panic+0x40
0xe0000000bf08dc70: at trap+0x1564
0xe0000000bf08dd00: at trap+0x62c
0xe0000000bf08de40: at powerpc_interrupt+0x1b8
0xe0000000bf08ded0: kernel DSI write trap @ 0xcaa24580 by idr_preload_init+0xc8: srr1=0x9000000000009032
            r1=0xe0000000bf08e180 cr=0x4488dc88 xer=0 ctr=0xc000000002ce7190 r2=0xe0000000ce401ed0 sr=0x42000000 frame=0xe0000000bf08df00
0xe0000000bf08e180: at 0xe0000000ce3c3a04
0xe0000000bf08e1f0: at linker_reference_module+0xfd4
0xe0000000bf08e630: at linker_load_dependencies+0x2b0
0xe0000000bf08e710: at elf_get_symname+0x132c
0xe0000000bf08e960: at linker_reference_module+0xc88
0xe0000000bf08eda0: at linker_load_dependencies+0x2b0
0xe0000000bf08ee80: at elf_get_symname+0x132c
0xe0000000bf08f0d0: at linker_reference_module+0xc88
0xe0000000bf08f510: at kern_kldload+0x138
0xe0000000bf08f5c0: at sys_kldload+0x8c
0xe0000000bf08f610: at trap+0xd58
0xe0000000bf08f750: at powerpc_interrupt+0x1b8
0xe0000000bf08f7e0: user SC trap by 0x810102f20: srr1=0x900000000200d032
            r1=0x3fffffffffffd460 cr=0x44222b80 xer=0 ctr=0x810102f10 r2=0x8102a2398 frame=0xe0000000bf08f810
KDB: enter: panic
[ thread pid 47271 tid 100787 ]
Stopped at      kdb_enter+0x78: ori     r0, r0, 0x0
db> kldstat
Id Refs Address            Size     Name
 1   21 0xc000000002550100 1a611c8  kernel
 2    1 0xe0000000c1900000 22000    tmpfs.ko
 3    1 0xe0000000c4640000 21000    icee.ko
 4    1 0xe0000000c5480000 21000    mac_ntpd.ko
 5    1 0xe0000000c7820000 21000    imgact_binmisc.ko
 6    1 0xe0000000ce600000 1e8000   radeonkms.ko
 7    1 0xe0000000ce300000 c9000    drm.ko
 8    1 0xe0000000ce3c9000 33000    linuxkpi.ko

You can see it's caused by idr_preload_init, at offset 0xe0000000ce3e5dc4 (srr0) - 0xe0000000ce3c9000 (address of linuxkpi.ko) = 0x1d00, which is this file:
sys/compat/linuxkpi/common/src/linux_idr.c, line:
118                         free(cacheval, M_IDR);
Comment 1 Piotr Kubaj freebsd_committer 2019-11-05 08:39:46 UTC
Adding to CC developers that recently commited some changes.
Comment 2 Hans Petter Selasky freebsd_committer 2019-11-05 08:42:52 UTC
Is this a regression issue?

Has all code be re-compiled from source?

--HPS
Comment 3 Piotr Kubaj freebsd_committer 2019-11-05 08:49:32 UTC
(In reply to Hans Petter Selasky from comment #2)
Yes, I successfully loaded radeonkms and used X11 in September (I unfortunately can't say what revision was that). World and kernel both have been recompiled and match what's in /usr/src.
Comment 4 Hans Petter Selasky freebsd_committer 2019-11-05 14:05:14 UTC
The piece of code you reference, line 118 in sys/compat/linuxkpi/common/src/linux_idr.c, should not be run during kldload. It is part of a SYSUNINIT():


SYSUNINIT(idr_preload_uninit, SI_SUB_LOCK, SI_ORDER_FIRST, idr_preload_uninit, N

In this function, try to add a printf:

static void
idr_preload_init(void *arg)
{
        int cpu;

        CPU_FOREACH(cpu) {
                struct linux_idr_cache *lic =
                    DPCPU_ID_PTR(cpu, linux_idr_cache);

printf("CPU=%d LIC=%p\n", cpu, lic);

                spin_lock_init(&lic->lock);
        }
}
SYSINIT(idr_preload_init, SI_SUB_CPU, SI_ORDER_ANY, idr_preload_init, NULL);
 
--HPS
Comment 5 Hans Petter Selasky freebsd_committer 2019-11-05 14:08:33 UTC
And I presume "radeonkms" has been re-built aswell.
Comment 6 Piotr Kubaj freebsd_committer 2019-11-05 14:23:31 UTC
FYI happens also on r354356.
Comment 7 Hans Petter Selasky freebsd_committer 2019-11-05 14:26:45 UTC
Sure, but did you re-build radeonkms.ko from ports?
Comment 8 Piotr Kubaj freebsd_committer 2019-11-05 14:30:14 UTC
(In reply to Hans Petter Selasky from comment #5)
Yes, it has been rebuilt. It happens also when doing only kldload linuxkpi, which doesn't involve radeonkms.

Our (PPC devs) hypothesis is that something is wrong and linuxkpi bails out and tries to exit, that's why sysuninit is executed.
Comment 9 Hans Petter Selasky freebsd_committer 2019-11-05 15:06:19 UTC
Can you build a kernel with
options VERBOSE_SYSINIT=1

And get the resulting dmesg when loading the LinuxKPI ?

--HPS
Comment 10 Hans Petter Selasky freebsd_committer 2019-11-05 15:13:51 UTC
You can also try to build the LinuxKPI into the kernel:

options COMPAT_LINUXKPI

I think it is a generic issue and not related to the LinuxKPI.

--HPS
Comment 11 Piotr Kubaj freebsd_committer 2019-11-05 16:03:51 UTC
I'm building it now with VERBOSE_SYSINIT (I will build kernel with linuxkpi later on). For now, I built with the printf. It throws in the console before the panic:
CPU=0 LIC=0xc6119000
Comment 12 Hans Petter Selasky freebsd_committer 2019-11-05 16:13:00 UTC
There has been some changes recently to per-CPU data. Maybe that is related.

Basically this code shouldn't panic.
Comment 13 Piotr Kubaj freebsd_committer 2019-11-05 16:23:48 UTC
Adding options VERBOSE_SYSINIT=1 didn't change the output on the console after loading linuxkpi.
Comment 14 Piotr Kubaj freebsd_committer 2019-11-05 16:43:04 UTC
Yes, putting linuxkpi to kernel doesn't make it panic during boot.
Comment 15 Piotr Kubaj freebsd_committer 2019-11-05 19:34:14 UTC
Is it acceptable for COMPAT_LINUXKPI to be put to GENERIC and GENERIC64 kernels? In that case I think it would be best to do it for all architectures on which linuxkpi works to avoid regressions.

Otherwise, let me know if you need additional assistance with debugging it. If it helps, I can provide a remote access to my Talos.
Comment 16 Hans Petter Selasky freebsd_committer 2019-11-06 07:03:29 UTC
If you can binary search when this regression was introduced that would be the best.

--HPS
Comment 17 Piotr Kubaj freebsd_committer 2019-11-07 09:55:53 UTC
(In reply to Hans Petter Selasky from comment #16)
After going as far back as to r350000 (only kernel), I was still experiencing this panic.

As it turns out, it doesn't happen when using ELFv1 environment (which built with base GCC). It only happens on ELFv2 (built with LLVM).

In the recent weeks there has actually been a major LLVM change - LLVM has been upgraded to 9.0.0. Since this bug didn't happen with 8.0.1, this must be the reason.
Comment 18 Piotr Kubaj freebsd_committer 2019-11-07 09:56:00 UTC
(In reply to Hans Petter Selasky from comment #16)
After going as far back as to r350000 (only kernel), I was still experiencing this panic.

As it turns out, it doesn't happen when using ELFv1 environment (which built with base GCC). It only happens on ELFv2 (built with LLVM).

In the recent weeks there has actually been a major LLVM change - LLVM has been upgraded to 9.0.0. Since this bug didn't happen with 8.0.1, this must be the reason.
Comment 19 Mark Millard 2019-11-11 02:40:57 UTC
(In reply to Piotr Kubaj from comment #18)

I got a little time with access to the PowerMac11,2 and have
produced a "kldload linuxkpi" crash on a system built with:

# cc -v
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
Target: powerpc64-unknown-freebsd13.0
Thread model: posix
InstalledDir: /usr/bin

# uname -apKU
FreeBSD FBSDG5L2 13.0-CURRENT FreeBSD 13.0-CURRENT #19 r352341M: Sat Sep 14 19:40:27 PDT 2019     markmi@FBSDFHUGE:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/sys/GENERIC64vtsc-NODBG  powerpc powerpc64 1300047 1300047

It was built using devel/powerpc64-binutils (the "altbinutils"). It is
also a non-debug build, but with symbols.

Despite other differences with your backtrace, both identify idr_preload_init .

Typed from a picture (so subject to typos):

fatal kernel trap:
exception       = 0x300 (data storage interrupt)
virtual address = 0x8e076000
dsidr           = 0x42000000
srr0            = 0xe00000008f4aec8c (0x200000008f3aec8c) (f4 vs. f3 is not a typo)
srr1            = 0x9000000000009032
current msr     = 0x0000000000009032
lr              = 0xc000000000832470 (0x732470)
frame           = 0xe0000000900aeed0
curthread       = 0xc00000000e086000
pid = 1256, comm = kldload

panic: data storage interrupt
cpuid = 0
time = 1573434333
KDB: stack backtrace
0xe0000000900aec00: at aiocb32_store_status+0x34
0xe0000000900aecb0: at panic+0x44
0xe0000000900aece0: at trap_fatal+0x2f4
0xe0000000900aed70: at trap+0x5d8
0xe0000000900aee50: at powerpc_interrupt+0x1a0
0xe0000000900aeea0: at kernel DSI write trap @0x8e076000
                    by idr_preload_init+0xa4
                    srr1 = 0x9000000000009032
                    r1   = 0xe0000000900af150
                    cr   = 0x4880fc88
                    xer  = 0
                    ctr  = 0xe00000008f4aebe8
                    r2   = 0xe00000008f4d6300
                    frame= 0xe0000000900aeed0
0xe0000000900af150: at 0xc00000000e085ffc
0xe0000000900af1c0: at sem_prison_check+0x170
0xe0000000900af5f0: at kern_kldload+0x158
0xe0000000900af660: at sys_kldload+0x8c
0xe0000000900af6b0: at trap+0xe68
0xe0000000900af790: at powerpc_interrupt+0x1a0
0xe0000000900af7e0: user SC trap
                    by 0x8100e9158
                    srr1 = 0x900000000200f032
                    r1   = 0x3fffffffffffd470
                    cr   = 0x22222800
                    xer  = 0x20000000
                    ctr  = 0x8100e9150
                    r2   = 0x8102a2400
                    frame= 0xe0000000900af810
KDB: enter: panic
[ thread pid 1256 tid 100172 ]
Stopped at ifaddr_byindex+0xc orl r0,r0,0x0
Comment 20 Mark Millard 2019-11-11 02:42:33 UTC
(In reply to Mark Millard from comment #19)

I see that I forgot to mention a major point: ELFv1, not ELFv2.
So it suggests that clang vs. gcc 4.2.1 makes the difference,
not ELFv2 vs. ELFv1.
Comment 21 Mark Millard 2019-11-11 03:05:18 UTC
(In reply to Piotr Kubaj from comment #18)

My example in comment 19 is based on FreeBSD clang version 8.0.1 used to
buildworld and buildkernel to produce the system that was running,
not based on clang 9. (As well as ELFv1, not ELFv2.) The context predates
the update to clang 9 in head. (It is just what I happened to already
have around for powerpc64.)
Comment 22 Mark Millard 2019-11-11 03:31:19 UTC
(In reply to Piotr Kubaj from comment #0)

This defect is from /usr/src/sys/compat/linuxkpi/common/src/linux_idr.c using:

struct linux_idr_cache {
        spinlock_t lock;
        struct idr_layer *head;
        unsigned count;
};

DPCPU_DEFINE_STATIC(struct linux_idr_cache, linux_idr_cache);

The same type of problem as for:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232387

I'll quote the resolution from 232387, to indicate what
would likely be analogous here:

Author: luporl
Date: Tue Jun 25 17:15:45 UTC 2019
New revision: 349377
URL: https://svnweb.freebsd.org/changeset/base/349377

Log:
  [PowerPC64] Don't mark module data as static

  Fixes panic when loading ipfw.ko and if_epair.ko built with modern compiler.

  Similar to arm64 and riscv, when using a modern compiler (!gcc4.2), code
  generated tries to access data in the wrong location, causing kernel panic
  (data storage interrupt trap) when loading if_epair and ipfw.

  Issue was reproduced with kernel/module compiled using gcc8 and clang8. It
  affects both ELFv1 and ELFv2 ABI environments.

  PR:		232387
  Submitted by:	alfredo.junior_eldorado.org.br
  Reported by:	Mark Millard
  Reviewed by:	jhibbits
  Differential Revision:	https://reviews.freebsd.org/D20461

Changes:
  head/sys/net/vnet.h
  head/sys/sys/pcpu.h
Comment 23 Mark Millard 2019-11-11 04:51:51 UTC
(In reply to Mark Millard from comment #22)

In my rush for time and interruptions, my wording sounds like
I was saying the workaround was not attempted. Better to show
details:

From the .kld:

Relocation section with addend (.rela.toc):
r_offset     r_info       r_type              st_value         st_name + r_addend
. . .
0000000000f8 018400000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_linux_idr_cache + 0
. . .

Relocation section with addend (.rela.debug_info):
r_offset     r_info       r_type              st_value         st_name + r_addend
. . .
00000001f60b 018400000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_linux_idr_cache + 0
. . .

Symbol table (.symtab) contains 818 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
. . .
   388: 0000000000000000    48 OBJECT  LOCAL  DEFAULT   19 pcpu_entry_linux_idr_cache
. . .


From the contributing .o:

Relocation section with addend (.rela.toc):
r_offset     r_info       r_type              st_value         st_name + r_addend
000000000000 004900000026 R_PPC64_ADDR64      0000000000000000 lockstat_enabled + 0
000000000008 005000000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_linux_idr_cache + 0
000000000010 002c00000026 R_PPC64_ADDR64      0000000000000000 all_cpus + 0
000000000018 004e00000026 R_PPC64_ADDR64      0000000000000000 mp_maxid + 0
000000000020 002d00000026 R_PPC64_ADDR64      0000000000000000 dpcpu_off + 0
. . .
Relocation section with addend (.rela.debug_info):
r_offset     r_info       r_type              st_value         st_name + r_addend
. . .
000000000096 005000000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_linux_idr_cache + 0
00000000009f 001900000001 R_PPC64_ADDR32      0000000000000000 .debug_str + 2c9
. . .

Relocation section with addend (.rela.debug_line):
r_offset     r_info       r_type              st_value         st_name + r_addend
000000000889 001300000026 R_PPC64_ADDR64      0000000000000000 .text + 0
. . .

Symbol table (.symtab) contains 81 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
. . .
    80: 0000000000000000    48 OBJECT  GLOBAL DEFAULT   13 pcpu_entry_linux_idr_cache

So a "LOCAL" resulted from a "GLOBAL". This looks to mean it was
turned back into a static.

Looking . . .

# Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/sys/GENERIC64vtsc-NODBG/modules/usr/src/sys/modules/linuxkpi/linuxkpi.kld.meta
CMD /usr/local/powerpc64-unknown-freebsd13.0/bin/ld -m elf64ppc_fbsd -d -warn-common --build-id=sha1 -r -d -o linuxkpi.kld linux_compat.o linux_current.o linux_hrtimer.o linux_idr.o linux_kmod.o linux
_kthread.o linux_lock.o linux_page.o linux_pci.o linux_radix.o linux_rcu.o linux_seq_file.o linux_schedule.o linux_slab.o linux_tasklet.o linux_usb.o linux_work.o
CMD ctfmerge -L VERSION -g -o linuxkpi.kld linux_compat.o linux_current.o linux_hrtimer.o linux_idr.o linux_kmod.o linux_kthread.o linux_lock.o linux_page.o linux_pci.o linux_radix.o linux_rcu.o linux
_seq_file.o linux_schedule.o linux_slab.o linux_tasklet.o linux_usb.o linux_work.o
CMD :> export_syms
CMD awk -f /usr/src/sys/conf/kmod_syms.awk linuxkpi.kld  export_syms | xargs -J% /usr/local/powerpc64-unknown-freebsd13.0/bin/objcopy % linuxkpi.kld

# more /usr/src/sys/conf/kmod_syms.awk
# $FreeBSD: head/sys/conf/kmod_syms.awk 101438 2002-08-06 19:31:04Z iedowse $

# Read global symbols from object file.
BEGIN {
        while ("${NM:='nm'} -g " ARGV[1] | getline) {
                if (match($0, /^[^[:space:]]+ [^AU] (.*)$/)) {
                        syms[$3] = $2
                }
        }
        delete ARGV[1]
}

# De-list symbols from the export list.
{
        delete syms[$0]
}

# Strip commons, make everything else local.
END {
        for (member in syms) {
                if (syms[member] == "C")
                        print "-N" member
                else
                        print "-L" member
        }
}
Comment 24 Mark Millard 2019-11-12 00:23:53 UTC
(In reply to Piotr Kubaj from comment #0)

[This note reports 5 or so more pcpu_entry_'s that are
LOCAL -- and notes that there are around 300 vnet_entry_'s
that are LOCAL (around 98 GLOBAL). I start with
pcpu_entry_'s.]

For my context, beyond linuxkpi.kld there is also . . .

ipfw.kld having:

   469: 0000000000000000     8 OBJECT  LOCAL  DEFAULT   20 pcpu_entry_dyn_hp

siftr.kld having:

    91: 0000000000000000    56 OBJECT  LOCAL  DEFAULT   18 pcpu_entry_ss

which would seem likely to have the same type of problem as linuxkpi.kld
ended up with because of the "LOCAL" final results in the .kld files.

ipfw, linuxkpi, and siftr were mentioned in bug 232387 (but possibly not
all completely fixed). This prompted for me to check my clang 8.0.1 ELFv1
based build tree targeting powerpc64 to see teh above.

I have not tested loading ipfw.ko or siftr.ko . I just looked for
the pcpu_entry_<?> information via readelf on the .kld files.

Going the other way, looking for .kld files with pcpu_entry_ text in
that build that I've been looking at: It has 3 more LOCAL's beyond the
3 noted so far:

pcpu_entry_epair_dpcpu ( /usr/src/sys/net/if_epair.c )
pcpu_entry_tasklet_worker ( sys/compat/linuxkpi/common/src/linux_tasklet.c )
pcpu_entry_linux_epoch_record ( sys/compat/linuxkpi/common/src/linux_rcu.c )

The first uses DPCPU_DEFINE and the other two use DPCPU_DEFINE_STATIC, but
LOCAL can be forced other ways than the macro definitions ( as in the earlier
linuxkld.kld notes ).

For reference:

# find /usr/obj/powerpc64vtsc_clang_altbinutils/ -name '*.kld' -exec readelf -a {} \; | grep pcpu_entry_
000000000020 017800000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_pmc_sampled + 0
000000000168 017800000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_pmc_sampled + 0
   376: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND pcpu_entry_pmc_sampled
000000000010 005000000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_epair_dpcpu + 0
00000000017d 005000000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_epair_dpcpu + 0
    80: 0000000000000000    56 OBJECT  LOCAL  DEFAULT   19 pcpu_entry_epair_dpcpu
0000000001f8 01d500000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_dyn_hp + 0
000000015f9b 01d500000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_dyn_hp + 0
   469: 0000000000000000     8 OBJECT  LOCAL  DEFAULT   20 pcpu_entry_dyn_hp
0000000000f8 018400000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_linux_idr_cache + 0
0000000001e0 020b00000026 R_PPC64_ADDR64      0000000000000080 pcpu_entry_linux_epoch_record + 0
000000000290 01c000000026 R_PPC64_ADDR64      0000000000000200 pcpu_entry_tasklet_worker + 0
00000001f60b 018400000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_linux_idr_cache + 0
00000004e25d 020b00000026 R_PPC64_ADDR64      0000000000000080 pcpu_entry_linux_epoch_record + 0
0000000647e9 01c000000026 R_PPC64_ADDR64      0000000000000200 pcpu_entry_tasklet_worker + 0
   388: 0000000000000000    48 OBJECT  LOCAL  DEFAULT   19 pcpu_entry_linux_idr_cache
   448: 0000000000000200   256 OBJECT  LOCAL  DEFAULT   19 pcpu_entry_tasklet_worker
   523: 0000000000000080   384 OBJECT  LOCAL  DEFAULT   19 pcpu_entry_linux_epoch_record
000000000020 005b00000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_ss + 0
0000000030e5 005b00000026 R_PPC64_ADDR64      0000000000000000 pcpu_entry_ss + 0
    91: 0000000000000000    56 OBJECT  LOCAL  DEFAULT   18 pcpu_entry_ss


The list for vnet_entry_'s that are LOCAL in my build is very long,
around 300 of them.

I've not done anything to certify that they all lead to failure. On
what I've done it just suggests that they need consideration.

There are around 98 vnet_entry_'s that are GLOBAL.

If there are more than just pcpu_entry_'s and vnet_entry_'s that
would have the issue, than I'm unaware and have not covered them
at all.
Comment 25 Brandon Bergren freebsd_committer 2019-11-12 03:38:05 UTC
See also https://reviews.freebsd.org/D22317

The pcpu and vnet linker sets (implemented as custom elf sections) depend on being able to be manually relocated by the kernel linker to an arbitrary location to actually work, so we can't allow relaxing references into displacement-via-r2 because it's not necessarily a specific distance from ".", and in this case is not even in the same map area (the symbols get relocated out of KVA and down to the DMAP manually by the in-kernel linker so they land in the reserved space for the data structure, and reads/writes are required to add a base address to get from there back up to the KVA copy)

Either we're doing a gross ABI violation or clang and lld need to not relax any references to sections with c identifier names (i.e. the ones not starting with a dot.)

I'm not sure which.
Comment 26 Mark Millard 2019-11-12 06:19:57 UTC
(In reply to Brandon Bergren from comment #25)

Looks like the patch is lld specific but my ELFv1/clang-8.0.1
based examples were from using devel/powerpc64-binutils instead
of using lld.

But is use of devel/powerpc64-binutils intended to be
supported via src.conf tailoring for how things are going
forwards? If yes, and if the ELFv1/clang-8.0.1 generalizes
to ELFv2/clang-9, then more may be needed. Otherwise, not.

(Note: I use WITHOUT_LIB32= when using devel/powerpc64-binutils
for other reasons so lib32 is not part of that test context.)