I'm on head on powerpc64 elfv2 (r353588) and experience kernel panic after kldload radeonkms. I found it's due to linuxkpi. root@talos:~ # kldload radeonkms.ko fatal kernel trap: exception = 0x300 (data storage interrupt) virtual address = 0xcaa24580 dsisr = 0x42000000 srr0 = 0xe0000000ce3e5dc4 (0x20000000cbf95dc4) srr1 = 0x9000000000009032 current msr = 0x9000000000009032 lr = 0xc000000002cced44 (0x87ed44) frame = 0xe0000000bf08df00 curthread = 0xc00000002c43a560 pid = 47271, comm = kldload panic: data storage interrupt trap cpuid = 16 time = 1572942809 KDB: stack backtrace: 0xe0000000bf08da80: at kdb_backtrace+0x60 0xe0000000bf08db90: at vpanic+0x1d8 0xe0000000bf08dc40: at panic+0x40 0xe0000000bf08dc70: at trap+0x1564 0xe0000000bf08dd00: at trap+0x62c 0xe0000000bf08de40: at powerpc_interrupt+0x1b8 0xe0000000bf08ded0: kernel DSI write trap @ 0xcaa24580 by idr_preload_init+0xc8: srr1=0x9000000000009032 r1=0xe0000000bf08e180 cr=0x4488dc88 xer=0 ctr=0xc000000002ce7190 r2=0xe0000000ce401ed0 sr=0x42000000 frame=0xe0000000bf08df00 0xe0000000bf08e180: at 0xe0000000ce3c3a04 0xe0000000bf08e1f0: at linker_reference_module+0xfd4 0xe0000000bf08e630: at linker_load_dependencies+0x2b0 0xe0000000bf08e710: at elf_get_symname+0x132c 0xe0000000bf08e960: at linker_reference_module+0xc88 0xe0000000bf08eda0: at linker_load_dependencies+0x2b0 0xe0000000bf08ee80: at elf_get_symname+0x132c 0xe0000000bf08f0d0: at linker_reference_module+0xc88 0xe0000000bf08f510: at kern_kldload+0x138 0xe0000000bf08f5c0: at sys_kldload+0x8c 0xe0000000bf08f610: at trap+0xd58 0xe0000000bf08f750: at powerpc_interrupt+0x1b8 0xe0000000bf08f7e0: user SC trap by 0x810102f20: srr1=0x900000000200d032 r1=0x3fffffffffffd460 cr=0x44222b80 xer=0 ctr=0x810102f10 r2=0x8102a2398 frame=0xe0000000bf08f810 KDB: enter: panic [ thread pid 47271 tid 100787 ] Stopped at kdb_enter+0x78: ori r0, r0, 0x0 db> kldstat Id Refs Address Size Name 1 21 0xc000000002550100 1a611c8 kernel 2 1 0xe0000000c1900000 22000 tmpfs.ko 3 1 0xe0000000c4640000 21000 icee.ko 4 1 0xe0000000c5480000 21000 mac_ntpd.ko 5 1 0xe0000000c7820000 21000 imgact_binmisc.ko 6 1 0xe0000000ce600000 1e8000 radeonkms.ko 7 1 0xe0000000ce300000 c9000 drm.ko 8 1 0xe0000000ce3c9000 33000 linuxkpi.ko You can see it's caused by idr_preload_init, at offset 0xe0000000ce3e5dc4 (srr0) - 0xe0000000ce3c9000 (address of linuxkpi.ko) = 0x1d00, which is this file: sys/compat/linuxkpi/common/src/linux_idr.c, line: 118 free(cacheval, M_IDR);
Adding to CC developers that recently commited some changes.
Is this a regression issue? Has all code be re-compiled from source? --HPS
(In reply to Hans Petter Selasky from comment #2) Yes, I successfully loaded radeonkms and used X11 in September (I unfortunately can't say what revision was that). World and kernel both have been recompiled and match what's in /usr/src.
The piece of code you reference, line 118 in sys/compat/linuxkpi/common/src/linux_idr.c, should not be run during kldload. It is part of a SYSUNINIT(): SYSUNINIT(idr_preload_uninit, SI_SUB_LOCK, SI_ORDER_FIRST, idr_preload_uninit, N In this function, try to add a printf: static void idr_preload_init(void *arg) { int cpu; CPU_FOREACH(cpu) { struct linux_idr_cache *lic = DPCPU_ID_PTR(cpu, linux_idr_cache); printf("CPU=%d LIC=%p\n", cpu, lic); spin_lock_init(&lic->lock); } } SYSINIT(idr_preload_init, SI_SUB_CPU, SI_ORDER_ANY, idr_preload_init, NULL); --HPS
And I presume "radeonkms" has been re-built aswell.
FYI happens also on r354356.
Sure, but did you re-build radeonkms.ko from ports?
(In reply to Hans Petter Selasky from comment #5) Yes, it has been rebuilt. It happens also when doing only kldload linuxkpi, which doesn't involve radeonkms. Our (PPC devs) hypothesis is that something is wrong and linuxkpi bails out and tries to exit, that's why sysuninit is executed.
Can you build a kernel with options VERBOSE_SYSINIT=1 And get the resulting dmesg when loading the LinuxKPI ? --HPS
You can also try to build the LinuxKPI into the kernel: options COMPAT_LINUXKPI I think it is a generic issue and not related to the LinuxKPI. --HPS
I'm building it now with VERBOSE_SYSINIT (I will build kernel with linuxkpi later on). For now, I built with the printf. It throws in the console before the panic: CPU=0 LIC=0xc6119000
There has been some changes recently to per-CPU data. Maybe that is related. Basically this code shouldn't panic.
Adding options VERBOSE_SYSINIT=1 didn't change the output on the console after loading linuxkpi.
Yes, putting linuxkpi to kernel doesn't make it panic during boot.
Is it acceptable for COMPAT_LINUXKPI to be put to GENERIC and GENERIC64 kernels? In that case I think it would be best to do it for all architectures on which linuxkpi works to avoid regressions. Otherwise, let me know if you need additional assistance with debugging it. If it helps, I can provide a remote access to my Talos.
If you can binary search when this regression was introduced that would be the best. --HPS
(In reply to Hans Petter Selasky from comment #16) After going as far back as to r350000 (only kernel), I was still experiencing this panic. As it turns out, it doesn't happen when using ELFv1 environment (which built with base GCC). It only happens on ELFv2 (built with LLVM). In the recent weeks there has actually been a major LLVM change - LLVM has been upgraded to 9.0.0. Since this bug didn't happen with 8.0.1, this must be the reason.
(In reply to Piotr Kubaj from comment #18) I got a little time with access to the PowerMac11,2 and have produced a "kldload linuxkpi" crash on a system built with: # cc -v FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1) Target: powerpc64-unknown-freebsd13.0 Thread model: posix InstalledDir: /usr/bin # uname -apKU FreeBSD FBSDG5L2 13.0-CURRENT FreeBSD 13.0-CURRENT #19 r352341M: Sat Sep 14 19:40:27 PDT 2019 markmi@FBSDFHUGE:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/sys/GENERIC64vtsc-NODBG powerpc powerpc64 1300047 1300047 It was built using devel/powerpc64-binutils (the "altbinutils"). It is also a non-debug build, but with symbols. Despite other differences with your backtrace, both identify idr_preload_init . Typed from a picture (so subject to typos): fatal kernel trap: exception = 0x300 (data storage interrupt) virtual address = 0x8e076000 dsidr = 0x42000000 srr0 = 0xe00000008f4aec8c (0x200000008f3aec8c) (f4 vs. f3 is not a typo) srr1 = 0x9000000000009032 current msr = 0x0000000000009032 lr = 0xc000000000832470 (0x732470) frame = 0xe0000000900aeed0 curthread = 0xc00000000e086000 pid = 1256, comm = kldload panic: data storage interrupt cpuid = 0 time = 1573434333 KDB: stack backtrace 0xe0000000900aec00: at aiocb32_store_status+0x34 0xe0000000900aecb0: at panic+0x44 0xe0000000900aece0: at trap_fatal+0x2f4 0xe0000000900aed70: at trap+0x5d8 0xe0000000900aee50: at powerpc_interrupt+0x1a0 0xe0000000900aeea0: at kernel DSI write trap @0x8e076000 by idr_preload_init+0xa4 srr1 = 0x9000000000009032 r1 = 0xe0000000900af150 cr = 0x4880fc88 xer = 0 ctr = 0xe00000008f4aebe8 r2 = 0xe00000008f4d6300 frame= 0xe0000000900aeed0 0xe0000000900af150: at 0xc00000000e085ffc 0xe0000000900af1c0: at sem_prison_check+0x170 0xe0000000900af5f0: at kern_kldload+0x158 0xe0000000900af660: at sys_kldload+0x8c 0xe0000000900af6b0: at trap+0xe68 0xe0000000900af790: at powerpc_interrupt+0x1a0 0xe0000000900af7e0: user SC trap by 0x8100e9158 srr1 = 0x900000000200f032 r1 = 0x3fffffffffffd470 cr = 0x22222800 xer = 0x20000000 ctr = 0x8100e9150 r2 = 0x8102a2400 frame= 0xe0000000900af810 KDB: enter: panic [ thread pid 1256 tid 100172 ] Stopped at ifaddr_byindex+0xc orl r0,r0,0x0
(In reply to Mark Millard from comment #19) I see that I forgot to mention a major point: ELFv1, not ELFv2. So it suggests that clang vs. gcc 4.2.1 makes the difference, not ELFv2 vs. ELFv1.
(In reply to Piotr Kubaj from comment #18) My example in comment 19 is based on FreeBSD clang version 8.0.1 used to buildworld and buildkernel to produce the system that was running, not based on clang 9. (As well as ELFv1, not ELFv2.) The context predates the update to clang 9 in head. (It is just what I happened to already have around for powerpc64.)
(In reply to Piotr Kubaj from comment #0) This defect is from /usr/src/sys/compat/linuxkpi/common/src/linux_idr.c using: struct linux_idr_cache { spinlock_t lock; struct idr_layer *head; unsigned count; }; DPCPU_DEFINE_STATIC(struct linux_idr_cache, linux_idr_cache); The same type of problem as for: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232387 I'll quote the resolution from 232387, to indicate what would likely be analogous here: Author: luporl Date: Tue Jun 25 17:15:45 UTC 2019 New revision: 349377 URL: https://svnweb.freebsd.org/changeset/base/349377 Log: [PowerPC64] Don't mark module data as static Fixes panic when loading ipfw.ko and if_epair.ko built with modern compiler. Similar to arm64 and riscv, when using a modern compiler (!gcc4.2), code generated tries to access data in the wrong location, causing kernel panic (data storage interrupt trap) when loading if_epair and ipfw. Issue was reproduced with kernel/module compiled using gcc8 and clang8. It affects both ELFv1 and ELFv2 ABI environments. PR: 232387 Submitted by: alfredo.junior_eldorado.org.br Reported by: Mark Millard Reviewed by: jhibbits Differential Revision: https://reviews.freebsd.org/D20461 Changes: head/sys/net/vnet.h head/sys/sys/pcpu.h
(In reply to Mark Millard from comment #22) In my rush for time and interruptions, my wording sounds like I was saying the workaround was not attempted. Better to show details: From the .kld: Relocation section with addend (.rela.toc): r_offset r_info r_type st_value st_name + r_addend . . . 0000000000f8 018400000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_linux_idr_cache + 0 . . . Relocation section with addend (.rela.debug_info): r_offset r_info r_type st_value st_name + r_addend . . . 00000001f60b 018400000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_linux_idr_cache + 0 . . . Symbol table (.symtab) contains 818 entries: Num: Value Size Type Bind Vis Ndx Name . . . 388: 0000000000000000 48 OBJECT LOCAL DEFAULT 19 pcpu_entry_linux_idr_cache . . . From the contributing .o: Relocation section with addend (.rela.toc): r_offset r_info r_type st_value st_name + r_addend 000000000000 004900000026 R_PPC64_ADDR64 0000000000000000 lockstat_enabled + 0 000000000008 005000000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_linux_idr_cache + 0 000000000010 002c00000026 R_PPC64_ADDR64 0000000000000000 all_cpus + 0 000000000018 004e00000026 R_PPC64_ADDR64 0000000000000000 mp_maxid + 0 000000000020 002d00000026 R_PPC64_ADDR64 0000000000000000 dpcpu_off + 0 . . . Relocation section with addend (.rela.debug_info): r_offset r_info r_type st_value st_name + r_addend . . . 000000000096 005000000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_linux_idr_cache + 0 00000000009f 001900000001 R_PPC64_ADDR32 0000000000000000 .debug_str + 2c9 . . . Relocation section with addend (.rela.debug_line): r_offset r_info r_type st_value st_name + r_addend 000000000889 001300000026 R_PPC64_ADDR64 0000000000000000 .text + 0 . . . Symbol table (.symtab) contains 81 entries: Num: Value Size Type Bind Vis Ndx Name . . . 80: 0000000000000000 48 OBJECT GLOBAL DEFAULT 13 pcpu_entry_linux_idr_cache So a "LOCAL" resulted from a "GLOBAL". This looks to mean it was turned back into a static. Looking . . . # Meta data file /usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/sys/GENERIC64vtsc-NODBG/modules/usr/src/sys/modules/linuxkpi/linuxkpi.kld.meta CMD /usr/local/powerpc64-unknown-freebsd13.0/bin/ld -m elf64ppc_fbsd -d -warn-common --build-id=sha1 -r -d -o linuxkpi.kld linux_compat.o linux_current.o linux_hrtimer.o linux_idr.o linux_kmod.o linux _kthread.o linux_lock.o linux_page.o linux_pci.o linux_radix.o linux_rcu.o linux_seq_file.o linux_schedule.o linux_slab.o linux_tasklet.o linux_usb.o linux_work.o CMD ctfmerge -L VERSION -g -o linuxkpi.kld linux_compat.o linux_current.o linux_hrtimer.o linux_idr.o linux_kmod.o linux_kthread.o linux_lock.o linux_page.o linux_pci.o linux_radix.o linux_rcu.o linux _seq_file.o linux_schedule.o linux_slab.o linux_tasklet.o linux_usb.o linux_work.o CMD :> export_syms CMD awk -f /usr/src/sys/conf/kmod_syms.awk linuxkpi.kld export_syms | xargs -J% /usr/local/powerpc64-unknown-freebsd13.0/bin/objcopy % linuxkpi.kld # more /usr/src/sys/conf/kmod_syms.awk # $FreeBSD: head/sys/conf/kmod_syms.awk 101438 2002-08-06 19:31:04Z iedowse $ # Read global symbols from object file. BEGIN { while ("${NM:='nm'} -g " ARGV[1] | getline) { if (match($0, /^[^[:space:]]+ [^AU] (.*)$/)) { syms[$3] = $2 } } delete ARGV[1] } # De-list symbols from the export list. { delete syms[$0] } # Strip commons, make everything else local. END { for (member in syms) { if (syms[member] == "C") print "-N" member else print "-L" member } }
(In reply to Piotr Kubaj from comment #0) [This note reports 5 or so more pcpu_entry_'s that are LOCAL -- and notes that there are around 300 vnet_entry_'s that are LOCAL (around 98 GLOBAL). I start with pcpu_entry_'s.] For my context, beyond linuxkpi.kld there is also . . . ipfw.kld having: 469: 0000000000000000 8 OBJECT LOCAL DEFAULT 20 pcpu_entry_dyn_hp siftr.kld having: 91: 0000000000000000 56 OBJECT LOCAL DEFAULT 18 pcpu_entry_ss which would seem likely to have the same type of problem as linuxkpi.kld ended up with because of the "LOCAL" final results in the .kld files. ipfw, linuxkpi, and siftr were mentioned in bug 232387 (but possibly not all completely fixed). This prompted for me to check my clang 8.0.1 ELFv1 based build tree targeting powerpc64 to see teh above. I have not tested loading ipfw.ko or siftr.ko . I just looked for the pcpu_entry_<?> information via readelf on the .kld files. Going the other way, looking for .kld files with pcpu_entry_ text in that build that I've been looking at: It has 3 more LOCAL's beyond the 3 noted so far: pcpu_entry_epair_dpcpu ( /usr/src/sys/net/if_epair.c ) pcpu_entry_tasklet_worker ( sys/compat/linuxkpi/common/src/linux_tasklet.c ) pcpu_entry_linux_epoch_record ( sys/compat/linuxkpi/common/src/linux_rcu.c ) The first uses DPCPU_DEFINE and the other two use DPCPU_DEFINE_STATIC, but LOCAL can be forced other ways than the macro definitions ( as in the earlier linuxkld.kld notes ). For reference: # find /usr/obj/powerpc64vtsc_clang_altbinutils/ -name '*.kld' -exec readelf -a {} \; | grep pcpu_entry_ 000000000020 017800000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_pmc_sampled + 0 000000000168 017800000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_pmc_sampled + 0 376: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND pcpu_entry_pmc_sampled 000000000010 005000000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_epair_dpcpu + 0 00000000017d 005000000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_epair_dpcpu + 0 80: 0000000000000000 56 OBJECT LOCAL DEFAULT 19 pcpu_entry_epair_dpcpu 0000000001f8 01d500000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_dyn_hp + 0 000000015f9b 01d500000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_dyn_hp + 0 469: 0000000000000000 8 OBJECT LOCAL DEFAULT 20 pcpu_entry_dyn_hp 0000000000f8 018400000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_linux_idr_cache + 0 0000000001e0 020b00000026 R_PPC64_ADDR64 0000000000000080 pcpu_entry_linux_epoch_record + 0 000000000290 01c000000026 R_PPC64_ADDR64 0000000000000200 pcpu_entry_tasklet_worker + 0 00000001f60b 018400000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_linux_idr_cache + 0 00000004e25d 020b00000026 R_PPC64_ADDR64 0000000000000080 pcpu_entry_linux_epoch_record + 0 0000000647e9 01c000000026 R_PPC64_ADDR64 0000000000000200 pcpu_entry_tasklet_worker + 0 388: 0000000000000000 48 OBJECT LOCAL DEFAULT 19 pcpu_entry_linux_idr_cache 448: 0000000000000200 256 OBJECT LOCAL DEFAULT 19 pcpu_entry_tasklet_worker 523: 0000000000000080 384 OBJECT LOCAL DEFAULT 19 pcpu_entry_linux_epoch_record 000000000020 005b00000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_ss + 0 0000000030e5 005b00000026 R_PPC64_ADDR64 0000000000000000 pcpu_entry_ss + 0 91: 0000000000000000 56 OBJECT LOCAL DEFAULT 18 pcpu_entry_ss The list for vnet_entry_'s that are LOCAL in my build is very long, around 300 of them. I've not done anything to certify that they all lead to failure. On what I've done it just suggests that they need consideration. There are around 98 vnet_entry_'s that are GLOBAL. If there are more than just pcpu_entry_'s and vnet_entry_'s that would have the issue, than I'm unaware and have not covered them at all.
See also https://reviews.freebsd.org/D22317 The pcpu and vnet linker sets (implemented as custom elf sections) depend on being able to be manually relocated by the kernel linker to an arbitrary location to actually work, so we can't allow relaxing references into displacement-via-r2 because it's not necessarily a specific distance from ".", and in this case is not even in the same map area (the symbols get relocated out of KVA and down to the DMAP manually by the in-kernel linker so they land in the reserved space for the data structure, and reads/writes are required to add a base address to get from there back up to the KVA copy) Either we're doing a gross ABI violation or clang and lld need to not relax any references to sections with c identifier names (i.e. the ones not starting with a dot.) I'm not sure which.
(In reply to Brandon Bergren from comment #25) Looks like the patch is lld specific but my ELFv1/clang-8.0.1 based examples were from using devel/powerpc64-binutils instead of using lld. But is use of devel/powerpc64-binutils intended to be supported via src.conf tailoring for how things are going forwards? If yes, and if the ELFv1/clang-8.0.1 generalizes to ELFv2/clang-9, then more may be needed. Otherwise, not. (Note: I use WITHOUT_LIB32= when using devel/powerpc64-binutils for other reasons so lib32 is not part of that test context.)
Fixed in r354713.