Created attachment 272088 [details] Simple test program that crashes on Pi-3 running 15.1 Various programs get segment faults. Simple test program attached. 15.0 works OK.
The discussion that led to this report is on freebsd-arm Subject: Seg Faults on Pi-3 after upgrading to 15.1 https://lists.freebsd.org/archives/freebsd-arm/2026-June/005962.html There is a report of it happening on a Pine64 LTS board. https://lists.freebsd.org/archives/freebsd-arm/2026-June/005965.html Bakul Shah has a test program in c https://lists.freebsd.org/archives/freebsd-arm/2026-June/005967.html
Created attachment 272127 [details] test program to crash on rpi3 + 15.1 add bakul's test program from their email I worked with Hal via email and collected some data. After ruling out voltage & temperature issues etc. & converting a python program Hal sent me to C and enhancing it (attached) this is what we discovered: The test program works fine on 15.0 & 15.1 on pi4. It works fine on amd64 (on 15-stable). It work fine on 15.0 but fails fairly quickly on 15.1 on pi3. When modified to call inet_ntop() on IPv4 addresses works fine on 15.1 on pi3. The test program fails after random number of cycles. I suspect some threading related bug but I have forgotten how to do source level debugging (in lldb) in library code. Not sure that will help it if is a concurrency bug.... The test was run on the same 2 pi3 boards for both 15.0 and 15.1 install images. The first optional arg is for N (default 0) µseconds sleep between inet_ntop calls. The second arg is to advance to next line after every N (default 10000) cycles.
Is it length based? eg if you set the ipv6 address to something much shorter, shorter than an ipv4 test address, does it still crash? (random shot in the dark - i wonder if it's a jemalloc update problem? was jemalloc updated from 15.0 -> 15.1?)
(In reply to Adrian Chadd from comment #2) > The test program fails after random number of cycles. Useful for others that might try to test would be to give them an idea of the maximum number of cycles observed to get a failure in each failure context. It gives a clue about when to stop, such as a few times the currently known maximum observed.
(In reply to Adrian Chadd from comment #3) There are 2 jemalloc commits in releng/15.1 that are not in releng/15.0 (shown here for when merged to stable/15): Fri, 13 Mar 2026 git: 3abef030d31a - stable/15 - Set errno to ENOMEM on rallocx() OOM failures Brooks Davis git: b4d8d9bde083 - stable/15 - rallocx path: only set errno on the realloc case. Brooks Davis
Note that my test program doesn't fail on pi4 (on 15.0 or 15.1), doesn't fail on pi3 (15.0) but fails on pi3 with 15.1. Just compiled statically thinking may be I can debug it more easily. And... it doesn't fail!
My guess would be some form of memory corruption, since the faulting line is: #0 vsprintf (str=0xffffffffd432, fmt=0x40c484f5, ap=...) at /usr/src/lib/libc/stdio/vsprintf.c:67 is unaligned access out of the question. Could you please provide the full GDB log? Including the faulting PC, the disassembly of the faulting function and a register dump. Thanks
> Is it length based? eg if you set the ipv6 address to > something much shorter, shorter than an ipv4 test address, > does it still crash? It still crashes on "2::2" > Useful for others that might try to test would be to give > them an idea of the maximum number of cycles observed to > get a failure in each failure context. It gives a clue > about when to stop, such as a few times the currently > known maximum observed. It crashes right away. The big delay is writing out the core dump.
lldb output included below (for the ntoptest.c program -- see the attachement to comment 2). The program fails after a random number of cycles and the fault addr is different (see the two runs below). $ cc -g ntoptest.c $ lldb a.out (lldb) target create "a.out" Current executable set to '/usr/remotes/gitlab/NTPsec/ntpsec/a.out' (aarch64). (lldb) run Process 5438 launched: '/usr/remotes/gitlab/NTPsec/ntpsec/a.out' (aarch64) 10000 20000 2477Process 5438 stopped * thread #1, name = 'a.out', stop reason = signal SIGSEGV: invalid permissions f or mapped object (fault address: 0x81e01220) frame #0: 0x0000000081d3c004 libc.so.7`vsprintf [inlined] __get_locale at xl ocale_private.h:0 Note: this address is compiler-generated code in function __get_locale that has no source code associated with it. (lldb) bt * thread #1, name = 'a.out', stop reason = signal SIGSEGV: invalid permissions f or mapped object (fault address: 0x81e01220) * frame #0: 0x0000000081d3c004 libc.so.7`vsprintf [inlined] __get_locale at xl ocale_private.h:0 frame #1: 0x0000000081d3c004 libc.so.7`vsprintf(str="", fmt="%x", ap=(__stac k = 0x0000000080da95a0, __gr_top = 0x0000000080da9530, __vr_top = 0x0000000080da 9500, __gr_offs = -48, __vr_offs = -128)) at vsprintf.c:67:25 frame #2: 0x0000000081d31090 libc.so.7`sprintf(str=<unavailable>, fmt=<unava ilable>) at sprintf.c:55:8 frame #3: 0x0000000081cdecf4 libc.so.7`__inet_ntop [inlined] inet_ntop6(src= "&", dst=<unavailable>, size=<unavailable>) at inet_ntop.c:174:9 frame #4: 0x0000000081cdeb8c libc.so.7`__inet_ntop(af=<unavailable>, src=0x0 000000080da9690, dst="2600:1700:a460:4b60::1:2", size=46) at inet_ntop.c:59:11 frame #5: 0x0000000000210864 a.out`main(c=1, v=0x0000000080da9748) at ntopte st.c:23:8 frame #6: 0x0000000081ca3020 libc.so.7`__libc_start1(argc=1, argv=0x00000000 80da9748, env=0x0000000080da9758, cleanup=<unavailable>, mainX=(a.out`main at nt optest.c:11)) at libc_start1.c:180:7 frame #7: 0x0000000000210774 a.out`_start at crt1_s.S:62 (lldb) run There is a running process, kill it and restart?: [Y/n] Process 5438 exited with status = 9 (0x00000009) killed Process 5464 launched: '/usr/remotes/gitlab/NTPsec/ntpsec/a.out' (aarch64) 3Process 5464 stopped * thread #1, name = 'a.out', stop reason = signal SIGSEGV: invalid permissions for mapped object (fault address: 0x82801220) frame #0: 0x000000008273c004 libc.so.7`vsprintf [inlined] __get_locale at xlocale_private.h:0 Note: this address is compiler-generated code in function __get_locale that has no source code associated with it. (lldb) bt * thread #1, name = 'a.out', stop reason = signal SIGSEGV: invalid permissions for mapped object (fault address: 0x82801220) * frame #0: 0x000000008273c004 libc.so.7`vsprintf [inlined] __get_locale at xlocale_private.h:0 frame #1: 0x000000008273c004 libc.so.7`vsprintf(str="", fmt="%x", ap=(__stack = 0x0000000080303ab0, __gr_top = 0x0000000080303a40, __vr_top = 0x0000000080303a10, __gr_offs = -48, __vr_offs = -128)) at vsprintf.c:67:25 frame #2: 0x0000000082731090 libc.so.7`sprintf(str=<unavailable>, fmt=<unavailable>) at sprintf.c:55:8 frame #3: 0x00000000826decf4 libc.so.7`__inet_ntop [inlined] inet_ntop6(src="&", dst=<unavailable>, size=<unavailable>) at inet_ntop.c:174:9 frame #4: 0x00000000826deb8c libc.so.7`__inet_ntop(af=<unavailable>, src=0x0000000080303ba0, dst="2600:1700:a460:4b60::1:2", size=46) at inet_ntop.c:59:11 frame #5: 0x0000000000210864 a.out`main(c=1, v=0x0000000080303c60) at ntoptest.c:23:8 frame #6: 0x00000000826a3020 libc.so.7`__libc_start1(argc=1, argv=0x0000000080303c60, env=0x0000000080303c70, cleanup=<unavailable>, mainX=(a.out`main at ntoptest.c:11)) at libc_start1.c:180:7 frame #7: 0x0000000000210774 a.out`_start at crt1_s.S:62
Sorry, but SSH access is a last resort choice. The unknown environment always adds another dimension to the puzzle. However, "signal SIGSEGV: invalid permissions for mapped object (fault address: 0x82801220)" sounds like a game changer. Please post the output of these commands (I know almost nothing about LLDB, only GDB syntax). - 'bt full' - 'disassemble' - 'info registers' -'info proc mappings' and log from GDB session. As you wrote that the faulting address is not stable, all these outputs must be from one session. Thanks.
So the code in question is the inet_ntop6() function and it likely seems to just be a simple buffer overflow. static const char * inet_ntop6(const u_char *src, char *dst, socklen_t size) { /* * Note that int32_t and int16_t need only be "at least" large enough * to contain a value of the specified size. On some systems, like * Crays, there is no such thing as an integer variable with 16 bits. * Keep this in mind if you think this function should have been coded * to use pointer overlays. All the world's not a VAX. */ char tmp[sizeof "ffff:ffff:ffff:ffff:ffff:ffff:255.255.255.255"], *tp; .... /* * Format the result. */ tp = tmp; for (i = 0; i < (NS_IN6ADDRSZ / NS_INT16SZ); i++) { /* Are we inside the best run of 0x00's? */ if (best.base != -1 && i >= best.base && i < (best.base + best.len)) { if (i == best.base) *tp++ = ':'; continue; } /* Are we following an initial run of 0x00s or any real hex? */ if (i != 0) *tp++ = ':'; /* Is this address an encapsulated IPv4? */ if (i == 6 && best.base == 0 && (best.len == 6 || (best.len == 7 && words[7] != 0x0001) || (best.len == 5 && words[5] == 0xffff))) { if (!inet_ntop4(src+12, tp, sizeof tmp - (tp - tmp))) { errno = ENOSPC; return (NULL); } tp += strlen(tp); break; } tp += sprintf(tp, "%x", words[i]); } /* Was it a trailing run of 0x00's? */ if (best.base != -1 && (best.base + best.len) == (NS_IN6ADDRSZ / NS_INT16SZ)) *tp++ = ':'; *tp++ = '\0'; The crash is inside of the call to sprintf(), and my guess it is overflowing tmp. The crash being deterministic or not is probably just an accident on where `tmp` is allocated at link time (e.g. is it at the end of .bss or not). The size of tmp seems ok (it matches INET6_ADDRSTRLEN). I wonder if for some reason word contains a value larger than 0xffff so the %x is outputting more than 4 characters. It would probably be useful in a debugger to go up to the frame for inet_ntop6 and see what is in tmp. It would also be good to compare the current value of tp to tmp.
The address of 'tmp' is from the previous log_. "#0: vsprintf (str=0xFFFFFFFFD432, fmt=0x40C484F5, ap=...)". However, the faulting address is in the 0x81E01220 range (not stable, but in the same range). Don't ask me what the 0x8000_0000 range is. The standard memory layout of the testing program is: (gdb) info proc mappings process 78249 Mapped address spaces: Start Addr End Addr Size Offset Flags File 0x200000 0x201000 0x1000 0x0 r-- CN-- /root/bbb/b 0x210000 0x211000 0x1000 0x0 r-x CN-- /root/bbb/b 0x220000 0x221000 0x1000 0x0 r-- ---- 0x230000 0x231000 0x1000 0x0 rw- ---- 0x40230000 0x40236000 0x6000 0x0 r-- CN-- /libexec/ld-elf.so.1 0x40245000 0x4025d000 0x18000 0x5000 r-x C--- /libexec/ld-elf.so.1 0x4026c000 0x4026d000 0x1000 0x1c000 r-- C--- /libexec/ld-elf.so.1 0x4026d000 0x4026e000 0x1000 0x0 r-- ---- 0x4027d000 0x402a0000 0x23000 0x0 rw- ---- 0x402a0000 0x402b4000 0x14000 0x0 r-- CN-- /lib/libsys.so.7 0x402b4000 0x402c3000 0xf000 0x14000 --- CN-- 0x402c3000 0x402c8000 0x5000 0x13000 r-x CN-- /lib/libsys.so.7 0x402c8000 0x402d7000 0xf000 0x28000 --- CN-- 0x402d7000 0x402d8000 0x1000 0x17000 r-- C--- /lib/libsys.so.7 0x402d8000 0x402e7000 0xf000 0x38000 --- CN-- 0x402e7000 0x402e8000 0x1000 0x17000 rw- C--- /lib/libsys.so.7 0x402e8000 0x4030c000 0x24000 0x0 rw- ---- 0x40400000 0x4049d000 0x9d000 0x0 r-- CN-- /lib/libc.so.7 0x4049d000 0x404ac000 0xf000 0x9d000 --- CN-- 0x404ac000 0x4060e000 0x162000 0x9c000 r-x CN-- /lib/libc.so.7 0x4060e000 0x4061d000 0xf000 0x20e000 --- CN-- 0x4061d000 0x40628000 0xb000 0x1fd000 r-- C--- /lib/libc.so.7 0x40628000 0x40637000 0xf000 0x228000 --- CN-- 0x40637000 0x4063e000 0x7000 0x207000 rw- C--- /lib/libc.so.7 0x4063e000 0x4085a000 0x21c000 0x0 rw- ---- 0x40a00000 0x41200000 0x800000 0x0 rw- ---- 0xffffbffff000 0xfffffffdf000 0x3ffe0000 0x70003 --- ---- 0xfffffffdf000 0xfffffffff000 0x20000 0x0 rw- ---D 0xfffffffff000 0x1000000000000 0x1000 0x0 r-x ----
(In reply to Michal Meloun from comment #12) Oh, hmm, I had thought tmp was static for some reason, it is indeed allocated on the stack. However, if you look at the lldb dump I was commenting on, the 8000_0000 range is indeed the stack (see how the ap members which are stack pointers are in that general range). It would be useful to provide a full GDB log (not LLDB) similar to comment 9.
Is there a way tell jemalloc to a) clear memory on allocation b) fill newly allocated area with a known pattern? is there a way to print every allocation/deallocation *without* the printing itself allocating anything? While it fails after a random number of cycles, it always fails in the same place. tp and tmp seem fine. With /usr/src mounted, I can glean a little more info. Note that while x9 contains 0x0000000083a00000 and the instruction is 0x8393c004: ldr x9, [x9, #0x220] The *fault address* is 0x83a01220 (exactly 0x1000 more) where nothing is mapped. $ lldb ./a.out (lldb) target create "./a.out" Current executable set to '/usr/remotes/gitlab/NTPsec/ntpsec/a.out' (aarch64). (lldb) run Process 8392 launched: '/usr/remotes/gitlab/NTPsec/ntpsec/a.out' (aarch64) 10000 1273Process 8392 stopped * thread #1, name = 'a.out', stop reason = signal SIGSEGV: invalid permissions for mapped object (fault address: 0x83a01220) frame #0: 0x000000008393c004 libc.so.7`vsprintf [inlined] __get_locale at xlocale_private.h:0 62 "XLC_COLLATE doesn't match the LC_COLLATE value."); 63 _Static_assert(XLC_CTYPE == LC_CTYPE - 1, 64 "XLC_CTYPE doesn't match the LC_CTYPE value."); -> 65 _Static_assert(XLC_MONETARY == LC_MONETARY - 1, 66 "XLC_MONETARY doesn't match the LC_MONETARY value."); 67 _Static_assert(XLC_NUMERIC == LC_NUMERIC - 1, 68 "XLC_NUMERIC doesn't match the LC_NUMERIC value."); Note: this address is compiler-generated code in function __get_locale that has no source code associated with it. (lldb) re r General Purpose Registers: x0 = 0x0000000080ee9ba2 x1 = 0x00000000838484f5 x2 = 0x0000000080ee9af0 x3 = 0x0000000000000000 x4 = 0x0000000080ee8f70 x5 = 0x0000000080ee9ba1 x6 = 0x0000000030303731 x7 = 0x0000000000000020 x8 = 0x0000000000000000 x9 = 0x0000000083a00000 x10 = 0x0000000000000000 x11 = 0x00007450bfc1f010 x12 = 0x0000000000000004 x13 = 0x0000000000000000 x14 = 0x0000000000000010 x15 = 0x0000000000000000 x16 = 0x0000000083a16610 x17 = 0x000000008393bf60 libc.so.7`vsprintf at vsprintf.c:66 x18 = 0x0000000000001000 x19 = 0x00000000838484f5 x20 = 0x0000000080ee9ba2 x21 = 0x0000000080ee9c50 x22 = 0x0000000080ee9ba2 x23 = 0x00000000838484f5 x24 = 0x0000000000000002 x25 = 0x0000000000000006 x26 = 0x0000000000000004 x27 = 0x0000000000000004 x28 = 0x0000000000000002 fp = 0x0000000080ee9a10 lr = 0x0000000083931090 libc.so.7`sprintf + 116 at sprintf.c sp = 0x0000000080ee9890 pc = 0x000000008393c004 libc.so.7`vsprintf + 164 [inlined] __get_locale at xlocale_private.h libc.so.7`vsprintf + 164 at vsprintf.c:67:25 cpsr = 0x60000200 (lldb) bt all * thread #1, name = 'a.out', stop reason = signal SIGSEGV: invalid permissions for mapped object (fault address: 0x83a01220) * frame #0: 0x000000008393c004 libc.so.7`vsprintf [inlined] __get_locale at xlocale_private.h:0 frame #1: 0x000000008393c004 libc.so.7`vsprintf(str="", fmt="%x", ap=(__stack = 0x0000000080ee9b60, __gr_top = 0x0000000080ee9af0, __vr_top = 0x0000000080ee9ac0, __gr_offs = -48, __vr_offs = -128)) at vsprintf.c:67:25 frame #2: 0x0000000083931090 libc.so.7`sprintf(str=<unavailable>, fmt=<unavailable>) at sprintf.c:55:8 frame #3: 0x00000000838decf4 libc.so.7`__inet_ntop [inlined] inet_ntop6(src="&", dst=<unavailable>, size=<unavailable>) at inet_ntop.c:174:9 frame #4: 0x00000000838deb8c libc.so.7`__inet_ntop(af=<unavailable>, src=0x0000000080ee9c50, dst="2600:1700:a460:4b60::1:2", size=46) at inet_ntop.c:59:11 frame #5: 0x0000000000210864 a.out`main(c=1, v=0x0000000080ee9d08) at ntoptest.c:23:8 frame #6: 0x00000000838a3020 libc.so.7`__libc_start1(argc=1, argv=0x0000000080ee9d08, env=0x0000000080ee9d18, cleanup=<unavailable>, mainX=(a.out`main at ntoptest.c:11)) at libc_start1.c:180:7 frame #7: 0x0000000000210774 a.out`_start at crt1_s.S:62 (lldb) x/10i 0x000000008393c004 -> 0x8393c004: ldr x9, [x9, #0x220] 0x8393c008: ccmp w8, #0x0, #0x4, ne 0x8393c00c: mov w8, #0xffff ; =65535 0x8393c010: stur q0, [sp, #0x58] 0x8393c014: stur q0, [sp, #0x48] 0x8393c018: csel x21, x9, x10, eq 0x8393c01c: strh w8, [sp, #0x52] 0x8393c020: bl 0x839e3320 ; symbol stub for: __error 0x8393c024: ldr w2, [x0] 0x8393c028: cbz x21, 0x8393c040 ; <+224> [inlined] vsprintf_l + 36 at xlocale_private.h (lldb) re r x9 x9 = 0x0000000083a00000 (lldb) x 0x0000000083a00220 0x83a00220: f0 28 a1 83 00 00 00 00 00 00 00 00 00 00 00 00 .(.............. 0x83a00230: a4 b7 8c 83 00 00 00 00 c8 b8 8c 83 00 00 00 00 ................ (lldb) x 0x0000000083a01220 error: memory read failed for 0x83a01200 (lldb) f 3 frame #3: 0x00000000838decf4 libc.so.7`__inet_ntop [inlined] inet_ntop6(src="&", dst=<unavailable>, size=<unavailable>) at inet_ntop.c:174:9 171 tp += strlen(tp); 172 break; 173 } -> 174 tp += sprintf(tp, "%x", words[i]); 175 } 176 /* Was it a trailing run of 0x00's? */ 177 if (best.base != -1 && (best.base + best.len) == (lldb) p tmp (char[46]) "2600:1700:\0\0\0\0\0\00" (lldb) p &tmp[0] (char *) 0x0000000080ee9b98 "2600:1700:" (lldb) p tp (char *) 0x0000000080ee9ba2 "" (lldb) p tp-tmp (long) 10
The jemalloc option can be changed by creating a symbolic link ln -s 'junk:true,abort:true' /etc/malloc.conf As you can see, LLDB does not give consistent results. If x9 is 0x83a00000, the 'ldr x9, [x9, #0x220]' cannot fault with the fault address 0x83a01220. So please use GDB and provide the requested output, it's practically impossible to continue without it. 'info proc mappings' is absolutely essential.
> As you can see, LLDB does not give consistent results. > If x9 is 0x83a00000, the 'ldr x9, [x9, #0x220]' cannot fault with the fault > address 0x83a01220. lldb is included, gdb has to be installed. And it can give you the same info. at any rate gdb produces exactly the same info (the fault addr is 0x1000 off). And feel free to talk in gdb-ese, I will translate to lldb-ese :-) $ gdb ./a.out ... Reading symbols from ./a.out... (gdb) run Starting program: /usr/remotes/gitlab/NTPsec/ntpsec/a.out Program received signal SIGSEGV, Segmentation fault. Invalid permissions for mapped object. vsprintf (str=0xffffffffe3ad "1:2", fmt=0x404484f5 "%x", ap=...) at /usr/src/lib/libc/stdio/vsprintf.c:67 warning: Source file is more recent than executable. 67 return vsprintf_l(str, __get_locale(), fmt, ap); (gdb) p/x $_siginfo.si_addr $1 = 0x40601220 (gdb) p/i $pc Format letter "i" is meaningless in "print" command. (gdb) x/i $pc => 0x4053c004 <vsprintf+164>: ldr x9, [x9, #544] (gdb) p/x $x9+544 $2 = 0x40600220 Update: I turned off ASLR completely and now it reliably dies with SIGSEGV at the same address but also after a random number of cycles. I played with various MALLOC_CONF options but no change. To summarize - It doesn't fail on either pi3 or pi4 under FreeBSD 15.0. - It doesn't fail on pi4 but *fails* on pi3 under FreeBSD 15.1. - It fails after a random number of cycles. - When it fails, *always* fails in the same place (__get_locale()). - The failure is when trying to read from an unmapped area (so SIGSEGV). - It only fails when using *shared* libs. - When compiled *statically* it doesn't fail. - At the failing instruction x9 contains a text address (or linkage) - The instruction is "ldr x9, [x9, 0x220]". - The fault addr always *x9+0x220 + 0x1000. Coincidence? - There are no concurrent threads running. - There is very little else going on. No undervoltage. Temp @ 33ºC. Because a static binary doesn't fail but a dynamic one does, this can be due to a different mmap behavior or malloc or scheduling (given a variable number of cycles) or just that something else changed that hides the behavior. Setting a breakpoint right before the faulting instruction seems to kill the run. Any other hypothesis I should try? Looks like I may have to grok the diff of 15.0 and 15.1 in lib/libc and may be more. Thanks for the malloc conf help.
Hey, wait! This isn't the test program from Adrian's attachment!
The same program. If you are referring to gdb complaining warning: Source file is more recent than executable. I am not sure why. One possibility is that I did a git worktree add ../15.1 releng/15.1 and nfs mounted on pi3. git doesn't restore file dates by default and gdb noticed the shared libs (from installation) are older. To see if scheduling may be affecting this, I changed kern.hz to 10 (from 1000) on reboot. The theory being the program will run much longer before scheduling affects it at lower hz. Now the program seems to run much longer on average. For example $ ./a.out ... 340000 34[2] 2459 segmentation fault ./a.out I never saw such high counts before at 1000 hz. Returning hz to 1000, the highest count I saw after 10s of repetitions was 70000or so, the average being more like 20000. Quite strange but at best circumstantial evidence.
oki, I just need to make sure we're talking about the same binary. Anyway, I cannot continue without an output from the above four commands (from GDB).
#include <stdio.h> char buf[20]; int main(int c, char**v) { for (int count = 0;;count++) sprintf(buf, "%x", 12345678); } - it works on pi3 (15.1) if compiled statically. - it fails on pi3 (15.1) if compiled w/ dynamic libs. - it dies exactly as before: same fault, same addr, after random cycles. - process map before and after the fault is the same. - setting a bkpt at the faulting pc (but *while* it is working fine) results in segfault right away once continued. This clearly shows the problem is in 15.0-15.1 changes. diff of 15.0 and 15.1 lib/libc/ shows quite a lot of differences, including introduction of "optimized" arm64 assembly language functions. Are there exhaustive tests for these functions that can be run on a pi3? Beyond me to trawl though these changes.
Ugh. Missed the first line: This is a far simpler test with the same symptom.
I think this is triggered by Cortex-A53 erratum 843419. We were somehow unlucky to generate code that would trigger it.
from PR 296395: 0x000000004053c000 <+160>: cmp x10, #0x0 => 0x000000004053c004 <+164>: ldr x9, [x9, #544] 0x000000004053c008 <+168>: ccmp w8, #0x0, #0x4, ne // ne = any the faulting instruction is not in 0xFF8 or 0xFFC.
Oups, sorry! The 'adrp' is in the affected range.
At the crash site: 0x4053bff8 <vsprintf+152>: adrp x9, 0x40600000 0x4053bffc <vsprintf+156>: stur q0, [sp, #104] 0x4053c000 <vsprintf+160>: cmp x10, #0x0 => 0x4053c004 <vsprintf+164>: ldr x9, [x9, #544] 0x4053c008 <vsprintf+168>: ccmp w8, #0x0, #0x4, ne // ne = any 0x4053c00c <vsprintf+172>: mov w8, #0xffff // #65535 0x4053c010 <vsprintf+176>: stur q0, [sp, #88] 0x4053c014 <vsprintf+180>: stur q0, [sp, #72] The erratum says: 1) An ADRP instruction, which writes to a register Rn. • This instruction must be located in memory at an address where the bottom 12 bits are equal to 0xFF8 or 0xFFC. 2) A load or store instruction: • This can be: • A single register load or store, of either integer or vector registers • Or an STP or STNP, of either integer or vector registers • Or an Advanced SIMD ST1 store instruction. • This must not write to Rn. • This may optionally read Rn (either as an address register or a data register to be stored). • If a load, this must access a region of memory which crosses a 64-bit boundary. The "must not write to Rn" condition is not satisfied....
We have sequence 1: 1) An adrp to x9 where the bottom 12 bits of the address are 0xff8 2) A store that doesn't write to x9 & crosses a 128-bit boundary 3) Another instruction that's not a branch & doesn't write to x9 4) A load using x9 as the base address It is instruction 4 that is the faulting instruction in the sequence.
I verified that 15.0 vsprintf has this code fragment at a different address that doesn't trigger this chip bug. And -static on 15.1 mangles the code enough that this chip bug is avoided. Sounds like --fix-cortex-a53-843419 should be added ld flags at least while building libraries and this bug is critical enough that a 15.1 patch should be released. Ideally this is verified where official binaries are built, using identical toolchain. I can try cross-compiling this locally but no guarantees I will get the same results.
Apparently, clang 6 or later automatically supplies lld with the command line option when the triple targeted is specifically: aarch64-linux-android That reportedly was because vintages of the Cortex-A53 with the problem were common on android devices.
In an old aarch64 stable/15 that I had around the failure happens. In a modern stable/15 that I have it does not. The code generation need not match in the problem spot: llvm19 based for old and llvm21 based for modern.
> In a modern stable/15 that I have it does not. The code generation > need not match in the problem spot: llvm19 based for old and > llvm21 based for modern. Not sure what you're trying to say here.
I also do not get the problem under: # freebsd-version -u 15.1-RELEASE-p1 Just under from before the recent patches.
(In reply to Bakul Shah from comment #30) The problem does not occur for stable/15 built with its llvm21 based clang. (Nor does the problem occur for 15.0-RELEASE-p1 which updated libc.so.7 .) Not much has to change to make it unlikely that a libc.so.7 rebuild will not happen to get the problem.
I need to rerun those tests: wrong context wired up.
(In reply to Mark Millard from comment #31) Ignore comment #31 (and the reference to it in command #32) With the right drive connected to the RPi3 I got: # freebsd-version -u 15.1-RELEASE-p1 # cc -v FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2) Target: aarch64-unknown-freebsd15.1 Thread model: posix InstalledDir: /usr/bin # gdb a.out . . . Reading symbols from a.out... (gdb) run Starting program: /root/c_tests/a.out Program received signal SIGSEGV, Segmentation fault. Invalid permissions for mapped object. vsprintf (str=0x230750 <buf> "bc614e", fmt=0x200490 "%x", ap=...) at /usr/src/lib/libc/stdio/vsprintf.c:67 67 return vsprintf_l(str, __get_locale(), fmt, ap); (gdb) bt #0 vsprintf (str=0x230750 <buf> "bc614e", fmt=0x200490 "%x", ap=...) at /usr/src/lib/libc/stdio/vsprintf.c:67 #1 0x0000000020531090 in sprintf (str=str@entry=0x230750 <buf> "bc614e", fmt=fmt@entry=0x200490 "%x") at /usr/src/lib/libc/stdio/sprintf.c:55 #2 0x0000000000210570 in main (c=<optimized out>, v=<optimized out>) at RPi3-15p1R-failure-non-static.c:9 So 15.1-RELEASE-p1 does have the problem still, like 15.1-RELEASE did. But for modern stable/15 : # freebsd-version -u 15.1-STABLE # cc -v FreeBSD clang version 21.1.8 (https://github.com/llvm/llvm-project.git llvmorg-21.1.8-0-g2078da43e25a) Target: aarch64-unknown-freebsd15.1 Thread model: posix InstalledDir: /usr/bin # gdb a.out . . . (gdb) run Starting program: /root/c_tests/a.out [Inferior 1 (process 2903) exited normally] (My old stable/15 snapshot from back in llvm19 based days did show the problem.)
You have demonstrated that stable-15 doesn't show this problem -- but that could be because 1. the --fix-cortex-a53-843419 flag is given to ld or 2. just that the code layout *happens* to not tickle this chip-bug. Which is it? I am arguing that as long as FreeBSD supports cortex-a53 based systems, the linker flag be the default so that it ensures the chip bug is not tickled (including for shared libs).
(In reply to Bakul Shah from comment #35) I'm not contradicting that the code pattern should be avoided. But you can not as easily test an environment that does not have the problem in its code in the first place: rare code patterns might still be possible. Another option for where to enforce the code generation behavior is to have clang and/or lld automatically have the option as a default for aarch64-*-freebsd* triples --including for the devel/llvm* ports and their packages, much like for aarch64-linux-android . lang/rust could be at issue as well (llvm based, with its own internal one that can be used). Release Engineering might have its own opinions on if 14.[34]-RELEASE , 1[45]-STABLE ,and 15.0-RELEASE should also have (some?) system code regenerated beyond the details of controlling code generation. It is very clear that 15.1-RELEASE should have some system code updated that is not inside a compiler/linker. I've no clue how many other errata for what types of cores might have similar sorts of issues, not necessarily limited to aarch64.
outside of hand written assembly, this is likely incredible luck, bad/changed flags or bugs in the toolchain. I believe that is the thing that needs to be established. How do we establish the compiler/linker flags being used in 19 and 21?
I added LDFLAGS+=--fix-cortex-a53-843419 for makeworld (cross-compiled on an amd64 system). My test program linked to this newly created libc.so.7 doesn't crash. 0x4053bfe4 <vsprintf+152>: adrp x9, 0x40600000 0x4053bfe8 <vsprintf+156>: stur q0, [sp, #104] 0x4053bfec <vsprintf+160>: cmp x10, #0x0 0x4053bff0 <vsprintf+164>: ldr x9, [x9, #544] 0x4053bff4 <vsprintf+168>: ccmp w8, #0x0, #0x4, ne // ne = any 0x4053bff8 <vsprintf+172>: mov w8, #0xffff // #65535 [Notice inst. addr of adrp] $ size save/libc.so.7 /lib/libc.so.7 text data bss dec hex filename 1927732 65456 2211833 4205021 4029dd save/libc.so.7 -- copy 15.1 libc.so.7 1927732 65456 2211833 4205021 4029dd /lib/libc.so.7 -- locally built $ readelf -p .comment save/libc.so.7 String dump of section '.comment': [ 1] Linker: LLD 19.1.7 (FreeBSD llvmorg-19.1.7-0-gcd708029e0b2-1500001) (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2) [ 8f] FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2) $ readelf -p .comment /lib/libc.so.7 String dump of section '.comment': [ 1] Linker: LLD 19.1.7 (FreeBSD llvmorg-19.1.7-0-gcd708029e0b2-1500001) (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2) [ 8f] FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2) So this says both were built using ld and clang built from the same source. I also noticed /usr/obj/.../arm64.aarch64/lib/clang/libllvm/Target/ARM/ARMFixCortexA57AES1742098Pass.pico while no such binary exists for my current build for arm64. [Haven't tried without makeworld the LDFLAGS addition].
(In reply to Adrian Chadd from comment #37) FreeBSD was just lucky to have avoided this so far. If you websearch for "Cortex-A53 erratum 843419" you will see quite a few hits, at least one for go as well. I should note that all the signal 11 crashes stopped with the newly built libc.
(In reply to Bakul Shah from comment #39) So it looks like it was working by luck, not by design. I haven't found anything that enabled that particular workaround on FreeBSD (only on android and linux kernel / distribution changes.) Chances are we need to audit the various arm workaround flags and enable things by default.
FYI, on aarch64 for 15.1-RELEASE (llvm 19 based): # ld.lld --help | grep -i '\-fix-' --fix-cortex-a53-843419 Apply fixes for AArch64 Cortex-A53 erratum 843419 --fix-cortex-a8 Apply fixes for ARM Cortex-A8 erratum 657417 The same list is shown on 15-STABLE (llvm 21 based): # ld.lld --help | grep -i '\-fix-' --fix-cortex-a53-843419 Apply fixes for AArch64 Cortex-A53 erratum 843419 --fix-cortex-a8 Apply fixes for ARM Cortex-A8 erratum 657417 Same for amd64 main (llvm 21 based): # ld.lld --help | grep -i '\-fix-' --fix-cortex-a53-843419 Apply fixes for AArch64 Cortex-A53 erratum 843419 --fix-cortex-a8 Apply fixes for ARM Cortex-A8 erratum 657417 (They are all from installations of upstream pkgbase materials, not personal builds.)
(In reply to Mark Millard from comment #41) The general type of judgments to be made here are: --fix-cortex-a8 Apply fixes for ARM Cortex-A8 erratum 657417 Does FreeBSD intend to target cortex-a8? Is the option worth the tradeoffs for all other aarch64 systems? --fix-cortex-a53-843419 Apply fixes for AArch64 Cortex-A53 erratum 843419 As I understand, Cortex-A53 revision r1p0 and later no longer involve erratum 843419. Is the option worth the tradeoffs for all other aarch64 systems, including those later Cortex-A53's? Another consideration might be: When specific cpus or the like are targeted by a FreeBSD build, and they are not one of the above, can the lld option be avoided? That might suggest not just using the target triple to enable the option(s).
(In reply to Mark Millard from comment #42) Dumb typo for Cortex-A8 context: it is ARMv7-A not aarch64. So, correcting . . . The general type of judgments to be made for cortex-a8 are: --fix-cortex-a8 Apply fixes for ARM Cortex-A8 erratum 657417 Does FreeBSD intend to target cortex-a8? Is the option worth the tradeoffs for all other armv7 systems? (Note: As I understand, a bunch of historical armv7 FreeBSD support may have been for Cortex-A8 based armv7 systems.)
As far as this bug is concerned, we know the cause. I am inclined to discuss what to do about it as well as the more general issues on the hackers or the arms mailing list.
*** Bug 296730 has been marked as a duplicate of this bug. ***
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f1e8b0ff4e13b52c5e41a5eea3e686410af3dd0b commit f1e8b0ff4e13b52c5e41a5eea3e686410af3dd0b Author: Andrew Turner <andrew@FreeBSD.org> AuthorDate: 2026-07-14 15:07:30 +0000 Commit: Andrew Turner <andrew@FreeBSD.org> CommitDate: 2026-07-14 15:47:15 +0000 bsd.cpu.mk: Add a workaround for erratum 843419 Add a workaround for the Arm Cortex-A53 erratum 843419. This has been targeted when the build is either unoptimised for any CPU/architecture or targets the Cortex-A53 or ARMv8.0 architecture. PR: 296240 PR: 296395 Reported by: Hal Murray <halmurray+freebsd@sonic.net> Reported by: Andreas Schuh <x55839@icloud.com> Reviewed by: cognet, mmel Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D58212 share/mk/bsd.cpu.mk | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
(In reply to commit-hook from comment #46) That commit should likely be MFC'd at some point.
(In reply to commit-hook from comment #46) The commit: URL: https://cgit.FreeBSD.org/src/commit/?id=f1e8b0ff4e13b52c5e41a5eea3e686410af3dd0b commit f1e8b0ff4e13b52c5e41a5eea3e686410af3dd0b Author: Andrew Turner <andrew@FreeBSD.org> AuthorDate: 2026-07-14 15:07:30 +0000 Commit: Andrew Turner <andrew@FreeBSD.org> CommitDate: 2026-07-14 15:47:15 +0000 bsd.cpu.mk: Add a workaround for erratum 843419 . . . is apparently making some port-package build fail with: "ld.lld: error: --fix-cortex-a53-843419 is only supported on AArch64" An example on ampere2 is: amd64-rust-bootstrap-1.97.1 lang/rust-bootstrap@amd64 build 0 ??? 06:35:04 https://pkg-status.freebsd.org/ampere2/data/main-arm64-default/pb5533b53ae5e_s297394e995e/logs/errors/amd64-rust-bootstrap-1.97.1.log Quoting: . . . It fails with the following output: Change Dir: '/wrkdirs/usr/ports/lang/rust-bootstrap/work-amd64/_build/x86_64-unknown-freebsd/llvm/build/CMakeFiles/CMakeScratch/TryCompile-N8ubzl' Run Build Command(s): /usr/local/bin/ninja -v cmTC_e0e9e [ 50% 1/2] /usr/local/freebsd-sysroot/amd64/bin/cc -ffunction-sections -fdata-sections -fPIC -m64 --target=x86_64-unknown-freebsd -pipe -fstack-protector-strong -fno-strict-aliasing -MD -MT CMakeFiles/cmTC_e0e9e.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_e0e9e.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_e0e9e.dir/testCCompiler.c.o -c /wrkdirs/usr/ports/lang/rust-bootstrap/work-amd64/_build/x86_64-unknown-freebsd/llvm/build/CMakeFiles/CMakeScratch/TryCompile-N8ubzl/testCCompiler.c [100% 2/2] : && /usr/local/freebsd-sysroot/amd64/bin/cc -ffunction-sections -fdata-sections -fPIC -m64 --target=x86_64-unknown-freebsd -pipe -fstack-protector-strong -fno-strict-aliasing -lz -Wl,--fix-cortex-a53-843419 CMakeFiles/cmTC_e0e9e.dir/testCCompiler.c.o -o cmTC_e0e9e && : FAILED: [code=1] cmTC_e0e9e : && /usr/local/freebsd-sysroot/amd64/bin/cc -ffunction-sections -fdata-sections -fPIC -m64 --target=x86_64-unknown-freebsd -pipe -fstack-protector-strong -fno-strict-aliasing -lz -Wl,--fix-cortex-a53-843419 CMakeFiles/cmTC_e0e9e.dir/testCCompiler.c.o -o cmTC_e0e9e && : ld.lld: error: --fix-cortex-a53-843419 is only supported on AArch64 cc: error: linker command failed with exit code 1 (use -v to see invocation) ninja: build stopped: subcommand failed.
*** Bug 297331 has been marked as a duplicate of this bug. ***