For some reason backtraces in crash dumps don't work for me anymore; this is arm64 Parallels guest: trasz@v3:~ % uname -a FreeBSD v3 15.0-CURRENT FreeBSD 15.0-CURRENT #0 main-n274011-0dab21248bc9: Wed Dec 4 19:00:01 UTC 2024 root@v3:/usr/obj/usr/home/trasz/git/freebsd/arm64.aarch64/sys/GENERIC arm64 trasz@v3:~ % doas kgdb /boot/kernel/kernel /var/crash/vmcore.last GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] Copyright (C) 2024 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "aarch64-portbld-freebsd15.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel... Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... Unread portion of the kernel message buffer: panic: kdb_sysctl_panic cpuid = 1 time = 1733474064 KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x38 vpanic() at vpanic+0x1a0 panic() at panic+0x48 kdb_sysctl_panic() at kdb_sysctl_panic+0x70 sysctl_root_handler_locked() at sysctl_root_handler_locked+0xe4 sysctl_root() at sysctl_root+0x1e0 userland_sysctl() at userland_sysctl+0x12c sys___sysctl() at sys___sysctl+0x84 do_el0_sync() at do_el0_sync+0x60c handle_el0_sync() at handle_el0_sync+0x4c --- exception, esr 0x56000000 Uptime: 4m24s Dumping 419 out of 8163 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81% Reading symbols from /boot/kernel/hwpmc.ko... Reading symbols from /usr/lib/debug//boot/kernel/hwpmc.ko.debug... Reading symbols from /boot/kernel/cc_dctcp.ko... Reading symbols from /usr/lib/debug//boot/kernel/cc_dctcp.ko.debug... Reading symbols from /boot/kernel/cc_cdg.ko... Reading symbols from /usr/lib/debug//boot/kernel/cc_cdg.ko.debug... Reading symbols from /boot/kernel/linux64.ko... Reading symbols from /usr/lib/debug//boot/kernel/linux64.ko.debug... Reading symbols from /boot/kernel/mqueuefs.ko... Reading symbols from /usr/lib/debug//boot/kernel/mqueuefs.ko.debug... Reading symbols from /boot/kernel/linux_common.ko... Reading symbols from /usr/lib/debug//boot/kernel/linux_common.ko.debug... Reading symbols from /boot/kernel/pty.ko... Reading symbols from /usr/lib/debug//boot/kernel/pty.ko.debug... Reading symbols from /boot/kernel/fdescfs.ko... Reading symbols from /usr/lib/debug//boot/kernel/fdescfs.ko.debug... Reading symbols from /boot/kernel/linprocfs.ko... Reading symbols from /usr/lib/debug//boot/kernel/linprocfs.ko.debug... Reading symbols from /boot/kernel/linsysfs.ko... Reading symbols from /usr/lib/debug//boot/kernel/linsysfs.ko.debug... Reading symbols from /boot/kernel/virtio_balloon.ko... Reading symbols from /usr/lib/debug//boot/kernel/virtio_balloon.ko.debug... Reading symbols from /boot/kernel/uhid.ko... Reading symbols from /usr/lib/debug//boot/kernel/uhid.ko.debug... Reading symbols from /boot/kernel/wmt.ko... Reading symbols from /usr/lib/debug//boot/kernel/wmt.ko.debug... Reading symbols from /boot/kernel/autofs.ko... Reading symbols from /usr/lib/debug//boot/kernel/autofs.ko.debug... Reading symbols from /boot/kernel/nullfs.ko... Reading symbols from /usr/lib/debug//boot/kernel/nullfs.ko.debug... 0xffff0000004b5aa8 in doadump (textdump=1) at /usr/home/trasz/git/freebsd/sys/kern/kern_shutdown.c:404 404 dump_savectx(); (kgdb) bt #0 0xffff0000004b5aa8 in doadump (textdump=1) at /usr/home/trasz/git/freebsd/sys/kern/kern_shutdown.c:404 #1 0x67fd0000004b5868 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) I've checked some other threads, and all the backtraces look like this. (kgdb) info frame Stack level 0, frame at 0xffff0000c00483c0: pc = 0xffff0000004b5aa8 in doadump (/usr/home/trasz/git/freebsd/sys/kern/kern_shutdown.c:404); saved pc = 0x67fd0000004b5868 called by frame at 0xffff0000c00483c0 source language c. Arglist at 0xffff0000c0048390, args: textdump=1 Locals at 0xffff0000c0048390, Previous frame's sp is 0xffff0000c00483c0 Saved registers: x19 at 0xffff0000c00483b8, x20 at 0xffff0000c00483b0, x21 at 0xffff0000c00483a0, x29 at 0xffff0000c0048390, x30 at 0xffff0000c0048398 (kgdb) up #1 0x67fd0000004b5868 in ?? () (kgdb) info frame Stack level 1, frame at 0xffff0000c00483c0: pc = 0x67fd0000004b5868; saved pc = <not saved> Outermost frame: previous frame identical to this frame (corrupt stack?) caller of frame at 0xffff0000c00483c0 Arglist at 0xffff0000c00483d0, args: Locals at 0xffff0000c00483d0, Previous frame's sp is 0xffff0000c00483c0 Why is saved pc not saved?
I can't reproduce this on my devkit: Reading symbols from /boot/kernel/zfs.ko... Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug... Reading symbols from /boot/kernel/mac_ntpd.ko... Reading symbols from /usr/lib/debug//boot/kernel/mac_ntpd.ko.debug... 0xffff0000004b5aac in doadump (textdump=textdump@entry=1) at /root/freebsd/sys/kern/kern_shutdown.c:404 warning: 404 /root/freebsd/sys/kern/kern_shutdown.c: No such file or directory (kgdb) bt #0 0xffff0000004b5aac in doadump (textdump=textdump@entry=1) at /root/freebsd/sys/kern/kern_shutdown.c:404 #1 0xffff0000004b586c in kern_reboot (howto=260) at /root/freebsd/sys/kern/kern_shutdown.c:524 #2 0xffff0000004b5dec in vpanic (fmt=<optimized out>, ap=...) at /root/freebsd/sys/kern/kern_shutdown.c:979 #3 0xffff0000004b5b90 in panic (fmt=0xffff000000cd5000 <sysctl___kern_racct_rctl_throttle_pct+104> "") at /root/freebsd/sys/kern/kern_shutdown.c:892 #4 0xffff000000505bbc in kdb_sysctl_panic (oidp=0xffff000000cdd7f8 <sysctl___debug_kdb_panic>, arg1=<optimized out>, arg2=<optimized out>, req=0xffff00010eb496e0) at /root/freebsd/sys/kern/subr_kdb.c:219 #5 0xffff0000004c7988 in sysctl_root_handler_locked (oid=oid@entry=0xffff000000cdd7f8 <sysctl___debug_kdb_panic>, arg1=arg1@entry=0x0, arg2=arg2@entry=0, req=req@entry=0xffff00010eb496e0, tracker=tracker@entry=0xffff00010eb49668) at /root/freebsd/sys/kern/kern_sysctl.c:199 #6 0xffff0000004c6ce8 in sysctl_root (oidp=<optimized out>, arg1=0x0, arg1@entry=0xffff00010eb497a0, arg2=0, arg2@entry=3, req=req@entry=0xffff00010eb496e0) at /root/freebsd/sys/kern/kern_sysctl.c:2405 #7 0xffff0000004c736c in userland_sysctl (td=td@entry=0xffff0001433032c0, name=name@entry=0xffff00010eb497a0, namelen=<optimized out>, old=<optimized out>, oldlenp=<optimized out>, inkernel=<optimized out>, inkernel@entry=246716288, new=<optimized out>, newlen=<optimized out>, retval=0xffff00010eb49798, flags=0) at /root/freebsd/sys/kern/kern_sysctl.c:2562 #8 0xffff0000004c7204 in sys___sysctl (td=0xffff0001433032c0, uap=0xffff0001433036c0) at /root/freebsd/sys/kern/kern_sysctl.c:2435 #9 0xffff00000088149c in syscallenter (td=0xffff0001433032c0) at /root/freebsd/sys/arm64/arm64/../../kern/subr_syscall.c:189 #10 svc_handler (td=0xffff0001433032c0, frame=<optimized out>) at /root/freebsd/sys/arm64/arm64/trap.c:198 #11 do_el0_sync (td=0xffff0001433032c0, frame=<optimized out>) at /root/freebsd/sys/arm64/arm64/trap.c:658 #12 <signal handler called> #13 0x0000764dd193503c in ?? () #14 0x0000764dc8cc1f94 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) jhb@ suggests that there might be a problem with the debug info file.
Nothing has changed in unwinding for kgdb in a long while, and the only frame unwinding that is custom in kgdb is for exception frames. Unwinding out of doadump() is a "normal" frame just like a userspace frame and should be using DWARF unwind info. The most likely regression here is an issue with the DWARF. Oh, is this with PAC? The PC value looks like the upper N bits are "wrong": #0 0xffff0000004b5aa8 in doadump (textdump=1) at /usr/home/trasz/git/freebsd/sys/kern/kern_shutdown.c:404 #1 0x67fd0000004b5868 in ?? () The real PC for frame 1 probably starts with 'ffff' instead of '67fd'. As a workaround you can disable PAC for now (not sure if that's a thing you can do currently). I'll try to see if I can reproduce.
Bingo! Adding "hw.pac.enable=0" to /boot/loader.conf makes the backtraces work again. I'll happily test patches if you can't reproduce locally.