Bug 293536 - kgdb: source code location lookup works for kernel but not for module
Summary: kgdb: source code location lookup works for kernel but not for module
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2026-03-02 00:40 UTC by Martin Cracauer
Modified: 2026-03-03 15:10 UTC (History)
2 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Cracauer 2026-03-02 00:40:06 UTC 
tl;dr:

I have a bhyve kgdb situation, debugging a module. The source code is synced between guest and host, both located at /usr/src-debug. However, the debugger insists on using /usr/src for the module, although the correct /usr/src-debug is used for the kernel. Consider this backtrace:

#0  panic (fmt=0xffffffff83012022 "No cnp")
    at /usr/src-debug/sys/kern/kern_shutdown.c:886
#1  0xffffffff830116a0 in sysctl_rule (oidp=<optimized out>,
    arg1=0x0 <ugidfw_check>, arg2=-2097078112, req=0xffffffff830120a0)
    at /xcarb3/usr/src/sys/security/mac_bsdextended/mac_bsdextended.c:145
#2  0xffffffff83011714 in sysctl_rule (oidp=<optimized out>,
    arg1=<optimized out>, arg2=<optimized out>, req=0x40 <ugidfw_check+64>)
    at /xcarb3/usr/src/sys/security/mac_bsdextended/mac_bsdextended.c:154
#3  0xffffffff80f3c56b in mac_vnode_check_lookup_impl (
    cred=cred@entry=0xfffff8037415ed80, dvp=dvp@entry=0xfffff80001b0a1b8,
    cnp=0xfffffe00d5fa68f8, cnp@entry=0xfffffe00d5fa6d18)
    at /usr/src-debug/sys/security/mac/mac_vfs.c:576
#4  0xffffffff80cc5a7d in mac_vnode_check_lookup (cred=0xfffff8037415ed80,
    dvp=0xfffff80001b0a1b8, cnp=0xfffffe00d5fa6d18)
    at /usr/src-debug/sys/security/mac/mac_framework.h:565
#5  vfs_lookup (ndp=ndp@entry=0xfffffe00d5fa6c98)
    at /usr/src-debug/sys/kern/vfs_lookup.c:1311
#6  0xffffffff80cc472d in namei (ndp=ndp@entry=0xfffffe00d5fa6c98)
    at /usr/src-debug/sys/kern/vfs_lookup.c:738
#7  0xffffffff80cefd52 in vn_open_cred (ndp=ndp@entry=0xfffffe00d5fa6c98,
    flagp=flagp@entry=0xfffffe00d5fa6db4, cmode=cmode@entry=420,
--Type <RET> for more, q to quit, c to continue without paging--
    vn_open_flags=vn_open_flags@entry=16, cred=0xfffff8037415ed80, fp=0xfffff8000583a050) at /usr/src-debug/sys/kern/vfs_vnops.c:369
#8  0xffffffff80ce5747 in openatfp (td=0xfffff8000584b000, dirfd=dirfd@entry=-100, path=0x824cba112 "/var/run/utx.active", pathseg=pathseg@entry=UIO_USERSPACE, flags=1048577,
    mode=<optimized out>, fpp=0x0 <ugidfw_check>) at /usr/src-debug/sys/kern/vfs_syscalls.c:1263
#9  0xffffffff80ce5408 in kern_openat (td=0xffffffff83012022, dirfd=-100, path=0xffffffff830120a0 "FOO3: '%s'\n", pathseg=UIO_USERSPACE, flags=92581888, mode=-2111371672)
    at /usr/src-debug/sys/kern/vfs_syscalls.c:1351


As you can see /xcarb3/usr/src is used for the module (which is the unedited host source code). The correct /usr/src-debug is used for the kernel.

Inside, kernel.debug and mac_bsdextended.ko.debug both clearly reference /usr/src-debug if you inspect the files in a hexdump.

Longer story:
-------------

I am bringing things into play like this:
- source code is in /usr/src-debug on both guest and host
- guest compiles, installs and runs the modified kernel and module
- kernel.debug and mac_bsdextended.ko.debug are copied to /var/tmp on the host
- invocation:
kgdb /var/tmp/kernel.debug
add-symbol-file /var/tmp/mac_bsdextended.ko.debug
target remote localhost:1234
b panic

When starting kgdb says
Reading symbols from /var/tmp/kernel.debug...                                   
add symbol table from file "/var/tmp/mac_bsdextended.ko.debug"                  
warning: remote target does not support file transfer, attempting to access file
s from local filesystem.                                                        


Obviously that "from local filesystem" could screw me up, but it doesn't say that it actually loads something.
Comment 1 Martin Cracauer 2026-03-02 15:15:19 UTC 
Truss on kgdb shows that it is not satisfied with finding mac_bsdextended.ko.debug as specified in /var/tmp. It probes OS default places in addition. In my setup it find wrong versions eventually and seems to throw away the .ko.debug that I specified.

I don't think it should do that.

__realpathat(AT_FDCWD,"/var/tmp/mac_bsdextended.ko.debug","/var/tmp/mac_bsdextended.ko.debug",1024,0) = 0 (0x0)
open("/var/tmp/mac_bsdextended.ko.debug-gdb.gdb",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/usr/lib/debug/var/tmp/mac_bsdextended.ko.debug-gdb.gdb",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/debug/var/tmp/mac_bsdextended.ko.debug-gdb.gdb",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/xcarb3/usr/local/share/gdb/auto-load/var/tmp/mac_bsdextended.ko.debug-gdb.gdb",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
__realpathat(AT_FDCWD,"/var/tmp/mac_bsdextended.ko.debug","/var/tmp/mac_bsdextended.ko.debug",1024,0) = 0 (0x0)
open("/var/tmp/mac_bsdextended.ko.debug-gdb.py",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/usr/lib/debug/var/tmp/mac_bsdextended.ko.debug-gdb.py",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/debug/var/tmp/mac_bsdextended.ko.debug-gdb.py",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/xcarb3/usr/local/share/gdb/auto-load/var/tmp/mac_bsdextended.ko.debug-gdb.py",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
__realpathat(AT_FDCWD,"/usr/lib/debug/boot/kernel/mac_bsdextended.ko.debug","/usr/lib/debug/boot/kernel/mac_bsdextended.ko.debug",1024,0) = 0 (0x0)
open("/usr/lib/debug/boot/kernel/mac_bsdextended.ko.debug-gdb.gdb",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/usr/lib/debug/usr/lib/debug/boot/kernel/mac_bsdextended.ko.debug-gdb.gdb",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/debug/usr/lib/debug/boot/kernel/mac_bsdextended.ko.debug-gdb.gdb",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/xcarb3/usr/local/share/gdb/auto-load/usr/lib/debug/boot/kernel/mac_bsdextended.ko.debug-gdb.gdb",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
__realpathat(AT_FDCWD,"/usr/lib/debug/boot/kernel/mac_bsdextended.ko.debug","/usr/lib/debug/boot/kernel/mac_bsdextended.ko.debug",1024,0) = 0 (0x0)
open("/usr/lib/debug/boot/kernel/mac_bsdextended.ko.debug-gdb.py",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/usr/lib/debug/usr/lib/debug/boot/kernel/mac_bsdextended.ko.debug-gdb.py",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/debug/usr/lib/debug/boot/kernel/mac_bsdextended.ko.debug-gdb.py",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/xcarb3/usr/local/share/gdb/auto-load/usr/lib/debug/boot/kernel/mac_bsdextended.ko.debug-gdb.py",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
__realpathat(AT_FDCWD,"/boot/kernel/mac_bsdextended.ko","/boot/kernel/mac_bsdextended.ko",1024,0) = 0 (0x0)
open("/boot/kernel/mac_bsdextended.ko-gdb.gdb",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/usr/lib/debug/boot/kernel/mac_bsdextended.ko-gdb.gdb",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/usr/local/lib/debug/boot/kernel/mac_bsdextended.ko-gdb.gdb",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
open("/xcarb3/usr/local/share/gdb/auto-load/boot/kernel/mac_bsdextended.ko-gdb.gdb",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
__realpathat(AT_FDCWD,"/boot/kernel/mac_bsdextended.ko","/boot/kernel/mac_bsdextended.ko",1024,0) = 0 (0x0)
open("/boot/kernel/mac_bsdextended.ko-gdb.py",O_RDONLY|O_CLOEXEC,0666) ERR#2 'No such file or directory'
Comment 2 Ed Maste freebsd_committer freebsd_triage 2026-03-02 17:10:09 UTC 
This issue is in devel/gdb not the base system presumably?
Can you check lldb on the same context?
Comment 3 Martin Cracauer 2026-03-02 19:42:10 UTC 
(In reply to Ed Maste from comment #2)

I don't think lldb has source-level live kernel debugging yet?

I just tried, no source display.
Comment 4 Ed Maste freebsd_committer freebsd_triage 2026-03-02 19:46:34 UTC 
> I don't think lldb has source-level live kernel debugging yet?

It should, although it might have regressed. It should have support for /dev/mem, core files, and remote debugging via the GDB protocol.
Comment 5 Martin Cracauer 2026-03-02 22:13:30 UTC 
(In reply to Ed Maste from comment #4)

Well, according to a quick test it has remote debugging kernel support. It attaches, it can interrupt and continue and display memory.

However, it does not display any source code in backtrace, or even symbols in the backtrace for that matter.

This is on AMD64-current.
Comment 6 Martin Cracauer 2026-03-03 15:10:39 UTC 
Right. So first of all there is
  set debug separate-debug-file
which will show you what kgdb loads without having to resort to truss or ktrace.

It shows that 
  add-symbol-file /var/tmp/mac_bsdextended.ko.debug
never actually does anything.


If you add the symbol file before the guest loads the module then it is never associated with the addresses, backtraces in that module will neither have symbols nor source locations.


If on the other hand you add the symbol file after the guest loads the module then automatic loading from default locations has already taken place and the add-symbol-file does not override.


It seems that overriding existing debug files on add-symbol-file is one good way to go here. What do you think?