Bug 282268
| Summary: | linker_load_module() panics with KASAN during post-panic reboot | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | Christos Margiolis <christos> | ||||
| Component: | kern | Assignee: | Mark Johnston <markj> | ||||
| Status: | Closed FIXED | ||||||
| Severity: | Affects Some People | CC: | markj | ||||
| Priority: | --- | Keywords: | crash, loader | ||||
| Version: | CURRENT | ||||||
| Hardware: | Any | ||||||
| OS: | Any | ||||||
| Attachments: |
|
||||||
Can you get a kernel dump? Which module is it trying to load? Not sure if this theory is correct, but I am noticing that the issue is more frequent when the panic was caused by a manual call to panic(9), as opposed to, say, a memory issue.
I sent you the dump file on Slack because it's too big to attach here. Seems like it's crashing when loading filemon(4):
(kgdb) bt
#0 __curthread () at /mnt/src/sys/amd64/include/pcpu_aux.h:57
#1 doadump (textdump=textdump@entry=0) at /mnt/src/sys/kern/kern_shutdown.c:404
#2 0xffffffff805a33b6 in db_dump (dummy=<optimized out>, dummy2=<optimized out>, dummy3=<optimized out>,
dummy4=<optimized out>) at /mnt/src/sys/ddb/db_command.c:595
#3 0xffffffff805a31eb in db_command (last_cmdp=<optimized out>, cmd_table=<optimized out>, dopager=true)
at /mnt/src/sys/ddb/db_command.c:508
#4 0xffffffff805a2bfd in db_command_loop () at /mnt/src/sys/ddb/db_command.c:555
#5 0xffffffff805a8979 in db_trap (type=<optimized out>, code=<optimized out>) at /mnt/src/sys/ddb/db_main.c:267
#6 0xffffffff81189485 in kdb_trap (type=3, code=code@entry=0, tf=tf@entry=0xfffffe0046ce0110)
at /mnt/src/sys/kern/subr_kdb.c:790
#7 0xffffffff81a201a8 in trap (frame=0xfffffe0046ce0110) at /mnt/src/sys/amd64/amd64/trap.c:606
#8 <signal handler called>
#9 kdb_enter (why=<optimized out>, msg=<optimized out>) at /mnt/src/sys/kern/subr_kdb.c:556
#10 0xffffffff810dde37 in vpanic (
fmt=fmt@entry=0xffffffff81c8f46f "ASan: Invalid access, %zu-byte %s at %#lx, %s(%x)",
ap=ap@entry=0xfffffe0046ce03e0) at /mnt/src/sys/kern/kern_shutdown.c:967
#11 0xffffffff810ddbd5 in panic (fmt=0xffffffff81c8f46f "ASan: Invalid access, %zu-byte %s at %#lx, %s(%x)")
at /mnt/src/sys/kern/kern_shutdown.c:892
#12 0xffffffff81160970 in kasan_report (addr=18446741875887136624, size=4, write=false, pc=<optimized out>,
code=<optimized out>) at /mnt/src/sys/kern/subr_asan.c:202
#13 0xffffffff8107f1f3 in linker_hints_lookup (
path=0xffffffff829c65e0 <linker_path> "/boot/kernel;/boot/modules;/boot/dtb;/boot/dtb/overlays",
modname=0xfffffe004a17ec00 "filemon", modnamelen=7, verinfo=0x0, pathlen=<optimized out>)
at /mnt/src/sys/kern/kern_linker.c:2046
#14 linker_search_module (modname=0xfffffe004a17ec00 "filemon", modnamelen=7, verinfo=0x0)
at /mnt/src/sys/kern/kern_linker.c:2134
#15 linker_load_module (kldname=kldname@entry=0x0, modname=0xfffffe004a17ec00 "filemon", parent=parent@entry=0x0,
verinfo=verinfo@entry=0x0, lfpp=lfpp@entry=0xfffffe0046ce0c00) at /mnt/src/sys/kern/kern_linker.c:2272
#16 0xffffffff81082e23 in kern_kldload (td=<optimized out>, file=file@entry=0xfffffe004a17ec00 "filemon",
fileid=fileid@entry=0xfffffe0046ce0ca0) at /mnt/src/sys/kern/kern_linker.c:1236
#17 0xffffffff81083052 in sys_kldload (td=0xffffffff826074b0 <panicstr>, uap=<optimized out>)
at /mnt/src/sys/kern/kern_linker.c:1259
#18 0xffffffff81a223ae in syscallenter (td=0xfffffe005bc4c000)
at /mnt/src/sys/amd64/amd64/../../kern/subr_syscall.c:189
#19 amd64_syscall (td=0xfffffe005bc4c000, traced=0) at /mnt/src/sys/amd64/amd64/trap.c:1192
#20 <signal handler called>
#21 0x000012614498a7da in ?? ()
Backtrace stopped: Cannot access memory at address 0x126142f959a8
(kgdb)
Are you using UFS? I bet your linker.hints is getting truncated during the panic, which triggers a small bug in the kernel linker: https://reviews.freebsd.org/D47240 (In reply to Mark Johnston from comment #3) Yes, the VM is using UFS. Why is this a problem on UFS specifically, by the way? (In reply to Christos Margiolis from comment #4) Yes, one of the consequences of the UFS soft updates implementation is that crashes can result in truncated files if those files had been written shortly before the panic. So if you do something like "make installkernel; <something panics the system>", the linker.hints file created by kldxref during installkernel might end up truncated. The kernel linker reads that file, and doesn't properly handle the case where it's truncated. (In reply to Mark Johnston from comment #5) Right, this makes since the panics would occur after recompiling the module I work working on. I'm now compiling the kernel with the patch applied to test if it fixes the problem. The crash is fixed, thanks. It turns that linker.hints was indeed getting truncated (see below). Should these messages be concerning by any chance?
Loading kernel modules:
linker.hints file truncated
devmatch: Linker hints version -1515870811 doesn't match expected 1.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg
32-bit compatibility ldconfig path: /usr/lib32 /usr/lib32
Setting hostname: freebsd.
Setting up harvesting: PURE_VMGENID,PURE_RDRAND,[CALLOUT],[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: .
lo0: link state changed to UP
vtnet0: link state changed to UP
Starting Network: lo0 vtnet0.
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 00:a0:98:0e:02:93
inet 169.254.0.2 netmask 0xffff0000 broadcast 169.254.255.255
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Starting devd.
devmatch: Linker hints version -1515870811 doesn't match expected 1.
devmatch: Linker hints version -1515870811 doesn't match expected 1.
devmatch: Linker hints version -1515870811 doesn't match expected 1.
devmatch: Linker hints version -1515870811 doesn't match expected 1.
devmatch: Linker hints version -1515870811 doesn't match expected 1.
devmatch: Linker hints version -1515870811 doesn't match expected 1.
devmatch: Linker hints version -1515870811 doesn't match expected 1.
(In reply to Christos Margiolis from comment #7) Those messages are the result of basically the same bug, but this time in devmatch. The devmatch patch catches the truncated linker files fine. Now I do not get any panic, but I do need to recompile the module in order to be able to load it. A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=9b2f2fbfcbc51f77e96a41e62b1c06a8ccd4ca15 commit 9b2f2fbfcbc51f77e96a41e62b1c06a8ccd4ca15 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-10-23 16:55:16 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-10-23 22:12:45 +0000 devmatch: Catch truncated linker hints files PR: 282268 Reviewed by: christos, imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D47243 sbin/devmatch/devmatch.c | 6 ++++++ 1 file changed, 6 insertions(+) A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b5149b265346c55994c7ebaab2a6a6fd1bd6fe5e commit b5149b265346c55994c7ebaab2a6a6fd1bd6fe5e Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-10-23 16:54:56 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-10-23 22:12:45 +0000 linker: Handle a truncated hints file properly If vattr.va_size is 0, we will end up accessing invalid memory. This is mostly harmless (because malloc(0) still allocates some memory), but it triggers a KASAN report. PR: 282268 Reviewed by: christos, imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D47240 sys/kern/kern_linker.c | 4 ++++ 1 file changed, 4 insertions(+) PRs affecting stable branches should be kept open until the MFC is done. A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=4990a7d344d985668917d00cde0675c042cd5649 commit 4990a7d344d985668917d00cde0675c042cd5649 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-10-23 16:55:16 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-10-30 13:21:06 +0000 devmatch: Catch truncated linker hints files PR: 282268 Reviewed by: christos, imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D47243 (cherry picked from commit 9b2f2fbfcbc51f77e96a41e62b1c06a8ccd4ca15) sbin/devmatch/devmatch.c | 6 ++++++ 1 file changed, 6 insertions(+) A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=9e3e111d749ac83f213a36254fd0717265534e05 commit 9e3e111d749ac83f213a36254fd0717265534e05 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-10-23 16:54:56 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-10-30 13:21:06 +0000 linker: Handle a truncated hints file properly If vattr.va_size is 0, we will end up accessing invalid memory. This is mostly harmless (because malloc(0) still allocates some memory), but it triggers a KASAN report. PR: 282268 Reviewed by: christos, imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D47240 (cherry picked from commit b5149b265346c55994c7ebaab2a6a6fd1bd6fe5e) sys/kern/kern_linker.c | 4 ++++ 1 file changed, 4 insertions(+) A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=d57d714238471eaa44c20538df1488fae378665e commit d57d714238471eaa44c20538df1488fae378665e Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-10-23 16:54:56 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-10-30 13:21:39 +0000 linker: Handle a truncated hints file properly If vattr.va_size is 0, we will end up accessing invalid memory. This is mostly harmless (because malloc(0) still allocates some memory), but it triggers a KASAN report. PR: 282268 Reviewed by: christos, imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D47240 (cherry picked from commit b5149b265346c55994c7ebaab2a6a6fd1bd6fe5e) sys/kern/kern_linker.c | 4 ++++ 1 file changed, 4 insertions(+) A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=390486cb25f9e4a2a3f592695bbdbffe5019e898 commit 390486cb25f9e4a2a3f592695bbdbffe5019e898 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-10-23 16:55:16 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-10-30 13:21:31 +0000 devmatch: Catch truncated linker hints files PR: 282268 Reviewed by: christos, imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D47243 (cherry picked from commit 9b2f2fbfcbc51f77e96a41e62b1c06a8ccd4ca15) sbin/devmatch/devmatch.c | 6 ++++++ 1 file changed, 6 insertions(+) |
Created attachment 254444 [details] linker_load_module() disassembly This is a relatively consistent bug, although it does not have a 100% reproduction rate. What I usually do is the following: 1. Boot into a KASAN kernel. 2. Panic the kernel somehow and reboot. 3. During the reboot, it is likely that linker_load_module() will panic when rc(8) is trying to load the modules. I have also attached the linker_load_module() disassembly. Sample panic message: Loading kernel modules: panic: ASan: Invalid access, 4-byte read at 0xfffffe0047935020, MallocRedZone(fb) cpuid = 1 time = 1729606284 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0xa5/frame 0xfffffe0046c3f070 kdb_backtrace() at kdb_backtrace+0xc6/frame 0xfffffe0046c3f1d0 vpanic() at vpanic+0x226/frame 0xfffffe0046c3f370 panic() at panic+0xb5/frame 0xfffffe0046c3f440 kasan_code_name() at kasan_code_name/frame 0xfffffe0046c3f510 linker_load_module() at linker_load_module+0xe03/frame 0xfffffe0046c3fbb0 kern_kldload() at kern_kldload+0x233/frame 0xfffffe0046c3fc70 sys_kldload() at sys_kldload+0xd2/frame 0xfffffe0046c3fd10 amd64_syscall() at amd64_syscall+0x39e/frame 0xfffffe0046c3ff30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0046c3ff30 --- syscall (304, FreeBSD ELF64, kldload), rip = 0x311d0dce37da, rsp = 0x311d0c9d2428, rbp = 0x311d0c9d29a0 --- KDB: enter: panic [ thread pid 92 tid 100096 ] Stopped at kdb_enter+0x34: movq $0,0x1f09b11(%rip) db>