Bug 247945
| Summary: | Kernel option KERN_TLS leads to an immediate panic while booting on ARM64 (RPi4) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | Gordon Bergling <gbe> | ||||||
| Component: | arm | Assignee: | John Baldwin <jhb> | ||||||
| Status: | Closed FIXED | ||||||||
| Severity: | Affects Some People | CC: | Andrew, emaste, gallatin, jhb, kib, markj | ||||||
| Priority: | --- | ||||||||
| Version: | CURRENT | ||||||||
| Hardware: | arm64 | ||||||||
| OS: | Any | ||||||||
| Attachments: |
|
||||||||
|
Description
Gordon Bergling
2020-07-13 06:48:20 UTC
(In reply to Gordon Bergling from comment #0) I don't see any attachment. I also can't reproduce any crashes when booting an arm64 KERN_TLS-enabled kernel on a test board. Created attachment 216709 [details]
arm64 kernel panic on RPi4b
(In reply to Mark Johnston from comment #1) It is maybe a combination of the kernel options above. I also compile world and kernel with the following src.conf. WITH_EXTRA_TCP_STACKS=1 WITH_BEARSSL=1 WITH_PIE=1 WITH_RETPOLINE=1 Ah, it does panic if I add all three of RATELIMIT, TCPHPTS and KERN_TLS:
TCP_ratelimit: Is now initialized
panic: Thread already setup for the VFP
cpuid = 1
time = 13
KDB: stack backtrace:
db_trace_self() at db_fetch_ksymtab+0x158
pc = 0xffff0000007719e8 lr = 0xffff0000001094ac
sp = 0xffff00005aa50570 fp = 0xffff00005aa50770
db_fetch_ksymtab() at vpanic+0x194
pc = 0xffff0000001094ac lr = 0xffff00000041bcb0
sp = 0xffff00005aa50780 fp = 0xffff00005aa507d0
vpanic() at panic+0x44
pc = 0xffff00000041bcb0 lr = 0xffff00000041ba58
sp = 0xffff00005aa507e0 fp = 0xffff00005aa50890
panic() at fpu_kern_thread+0x68
pc = 0xffff00000041ba58 lr = 0xffff000000792cd8
sp = 0xffff00005aa508a0 fp = 0xffff00005aa508a0
fpu_kern_thread() at ktls_enqueue+0x47c
pc = 0xffff000000792cd8 lr = 0xffff0000004aa980
sp = 0xffff00005aa508b0 fp = 0xffff00005aa508f0
ktls_enqueue() at fork_exit+0x7c
pc = 0xffff0000004aa980 lr = 0xffff0000003d8144
sp = 0xffff00005aa50900 fp = 0xffff00005aa50950
fork_exit() at fork_trampoline+0x10
pc = 0xffff0000003d8144 lr = 0xffff000000790dc4
sp = 0xffff00005aa50960 fp = 0x0000000000000000
This may be a bug in the arm64 fpu interface? Note that the kthread calls fpu_kern_thread() first thing before it's done any work at all. Ah, I think I see the bug. cpu_copy_thread() in vm_machdep.c just blindly copies pcb_flags which means that if the first kthread runs before another one is forked, the newly forked thread will bogusly have PCB_FP_KERN set in pcb_flags. The amd64 version is careful to do this in cpu_copy_thread: clear_pcb_flags(pcb2, PCB_FPUINITDONE | PCB_USERFPUINITDONE | PCB_KERNFPU); cpu_fork() for amd64 does an fpuexit() before copying the pcb which effectively does the same thing. On arm64, cpu_fork() calls vfp_save_state which does not clear any pcb_flags after doing vfp_store/vfp_disable. It also doesn't check pcb_flags to determine if it should store/disable which is probably wrong. cpu_copy_thread() just needs to clear the relevant flags in the new pcb I think. Any update on this topic regarding a bugfix? ping? You can find a branch with a fix here (along with a cleanup patch): https://github.com/freebsd/freebsd-src/compare/main...bsdjhb:arm64_fpu_pcbinit I was wrong about fpuexit() in amd64 clearing flags, for fork we want both the register state and the flags copied to the new thread rather than cleared. It is only the case of new threads where we need to clear the flags explicitly. (In reply to John Baldwin from comment #8) Thanks for the patch, I'll test it as soon https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252541 is solved. (panic: Too many early devmatch mappings). Created attachment 221599 [details]
dmesg output from RPi4B
(In reply to John Baldwin from comment #8) I can confirm that the kernel panic doesn't happen anymore, once I apply your patch to sys/arm64/arm64/vm_machdep.c. The strange thing is, that KTLS isn't announce in the dmesg output. Please see the attached dmesg output. On amd64 I see, KTLS: Initialized 2 threads in a non-verbose dmesg. Hmm, I've posted the FPU patches for review so they should hopefully land soon. Not seeing the printf is indeed odd, and I can't think of why you would not see that. The printf is not hidden behind boot verbose so you should always see it. Do you have a "KTLS" process in 'ps ax' output? (In reply to John Baldwin from comment #13) I don't see a KTLS thread like I see on an amd64 VM. Maybe kTLS is scanning for a suitable hardware encryption support and the RPi4B isn't supporting the necessary methods. See PR 252543. (In reply to Gordon Bergling from comment #14) No, ktls_init() always creates the worker threads, it's only at the time of an attempt to offload an individual socket that we check for backends. Perhaps add some additional printf's (or even a panic) in the sysinit in sys/kern/uipc_ktls.c to verify it is getting called? In particular, the panic in this case happened when those worker threads started, so if you don't have a KTLS proc then you aren't really testing the patch. (In reply to John Baldwin from comment #15) Sorry, that's by bad, I messed up the objdir. ps shows root 24 0.0 0.0 0 64 - DL 09:24 0:00.00 [KTLS] and dmesg KTLS: Initialized 4 threads A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=c6e27f5697c28e188739ea1b4994dc8869dfb6c2 commit c6e27f5697c28e188739ea1b4994dc8869dfb6c2 Author: John Baldwin <jhb@FreeBSD.org> AuthorDate: 2021-01-08 22:56:54 +0000 Commit: John Baldwin <jhb@FreeBSD.org> CommitDate: 2021-01-19 19:05:48 +0000 arm64: Clear FPU flags in the pcb in cpu_copy_thread(). New threads start off with clean FPU state instead of inheriting state from the parent thread. PR: 247945 Sponsored by: Netflix sys/arm64/arm64/vm_machdep.c | 1 + 1 file changed, 1 insertion(+) |