I have the following kernel configuration for amd64, which I try to use for arm64 on a Raspberry Pi 4b.
After some trail and error I nailed the panic down to the KERN_TLS option. I am unable to get a crash dump, because my keyboard isn't working over USB, but I have attached a screenshot of the kernel panic.
The message of the panic is "panic: Thread already setup for VFP".
(In reply to Gordon Bergling from comment #0)
I don't see any attachment. I also can't reproduce any crashes when booting an arm64 KERN_TLS-enabled kernel on a test board.
Created attachment 216709 [details]
arm64 kernel panic on RPi4b
(In reply to Mark Johnston from comment #1)
It is maybe a combination of the kernel options above. I also compile world and kernel with the following src.conf.
Ah, it does panic if I add all three of RATELIMIT, TCPHPTS and KERN_TLS:
TCP_ratelimit: Is now initialized
panic: Thread already setup for the VFP
cpuid = 1
time = 13
KDB: stack backtrace:
db_trace_self() at db_fetch_ksymtab+0x158
pc = 0xffff0000007719e8 lr = 0xffff0000001094ac
sp = 0xffff00005aa50570 fp = 0xffff00005aa50770
db_fetch_ksymtab() at vpanic+0x194
pc = 0xffff0000001094ac lr = 0xffff00000041bcb0
sp = 0xffff00005aa50780 fp = 0xffff00005aa507d0
vpanic() at panic+0x44
pc = 0xffff00000041bcb0 lr = 0xffff00000041ba58
sp = 0xffff00005aa507e0 fp = 0xffff00005aa50890
panic() at fpu_kern_thread+0x68
pc = 0xffff00000041ba58 lr = 0xffff000000792cd8
sp = 0xffff00005aa508a0 fp = 0xffff00005aa508a0
fpu_kern_thread() at ktls_enqueue+0x47c
pc = 0xffff000000792cd8 lr = 0xffff0000004aa980
sp = 0xffff00005aa508b0 fp = 0xffff00005aa508f0
ktls_enqueue() at fork_exit+0x7c
pc = 0xffff0000004aa980 lr = 0xffff0000003d8144
sp = 0xffff00005aa50900 fp = 0xffff00005aa50950
fork_exit() at fork_trampoline+0x10
pc = 0xffff0000003d8144 lr = 0xffff000000790dc4
sp = 0xffff00005aa50960 fp = 0x0000000000000000
This may be a bug in the arm64 fpu interface? Note that the kthread calls fpu_kern_thread() first thing before it's done any work at all.
Ah, I think I see the bug. cpu_copy_thread() in vm_machdep.c just blindly copies pcb_flags which means that if the first kthread runs before another one is forked, the newly forked thread will bogusly have PCB_FP_KERN set in pcb_flags.
The amd64 version is careful to do this in cpu_copy_thread:
clear_pcb_flags(pcb2, PCB_FPUINITDONE | PCB_USERFPUINITDONE |
cpu_fork() for amd64 does an fpuexit() before copying the pcb which effectively does the same thing.
On arm64, cpu_fork() calls vfp_save_state which does not clear any pcb_flags after doing vfp_store/vfp_disable. It also doesn't check pcb_flags to determine if it should store/disable which is probably wrong. cpu_copy_thread() just needs to clear the relevant flags in the new pcb I think.
Any update on this topic regarding a bugfix?
You can find a branch with a fix here (along with a cleanup patch):
I was wrong about fpuexit() in amd64 clearing flags, for fork we want both the register state and the flags copied to the new thread rather than cleared. It is only the case of new threads where we need to clear the flags explicitly.
(In reply to John Baldwin from comment #8)
Thanks for the patch, I'll test it as soon https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252541
is solved. (panic: Too many early devmatch mappings).
Created attachment 221599 [details]
dmesg output from RPi4B
(In reply to John Baldwin from comment #8)
I can confirm that the kernel panic doesn't happen anymore, once I apply your patch to sys/arm64/arm64/vm_machdep.c.
The strange thing is, that KTLS isn't announce in the dmesg output. Please see the attached dmesg output.
On amd64 I see,
KTLS: Initialized 2 threads
in a non-verbose dmesg.
Hmm, I've posted the FPU patches for review so they should hopefully land soon.
Not seeing the printf is indeed odd, and I can't think of why you would not see that. The printf is not hidden behind boot verbose so you should always see it. Do you have a "KTLS" process in 'ps ax' output?
(In reply to John Baldwin from comment #13)
I don't see a KTLS thread like I see on an amd64 VM. Maybe kTLS is scanning for a suitable hardware encryption support and the RPi4B isn't supporting the necessary methods. See PR 252543.
(In reply to Gordon Bergling from comment #14)
No, ktls_init() always creates the worker threads, it's only at the time of an attempt to offload an individual socket that we check for backends. Perhaps add some additional printf's (or even a panic) in the sysinit in sys/kern/uipc_ktls.c to verify it is getting called? In particular, the panic in this case happened when those worker threads started, so if you don't have a KTLS proc then you aren't really testing the patch.
(In reply to John Baldwin from comment #15)
Sorry, that's by bad, I messed up the objdir.
root 24 0.0 0.0 0 64 - DL 09:24 0:00.00 [KTLS]
KTLS: Initialized 4 threads
A commit in branch main references this bug:
Author: John Baldwin <jhb@FreeBSD.org>
AuthorDate: 2021-01-08 22:56:54 +0000
Commit: John Baldwin <jhb@FreeBSD.org>
CommitDate: 2021-01-19 19:05:48 +0000
arm64: Clear FPU flags in the pcb in cpu_copy_thread().
New threads start off with clean FPU state instead of inheriting state
from the parent thread.
Sponsored by: Netflix
sys/arm64/arm64/vm_machdep.c | 1 +
1 file changed, 1 insertion(+)