Bug 247945 - Kernel option KERN_TLS leads to an immediate panic while booting on ARM64 (RPi4)
Summary: Kernel option KERN_TLS leads to an immediate panic while booting on ARM64 (RPi4)
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: arm (show other bugs)
Version: CURRENT
Hardware: arm64 Any
: --- Affects Some People
Assignee: John Baldwin
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-07-13 06:48 UTC by Gordon Bergling
Modified: 2021-01-19 19:07 UTC (History)
6 users (show)

See Also:


Attachments
arm64 kernel panic on RPi4b (284.88 KB, image/jpeg)
2020-07-23 15:55 UTC, Gordon Bergling
no flags Details
dmesg output from RPi4B (15.35 KB, text/plain)
2021-01-15 13:47 UTC, Gordon Bergling
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Gordon Bergling freebsd_committer 2020-07-13 06:48:20 UTC
I have the following kernel configuration for amd64, which I try to use for arm64 on a Raspberry Pi 4b. 

------------------------------------
include		GENERIC
options		RATELIMIT
options		TCPHPTS
options		KERN_TLS
------------------------------------

After some trail and error I nailed the panic down to the KERN_TLS option. I am unable to get a crash dump, because my keyboard isn't working over USB, but I have attached a screenshot of the kernel panic.

The message of the panic is "panic: Thread already setup for VFP".
Comment 1 Mark Johnston freebsd_committer 2020-07-23 15:51:03 UTC
(In reply to Gordon Bergling from comment #0)
I don't see any attachment.  I also can't reproduce any crashes when booting an arm64 KERN_TLS-enabled kernel on a test board.
Comment 2 Gordon Bergling freebsd_committer 2020-07-23 15:55:27 UTC
Created attachment 216709 [details]
arm64 kernel panic on RPi4b
Comment 3 Gordon Bergling freebsd_committer 2020-07-23 15:57:50 UTC
(In reply to Mark Johnston from comment #1)

It is maybe a combination of the kernel options above. I also compile world and kernel with the following src.conf.

WITH_EXTRA_TCP_STACKS=1
WITH_BEARSSL=1
WITH_PIE=1
WITH_RETPOLINE=1
Comment 4 Mark Johnston freebsd_committer 2020-07-23 15:58:02 UTC
Ah, it does panic if I add all three of RATELIMIT, TCPHPTS and KERN_TLS:

TCP_ratelimit: Is now initialized                                              
panic: Thread already setup for the VFP                                        
cpuid = 1                                                                      
time = 13                                                                      
KDB: stack backtrace:                                                          
db_trace_self() at db_fetch_ksymtab+0x158                                                                                                                      
         pc = 0xffff0000007719e8  lr = 0xffff0000001094ac
         sp = 0xffff00005aa50570  fp = 0xffff00005aa50770  
                                                                               
db_fetch_ksymtab() at vpanic+0x194                                                                                                                             
         pc = 0xffff0000001094ac  lr = 0xffff00000041bcb0                      
         sp = 0xffff00005aa50780  fp = 0xffff00005aa507d0                      
                                                                                                                                                               
vpanic() at panic+0x44                                                                                                                                         
         pc = 0xffff00000041bcb0  lr = 0xffff00000041ba58                      
         sp = 0xffff00005aa507e0  fp = 0xffff00005aa50890                      
                                                                                                                                                               
panic() at fpu_kern_thread+0x68                                                
         pc = 0xffff00000041ba58  lr = 0xffff000000792cd8                      
         sp = 0xffff00005aa508a0  fp = 0xffff00005aa508a0                      
                                                                               
fpu_kern_thread() at ktls_enqueue+0x47c                                        
         pc = 0xffff000000792cd8  lr = 0xffff0000004aa980                      
         sp = 0xffff00005aa508b0  fp = 0xffff00005aa508f0                      
                                                                               
ktls_enqueue() at fork_exit+0x7c                                               
         pc = 0xffff0000004aa980  lr = 0xffff0000003d8144
         sp = 0xffff00005aa50900  fp = 0xffff00005aa50950
                                                                               
fork_exit() at fork_trampoline+0x10                                            
         pc = 0xffff0000003d8144  lr = 0xffff000000790dc4                      
         sp = 0xffff00005aa50960  fp = 0x0000000000000000
Comment 5 John Baldwin freebsd_committer freebsd_triage 2020-07-23 16:22:31 UTC
This may be a bug in the arm64 fpu interface?  Note that the kthread calls fpu_kern_thread() first thing before it's done any work at all.

Ah, I think I see the bug.  cpu_copy_thread() in vm_machdep.c just blindly copies pcb_flags which means that if the first kthread runs before another one is forked, the newly forked thread will bogusly have PCB_FP_KERN set in pcb_flags.

The amd64 version is careful to do this in cpu_copy_thread:

	clear_pcb_flags(pcb2, PCB_FPUINITDONE | PCB_USERFPUINITDONE |
	    PCB_KERNFPU);

cpu_fork() for amd64 does an fpuexit() before copying the pcb which effectively does the same thing.

On arm64, cpu_fork() calls vfp_save_state which does not clear any pcb_flags after doing vfp_store/vfp_disable.  It also doesn't check pcb_flags to determine if it should store/disable which is probably wrong.  cpu_copy_thread() just needs to clear the relevant flags in the new pcb I think.
Comment 6 Gordon Bergling freebsd_committer 2020-10-15 15:07:10 UTC
Any update on this topic regarding a bugfix?
Comment 7 Gordon Bergling freebsd_committer 2020-11-10 13:19:37 UTC
ping?
Comment 8 John Baldwin freebsd_committer freebsd_triage 2021-01-08 23:00:06 UTC
You can find a branch with a fix here (along with a cleanup patch):

https://github.com/freebsd/freebsd-src/compare/main...bsdjhb:arm64_fpu_pcbinit
Comment 9 John Baldwin freebsd_committer freebsd_triage 2021-01-08 23:01:51 UTC
I was wrong about fpuexit() in amd64 clearing flags, for fork we want both the register state and the flags copied to the new thread rather than cleared.  It is only the case of new threads where we need to clear the flags explicitly.
Comment 10 Gordon Bergling freebsd_committer 2021-01-09 11:02:10 UTC
(In reply to John Baldwin from comment #8)

Thanks for the patch, I'll test it as soon https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252541 
is solved. (panic: Too many early devmatch mappings).
Comment 11 Gordon Bergling freebsd_committer 2021-01-15 13:47:49 UTC
Created attachment 221599 [details]
dmesg output from RPi4B
Comment 12 Gordon Bergling freebsd_committer 2021-01-15 13:51:34 UTC
(In reply to John Baldwin from comment #8)

I can confirm that the kernel panic doesn't happen anymore, once I apply your patch to sys/arm64/arm64/vm_machdep.c.

The strange thing is, that KTLS isn't announce in the dmesg output. Please see the attached dmesg output.

On amd64 I see,
KTLS: Initialized 2 threads
in a non-verbose dmesg.
Comment 13 John Baldwin freebsd_committer freebsd_triage 2021-01-15 19:02:24 UTC
Hmm, I've posted the FPU patches for review so they should hopefully land soon.

Not seeing the printf is indeed odd, and I can't think of why you would not see that.  The printf is not hidden behind boot verbose so you should always see it.  Do you have a "KTLS" process in 'ps ax' output?
Comment 14 Gordon Bergling freebsd_committer 2021-01-15 19:33:16 UTC
(In reply to John Baldwin from comment #13)

I don't see a KTLS thread like I see on an amd64 VM. Maybe kTLS is scanning for a suitable hardware encryption support and the RPi4B isn't supporting the necessary methods. See PR 252543.
Comment 15 John Baldwin freebsd_committer freebsd_triage 2021-01-15 20:41:20 UTC
(In reply to Gordon Bergling from comment #14)
No, ktls_init() always creates the worker threads, it's only at the time of an attempt to offload an individual socket that we check for backends.  Perhaps add some additional printf's (or even a panic) in the sysinit in sys/kern/uipc_ktls.c to verify it is getting called?  In particular, the panic in this case happened when those worker threads started, so if you don't have a KTLS proc then you aren't really testing the patch.
Comment 16 Gordon Bergling freebsd_committer 2021-01-16 08:27:08 UTC
(In reply to John Baldwin from comment #15)
Sorry, that's by bad, I messed up the objdir.

ps shows
root         24   0.0  0.0      0   64  -  DL   09:24   0:00.00 [KTLS]

and dmesg
KTLS: Initialized 4 threads
Comment 17 commit-hook freebsd_committer 2021-01-19 19:07:27 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=c6e27f5697c28e188739ea1b4994dc8869dfb6c2

commit c6e27f5697c28e188739ea1b4994dc8869dfb6c2
Author:     John Baldwin <jhb@FreeBSD.org>
AuthorDate: 2021-01-08 22:56:54 +0000
Commit:     John Baldwin <jhb@FreeBSD.org>
CommitDate: 2021-01-19 19:05:48 +0000

    arm64: Clear FPU flags in the pcb in cpu_copy_thread().

    New threads start off with clean FPU state instead of inheriting state
    from the parent thread.

    PR:             247945
    Sponsored by:   Netflix

 sys/arm64/arm64/vm_machdep.c | 1 +
 1 file changed, 1 insertion(+)