Bug 247945 - Kernel option KERN_TLS leads to an immediate panic while booting on ARM64 (RPi4)
Summary: Kernel option KERN_TLS leads to an immediate panic while booting on ARM64 (RPi4)
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: arm (show other bugs)
Version: CURRENT
Hardware: arm64 Any
: --- Affects Some People
Assignee: freebsd-arm (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-07-13 06:48 UTC by Gordon Bergling
Modified: 2020-10-15 15:07 UTC (History)
4 users (show)

See Also:


Attachments
arm64 kernel panic on RPi4b (284.88 KB, image/jpeg)
2020-07-23 15:55 UTC, Gordon Bergling
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Gordon Bergling freebsd_committer 2020-07-13 06:48:20 UTC
I have the following kernel configuration for amd64, which I try to use for arm64 on a Raspberry Pi 4b. 

------------------------------------
include		GENERIC
options		RATELIMIT
options		TCPHPTS
options		KERN_TLS
------------------------------------

After some trail and error I nailed the panic down to the KERN_TLS option. I am unable to get a crash dump, because my keyboard isn't working over USB, but I have attached a screenshot of the kernel panic.

The message of the panic is "panic: Thread already setup for VFP".
Comment 1 Mark Johnston freebsd_committer 2020-07-23 15:51:03 UTC
(In reply to Gordon Bergling from comment #0)
I don't see any attachment.  I also can't reproduce any crashes when booting an arm64 KERN_TLS-enabled kernel on a test board.
Comment 2 Gordon Bergling freebsd_committer 2020-07-23 15:55:27 UTC
Created attachment 216709 [details]
arm64 kernel panic on RPi4b
Comment 3 Gordon Bergling freebsd_committer 2020-07-23 15:57:50 UTC
(In reply to Mark Johnston from comment #1)

It is maybe a combination of the kernel options above. I also compile world and kernel with the following src.conf.

WITH_EXTRA_TCP_STACKS=1
WITH_BEARSSL=1
WITH_PIE=1
WITH_RETPOLINE=1
Comment 4 Mark Johnston freebsd_committer 2020-07-23 15:58:02 UTC
Ah, it does panic if I add all three of RATELIMIT, TCPHPTS and KERN_TLS:

TCP_ratelimit: Is now initialized                                              
panic: Thread already setup for the VFP                                        
cpuid = 1                                                                      
time = 13                                                                      
KDB: stack backtrace:                                                          
db_trace_self() at db_fetch_ksymtab+0x158                                                                                                                      
         pc = 0xffff0000007719e8  lr = 0xffff0000001094ac
         sp = 0xffff00005aa50570  fp = 0xffff00005aa50770  
                                                                               
db_fetch_ksymtab() at vpanic+0x194                                                                                                                             
         pc = 0xffff0000001094ac  lr = 0xffff00000041bcb0                      
         sp = 0xffff00005aa50780  fp = 0xffff00005aa507d0                      
                                                                                                                                                               
vpanic() at panic+0x44                                                                                                                                         
         pc = 0xffff00000041bcb0  lr = 0xffff00000041ba58                      
         sp = 0xffff00005aa507e0  fp = 0xffff00005aa50890                      
                                                                                                                                                               
panic() at fpu_kern_thread+0x68                                                
         pc = 0xffff00000041ba58  lr = 0xffff000000792cd8                      
         sp = 0xffff00005aa508a0  fp = 0xffff00005aa508a0                      
                                                                               
fpu_kern_thread() at ktls_enqueue+0x47c                                        
         pc = 0xffff000000792cd8  lr = 0xffff0000004aa980                      
         sp = 0xffff00005aa508b0  fp = 0xffff00005aa508f0                      
                                                                               
ktls_enqueue() at fork_exit+0x7c                                               
         pc = 0xffff0000004aa980  lr = 0xffff0000003d8144
         sp = 0xffff00005aa50900  fp = 0xffff00005aa50950
                                                                               
fork_exit() at fork_trampoline+0x10                                            
         pc = 0xffff0000003d8144  lr = 0xffff000000790dc4                      
         sp = 0xffff00005aa50960  fp = 0x0000000000000000
Comment 5 John Baldwin freebsd_committer freebsd_triage 2020-07-23 16:22:31 UTC
This may be a bug in the arm64 fpu interface?  Note that the kthread calls fpu_kern_thread() first thing before it's done any work at all.

Ah, I think I see the bug.  cpu_copy_thread() in vm_machdep.c just blindly copies pcb_flags which means that if the first kthread runs before another one is forked, the newly forked thread will bogusly have PCB_FP_KERN set in pcb_flags.

The amd64 version is careful to do this in cpu_copy_thread:

	clear_pcb_flags(pcb2, PCB_FPUINITDONE | PCB_USERFPUINITDONE |
	    PCB_KERNFPU);

cpu_fork() for amd64 does an fpuexit() before copying the pcb which effectively does the same thing.

On arm64, cpu_fork() calls vfp_save_state which does not clear any pcb_flags after doing vfp_store/vfp_disable.  It also doesn't check pcb_flags to determine if it should store/disable which is probably wrong.  cpu_copy_thread() just needs to clear the relevant flags in the new pcb I think.
Comment 6 Gordon Bergling freebsd_committer 2020-10-15 15:07:10 UTC
Any update on this topic regarding a bugfix?