Bug 233851 - Kernel panic, stable/12 r341604, swapon -a in SU mode, geli encrypted swap, Chelsio T6225-CR, and ccr(4)
Summary: Kernel panic, stable/12 r341604, swapon -a in SU mode, geli encrypted swap, C...
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: John Baldwin
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-12-07 15:01 UTC by Trond.Endrestol
Modified: 2019-01-15 18:54 UTC (History)
3 users (show)

See Also:
jhb: mfc-stable12?


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Trond.Endrestol 2018-12-07 15:01:27 UTC
After booting stable/12 r341604, running a custom kernel including cxgbe(4), cxgbev(4), and ccr(4), and running swapon -a in SU mode:

GEOM_ELI: Device gpt/swap0.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware


Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address   = 0x0
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff805be1b2
stack pointer           = 0x28:0xfffffe00a6253770
frame pointer           = 0x28:0xfffffe00a6253770
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 65 (g_eli[3] gpt/swap0)
trap number             = 12
panic: page fault
cpuid = 3
time = 1544087308
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff8055d9eb = db_trace_self_wrapper+0x2b/frame 0xfffffe00a6253420
vpanic() at 0xffffffff808754d3 = vpanic+0x1a3/frame 0xfffffe00a6253480
panic() at 0xffffffff80875323 = panic+0x43/frame 0xfffffe00a62534e0
trap_fatal() at 0xffffffff80bd745f = trap_fatal+0x35f/frame 0xfffffe00a6253530
trap_pfault() at 0xffffffff80bd74b9 = trap_pfault+0x49/frame 0xfffffe00a4b7f590
trap() at 0xffffffff80bd6ade = trap+0x29e/frame 0xfffffe00a4b7f6a0
calltrap() at 0xffffffff80bb3935 = calltrap+0x8/frame 0xfffffe00a4b7f6a0
--- trap 0xc, rip = 0xffffffff805be1b2, rsp = 0xfffffe00a4b7f770, rbp = 0xfffffe00a4b7f770
t4_wrq_tx_locked() at 0xffffffff805be1b2 = t4_wrq_tx_locked+0x12/frame 0xfffffe00a6253770
ccr_process() at 0xffffffff805e3fc3 = ccr_process+0x1953/frame 0xfffffe00a4b7f970
crypto_dispatch() at 0xffffffff80ae3fa4 = crypto_dispatch+0x144/frame 0xfffffe00a4b7f9b0
g_eli_crypto_run() at 0xffffffff807b5cb3 = g_eli_crypto_run+0x273/frame 0xfffffe00a4b7fa10
g_eli_worker() at 0xffffffff807aebc8 = g_eli_worker+0x3c8/frame 0xfffffe00a4b7fa70
fork_exit() at 0xffffffff80834d93 = fork_exit+0x83/frame 0xfffffe00a4b7fab0
fork_trampoline() at 0xffffffff80bb491e = fork_trampoline+0xe/frame 0xfffffe00a4b7fab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 14s
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,

The Chelsio NIC is a T6225-CR.

Kernel config is https://ximalas.info/~trond/create-zfs/canmount/ENTERPRISE-amd64-stable-12

I tried stable/12 r341623 with ccr(4) removed from the kernel, and there was no problems when I engaged geli encrypted swap in SU mode.

That was yesterday.

Today, while doing the finishing touches after upgrading to stable/12 r341676, and still in SU mode, I ran the following sequence:

killall -TERM moused
killall -TERM devd
umount -a
zfs umount -a
swapoff -a
kldload ccr
swapon -a

To my amazement, I saw the crypto accelerator being loaded and recognized. Clearly, the system didn't panic, but maybe it would have later on, leading me to believe the crypto or the cxbge/ccr subsystem isn't properly initialized early after startup. Please prove me wrong.

I rebooted to SU mode to start fresh, and ran this sequence:

swapon -a
swapoff -a
kldload ccr
swapon -a

I rebooted again to SU mode, and ran this simpler sequence:

kldload ccr
swapon -a

In both cases the kernel panicked as soon as I executed the final command.
Comment 1 Trond.Endrestol 2018-12-07 15:02:41 UTC
Adding np@ and jhb@ to the CC list since they are the authors of cxbge(4) and ccr(4).
Comment 2 John Baldwin freebsd_committer freebsd_triage 2018-12-07 19:30:32 UTC
Currently ccr shares queues with cc0 and needs 'ifconfig cc0 up' before it will work.  We were planning to fix that before now but haven't.  I thought I had added a safety belt for that, but clearly it's only a TODO in the source. :-/

I have a patch for review in https://reviews.freebsd.org/D18478.  This isn't the long term fix, but will avoid panicking.  In your case it will fallback to software crypto when doing the swapon -a.
Comment 3 Trond.Endrestol 2018-12-07 20:02:04 UTC
(In reply to John Baldwin from comment #2)
Thanks, John. That might explain why it succeeded once. I usually run https://ximalas.info/~trond/create-zfs/canmount/single-user-mode.sh.txt from / when I enter SU mode, to save some typing. I'll change my script to enable cc0 before activating the swap device(s). Unfortunately, the machine in question is at work. I might be able to try out your suggestion sometime during the weekend.
Comment 4 Trond.Endrestol 2018-12-08 16:24:47 UTC
(In reply to Trond.Endrestol from comment #3)
After some thinking, this startup script, named /etc/rc.d/ifconfig_cc0_up_before_swap, seems appropriate for my case, at least until t4_crypto.c is updated in head, and later in stable/12. Running rcorder manually places this script near the top of the list, well before /etc/rc.d/swap. I'll try to pop in at $WORK on Sunday to give the script a spin. The alternative, I figured, is to run "kldload ccr" in /etc/rc.local or some other script placed in either /etc/rc.d or /usr/local/etc/rc.d. I opted for covering the case where ccr(4) is part of the static kernel.

#!/bin/sh

# PROVIDE: ifconfig_cc0_up_before_swap
# REQUIRE: geli
# BEFORE: swap

# Set
# ifconfig_cc0_up_before_swap_enable="YES"
# in /etc/rc.conf.

. /etc/rc.subr

name="ifconfig_cc0_up_before_swap"
desc="Ensure interface cc0 is up before utilizing any crypto(4) services handled by ccr(4), including geli(8) encrypted swap devices"
start_cmd='/sbin/ifconfig cc0 up || true' # Any errors are usually tolerable.
stop_cmd=':'

load_rc_config $name
run_rc_command "$1"

# EOF
Comment 5 Conrad Meyer freebsd_committer 2018-12-08 19:39:24 UTC
(In reply to Trond.Endrestol from comment #4)
Shouldn't that run BEFORE geli too?
Comment 6 Trond.Endrestol 2018-12-09 08:13:53 UTC
(In reply to Conrad Meyer from comment #5)
Before I changed anything, the beginning of the output of rcorder /etc/rc.d/* went like this:

/etc/rc.d/growfs
/etc/rc.d/sysctl
/etc/rc.d/hostid
/etc/rc.d/zvol
/etc/rc.d/ifconfig_cc0_up_before_swap
/etc/rc.d/dumpon
/etc/rc.d/ddb
/etc/rc.d/geli
/etc/rc.d/gbde
/etc/rc.d/ccd
/etc/rc.d/swap
/etc/rc.d/fsck
/etc/rc.d/root
/etc/rc.d/mdconfig
/etc/rc.d/hostid_save
/etc/rc.d/mountcritlocal

On inspection of /etc/rc.d/geli, it actually provides disks, I have thus added disks to BEFORE instead of geli and removed REQUIRE.

Now the output of rcorder /etc/rc.d/* begins like this:

/etc/rc.d/growfs
/etc/rc.d/sysctl
/etc/rc.d/hostid
/etc/rc.d/zvol
/etc/rc.d/ifconfig_cc0_up_before_swap
/etc/rc.d/dumpon
/etc/rc.d/ddb
/etc/rc.d/geli
/etc/rc.d/gbde
/etc/rc.d/ccd
/etc/rc.d/swap
/etc/rc.d/fsck
/etc/rc.d/root
/etc/rc.d/mdconfig
/etc/rc.d/hostid_save
/etc/rc.d/mountcritlocal

The output remains unchanged, but the handling of my script is probably more robust. Thank you for your input.
Comment 7 Trond.Endrestol 2018-12-10 15:46:09 UTC
(In reply to Trond.Endrestol from comment #6)
Both scripts are very successful. The system is now at r341794. I incorporated jkim@'s patch for OpenSSL's devcrypto engine, see https://lists.freebsd.org/pipermail/freebsd-stable/2018-December/090203.html

I'm not sure if ccr4() is an improvement over AES-NI, I'm only curious to see what the T6225-CR NIC can do for me. OpenSSL/OpenSSH should be able to utilize the crypto accelerator:

# openssl version
OpenSSL 1.1.1a-freebsd  20 Nov 2018
# openssl engine -t -v
(devcrypto) /dev/crypto engine
     [ available ]
(dynamic) Dynamic engine loading support
     [ unavailable ]
     SO_PATH, NO_VCHECK, ID, LIST_ADD, DIR_LOAD, DIR_ADD, LOAD
# fuser /dev/crypto
/dev/crypto:  7314w  7313w  7308w  6999w  6998w  5658w  5398w  5331w  5329w  5326w  5325w  5324w  5323w  5248w  2522w  1196w  1193w  1190w  1164w  1147w  1146w  1145w
Comment 8 John Baldwin freebsd_committer freebsd_triage 2018-12-10 17:03:54 UTC
(In reply to Trond.Endrestol from comment #7)
So the /dev/crypto engine in OpenSSL is a bit limited in what it can do.  The engine in OpenSSL 1.0.2 only supported a few ciphers like AES-CBC and older SHA1 and SHA2/256 digests.  It did not support AES-GCM, etc.  I had patches to 1.0.2 to add GCM support as a proof of concept, though I was focused on HTTPS testing rather than SSH.  I haven't tested the 1.1.1 engine yet, but when I looked at the code in 1.1.x a few months ago it was even more limited in terms of what ciphers it supported than 1.0.2.  It certainly did not support offloading AES-GCM.  So, I currently would not expect OpenSSH in FreeBSD to use /dev/crypto.  Web servers like apache and nginx will not make use of /dev/crypto either with the current engine.  I would like to eventually upstream support for AES-GCM and TLS to OpenSSL 1.1.x so that they are not local patches in the FreeBSD version, but I haven't circled back to working on that.

The reason OpenSSH is probably opening /dev/crypto currently is to try to offload the RSA operations during connection setup, but the requests are probably failing and it is still doing RSA in software.

Places where I expect ccr(4) to be useful currently are in-kernel crypto acceleration for things like GELI and IPSec.
Comment 9 Trond.Endrestol 2018-12-10 18:37:42 UTC
(In reply to John Baldwin from comment #8)
At least my encrypted swap devices will be served by ccr(4). :-) But after I set vm.pageout_update_period to 0 a couple of weeks ago, the swap devices doesn't see much action. Thank you to everyone involved.
Comment 10 commit-hook freebsd_committer 2019-01-15 18:54:00 UTC
A commit references this bug:

Author: jhb
Date: Tue Jan 15 18:53:45 UTC 2019
New revision: 343056
URL: https://svnweb.freebsd.org/changeset/base/343056

Log:
  Reject new sessions if the necessary queues aren't initialized.

  ccr reuses the control queue and first rx queue from the first port on
  each adapter.  The driver cannot send requests until those queues are
  initialized.  Refuse to create sessions for now if the queues aren't
  ready.  This is a workaround until cxgbe allocates one or more
  dedicated queues for ccr.

  PR:		233851
  MFC after:	1 week
  Sponsored by:	Chelsio Communications
  Differential Revision:	https://reviews.freebsd.org/D18478

Changes:
  head/sys/dev/cxgbe/crypto/t4_crypto.c