Bug 246207 - [geom] geli livelocks during panic
Summary: [geom] geli livelocks during panic
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.1-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: Alan Somers
URL: https://reviews.freebsd.org/D24697
Keywords: panic
Depends on:
Blocks:
 
Reported: 2020-05-05 01:44 UTC by Alan Somers
Modified: 2020-06-12 21:16 UTC (History)
1 user (show)

See Also:


Attachments
Fix dumping core on panic when there are unused geli devices (624 bytes, patch)
2020-05-05 14:38 UTC, Alan Somers
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alan Somers freebsd_committer 2020-05-05 01:44:39 UTC
Some geli-using machines I administer occasionally panic.  When they do, they sometimes dump core but often don't.  When they don't, they simply hang after printing the stack trace, but before printing the uptime.

I've traced the problem to geli's shutdown_pre_sync event handler.  It tries to destroy each geli device.  We can't simply skip that step if a panic is underway; erasing the keys is necessary to prevent warm-boot attacks.  The problem lies in the following lines.  

g_eli_destroy:
	sc->sc_flags |= G_ELI_FLAG_DESTROY;
	wakeup(sc);
	/*
	 * Wait for kernel threads self destruction.
	 */
	while (!LIST_EMPTY(&sc->sc_workers)) {
		msleep(&sc->sc_workers, &sc->sc_queue_mtx, PRIBIO,
		    "geli:destroy", 0);
	}

_sleep:
	if (SCHEDULER_STOPPED_TD(td)) {
		if (lock != NULL && priority & PDROP)
			class->lc_unlock(lock);
		return (0);
	}

As you can see, if the scheduler is stopped for the current thread (which it will be during a panic), then msleep does nothing, cause g_eli_destroy to loop indefinitely.  The obvious solution, which I haven't yet tested, would be to skip that section in g_eli_destroy when the scheduler is stopped.  What I don't understand is why g_eli_destroy _ever_ works during a panic.  Perhaps it has something to do with the allocation of worker threads among cores?  Perhaps it only succeeds when all worker threads happen to be on different cores?  I find that unlikely though, because these servers have thousands of worker threads.
Comment 1 Alan Somers freebsd_committer 2020-05-05 04:00:49 UTC
Patch in-progress.
Comment 2 Alan Somers freebsd_committer 2020-05-05 14:28:22 UTC
I found the reason why the livelock doesn't happen every time: g_eli_shutdown_pre_sync only destroys unused geli devices.  In-use geli devices it marks with G_ELI_FLAG_RW_DETACH, which will cause they to be destroyed on last close.  Most systems don't typically have unused geli devices, which is why they don't livelock.  This also suggests an easier solution to the problem.  Instead of modifying g_eli_destroy, just modify g_eli_shutdown_pre_sync to destroy nothing in the event of a panic

Steps to Reproduce
==================

$ sudo mdconfig -a -t swap -s 128M
md0
$ sudo geli init -i0 md0
Enter new passphrase: 
Reenter new passphrase: 

Metadata backup for provider md0 can be found in /var/backups/md0.eli
and can be restored with the following command:

	# geli restore /var/backups/md0.eli md0

$ sudo geli attach md0
Enter passphrase: 
$ geli list md0 # Note that the provider's mode is r0w0e0
# Disable ddb.  Otherwise we'll enter ddb before calling pre-sync event hooks
$ sudo sysctl debug.debugger_on_panic=0 
$ sudo sysctl debug.kdb.panic=1

# The console will print something like this.
# Note the lack of a core dump or uptime
panic: kdb_sysctl_panic
cpuid = 0
time = 1588688768
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00603037c0
vpanic() at vpanic+0x182/frame 0xfffffe0060303810
panic() at panic+0x43/frame 0xfffffe0060303870
kdb_sysctl_panic() at kdb_sysctl_panic+0x61/frame 0xfffffe00603038a0
sysctl_root_handler_locked() at sysctl_root_handler_locked+0x9c/frame 0xfffffe00603038f0
sysctl_root() at sysctl_root+0x20a/frame 0xfffffe0060303970
userland_sysctl() at userland_sysctl+0x17b/frame 0xfffffe0060303a20
sys___sysctl() at sys___sysctl+0x5f/frame 0xfffffe0060303ad0
amd64_syscall() at amd64_syscall+0x140/frame 0xfffffe0060303bf0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0060303bf0
--- syscall (202, FreeBSD ELF64, sys___sysctl), rip = 0x80042d56a, rsp = 0x7fffffffd538, rbp = 0x7fffffffd570 ---
Comment 3 Alan Somers freebsd_committer 2020-05-05 14:38:01 UTC
Created attachment 214156 [details]
Fix dumping core on panic when there are unused geli devices

The attached patch is tested on stable/12 and head.
Comment 4 commit-hook freebsd_committer 2020-05-27 19:13:49 UTC
A commit references this bug:

Author: asomers
Date: Wed May 27 19:13:26 UTC 2020
New revision: 361562
URL: https://svnweb.freebsd.org/changeset/base/361562

Log:
  geli: fix a livelock during panic

  During any kind of shutdown, kern_reboot calls geli's pre_sync event hook,
  which tries to destroy all unused geli devices. But during a panic, geli
  can't destroy any devices, because the scheduler is stopped, so it can't
  switch threads. A livelock results, and the system never dumps core.

  This commit fixes the problem by refusing to destroy any devices during
  panic, used or otherwise.

  PR:		246207
  Reviewed by:	jhb
  MFC after:	2 weeks
  Sponsored by:	Axcient
  Differential Revision:	https://reviews.freebsd.org/D24697

Changes:
  head/sys/geom/eli/g_eli.c
Comment 5 commit-hook freebsd_committer 2020-06-12 20:39:58 UTC
A commit references this bug:

Author: asomers
Date: Fri Jun 12 20:39:42 UTC 2020
New revision: 362118
URL: https://svnweb.freebsd.org/changeset/base/362118

Log:
  MFC r361562:

  geli: fix a livelock during panic

  During any kind of shutdown, kern_reboot calls geli's pre_sync event hook,
  which tries to destroy all unused geli devices. But during a panic, geli
  can't destroy any devices, because the scheduler is stopped, so it can't
  switch threads. A livelock results, and the system never dumps core.

  This commit fixes the problem by refusing to destroy any devices during
  panic, used or otherwise.

  PR:		246207
  Reviewed by:	jhb
  Sponsored by:	Axcient
  Differential Revision:	https://reviews.freebsd.org/D24697

Changes:
_U  stable/12/
  stable/12/sys/geom/eli/g_eli.c