Bug 232835 - [gmirror] gmirror fails to recover from degraded mirror sets in some circumstances (2/n)
Summary: [gmirror] gmirror fails to recover from degraded mirror sets in some circumst...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: Conrad Meyer
URL:
Keywords:
Depends on: 232671
Blocks: 232683 232684
  Show dependency treegraph
 
Reported: 2018-10-30 23:27 UTC by Conrad Meyer
Modified: 2018-12-07 02:44 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Conrad Meyer freebsd_committer 2018-10-30 23:27:39 UTC
This is related to bug 232671 , but not identical.

Here is the example scenario:

1. I start with a GMIRROR with three ACTIVE disks
   (sc_ndisks = md_all = 3)

2. I essentially disconnect one of the disks

3. The remaining mirrors lower md_all to 2 and the syncid /
   generation gets bumped

4. I reboot, and the removed disk reappears

5. Geom tastes the stale removed disk first, and populates
   sc_ndisks from its md_all (3)

6. The two valid mirrors are tasted afterwards and the gmirror
   rejects both as having invalid metadata (md_all=2), despite
   their having a higher generation / sync id

The problem is basically that gmirror doesn't "upgrade" its metadata
to the newest valid mirrorset it finds -- it just sticks with whatever
it found first.

I think the solution is being a bit clever about detecting the latest
mirror generation while a gmirror is still in the STARTING state;
and also perhaps a little more clever about when we transition from
STARTING to RUNNING (at which point a newer generation mirror showing
up means we have corruption).
Comment 1 Conrad Meyer freebsd_committer 2018-11-09 06:17:40 UTC
An Isilon engineer is working on this and in communication with me.  We'll probably do a first round code-review in house and then I'll forward it upstream.
Comment 2 Conrad Meyer freebsd_committer 2018-11-21 06:42:49 UTC
https://reviews.freebsd.org/D18062
Comment 3 commit-hook freebsd_committer 2018-12-06 23:56:09 UTC
A commit references this bug:

Author: cem
Date: Thu Dec  6 23:55:41 UTC 2018
New revision: 341665
URL: https://svnweb.freebsd.org/changeset/base/341665

Log:
  gmirror: Evaluate mirror components against newest metadata copy

  If we happen to taste a stale mirror component first, don't reject valid,
  newer components that have differing metadata from the stale component
  (during STARTING).  Instead, update our view of the most recent metadata as
  we taste components.

  Like mediasize beforehand, remove some checks from g_mirror_check_metadata
  which would evict valid components due to metadata that can change over a
  mirror's lifetime.  g_mirror_check_metadata is invoked long before we check
  genid/syncid and decide which component(s) are newest and whether or not we
  have quorum.

  Before checking if we can enter RUNNING (i.e., we have quorum) after a NEW
  component is added, first remove any known stale or inconsistent disks from
  the mirrorset, rather than removing them *after* deciding we have quorum.
  Check if we have quorum after removing these components.

  Additionally, add a knob, kern.geom.mirror.launch_mirror_before_timeout, to
  force gmirrors to wait out the full timeout (kern.geom.mirror.timeout)
  before transitioning from STARTING to RUNNING.  This is a kludge to help
  ensure all eligible, boot-time available mirror components are tasted before
  RUNNING a gmirror.

  When we are instructed to forget mirror components, bump the generation id
  to avoid confusion with such stale components later.

  Add a basic test case for STARTING -> RUNNING startup behavior around stale
  genids.

  PR:		232671, 232835
  Submitted by:	Cindy Yang <cyang AT isilon.com> (previous version)
  Reviewed by:	markj (kernel portions)
  Discussed with:	asomers, Cindy Yang
  Tested by:	pho
  Sponsored by:	Dell EMC Isilon
  Differential Revision:	https://reviews.freebsd.org/D18062

Changes:
  head/sys/geom/mirror/g_mirror.c
  head/sys/geom/mirror/g_mirror.h
  head/tests/sys/geom/class/mirror/Makefile
  head/tests/sys/geom/class/mirror/component_selection.sh
  head/tests/sys/geom/class/mirror/conf.sh
Comment 4 Conrad Meyer freebsd_committer 2018-12-07 00:11:09 UTC
Fixed by (1) ignoring md_all in check_metadata and (2) evicting the stale mirror / refreshing metadata during STARTING.  This particular scenario isn't exactly covered by the committed test, so that would be room for improvement.  But it is believed to fix the issue.
Comment 5 commit-hook freebsd_committer 2018-12-07 02:44:27 UTC
A commit references this bug:

Author: cem
Date: Fri Dec  7 02:44:05 UTC 2018
New revision: 341674
URL: https://svnweb.freebsd.org/changeset/base/341674

Log:
  gmirror: Evaluate mirror components against newest metadata copy

  Re-apply r341665 with format strings fixed.

  If we happen to taste a stale mirror component first, don't reject valid,
  newer components that have differing metadata from the stale component
  (during STARTING).  Instead, update our view of the most recent metadata as
  we taste components.

  Like mediasize beforehand, remove some checks from g_mirror_check_metadata
  which would evict valid components due to metadata that can change over a
  mirror's lifetime.  g_mirror_check_metadata is invoked long before we check
  genid/syncid and decide which component(s) are newest and whether or not we
  have quorum.

  Before checking if we can enter RUNNING (i.e., we have quorum) after a NEW
  component is added, first remove any known stale or inconsistent disks from
  the mirrorset, rather than removing them *after* deciding we have quorum.
  Check if we have quorum after removing these components.

  Additionally, add a knob, kern.geom.mirror.launch_mirror_before_timeout, to
  force gmirrors to wait out the full timeout (kern.geom.mirror.timeout)
  before transitioning from STARTING to RUNNING.  This is a kludge to help
  ensure all eligible, boot-time available mirror components are tasted before
  RUNNING a gmirror.

  Add a basic test case for STARTING -> RUNNING startup behavior around stale
  genids.

  PR:		232671, 232835
  Submitted by:	Cindy Yang <cyang AT isilon.com> (previous version)
  Reviewed by:	markj (kernel portions)
  Discussed with:	asomers, Cindy Yang
  Tested by:	pho
  Sponsored by:	Dell EMC Isilon
  Differential Revision:	https://reviews.freebsd.org/D18062

Changes:
  head/sys/geom/mirror/g_mirror.c
  head/sys/geom/mirror/g_mirror.h
  head/tests/sys/geom/class/mirror/Makefile
  head/tests/sys/geom/class/mirror/component_selection.sh
  head/tests/sys/geom/class/mirror/conf.sh