This is related to bug 232671 , but not identical. Here is the example scenario: 1. I start with a GMIRROR with three ACTIVE disks (sc_ndisks = md_all = 3) 2. I essentially disconnect one of the disks 3. The remaining mirrors lower md_all to 2 and the syncid / generation gets bumped 4. I reboot, and the removed disk reappears 5. Geom tastes the stale removed disk first, and populates sc_ndisks from its md_all (3) 6. The two valid mirrors are tasted afterwards and the gmirror rejects both as having invalid metadata (md_all=2), despite their having a higher generation / sync id The problem is basically that gmirror doesn't "upgrade" its metadata to the newest valid mirrorset it finds -- it just sticks with whatever it found first. I think the solution is being a bit clever about detecting the latest mirror generation while a gmirror is still in the STARTING state; and also perhaps a little more clever about when we transition from STARTING to RUNNING (at which point a newer generation mirror showing up means we have corruption).
An Isilon engineer is working on this and in communication with me. We'll probably do a first round code-review in house and then I'll forward it upstream.
https://reviews.freebsd.org/D18062
A commit references this bug: Author: cem Date: Thu Dec 6 23:55:41 UTC 2018 New revision: 341665 URL: https://svnweb.freebsd.org/changeset/base/341665 Log: gmirror: Evaluate mirror components against newest metadata copy If we happen to taste a stale mirror component first, don't reject valid, newer components that have differing metadata from the stale component (during STARTING). Instead, update our view of the most recent metadata as we taste components. Like mediasize beforehand, remove some checks from g_mirror_check_metadata which would evict valid components due to metadata that can change over a mirror's lifetime. g_mirror_check_metadata is invoked long before we check genid/syncid and decide which component(s) are newest and whether or not we have quorum. Before checking if we can enter RUNNING (i.e., we have quorum) after a NEW component is added, first remove any known stale or inconsistent disks from the mirrorset, rather than removing them *after* deciding we have quorum. Check if we have quorum after removing these components. Additionally, add a knob, kern.geom.mirror.launch_mirror_before_timeout, to force gmirrors to wait out the full timeout (kern.geom.mirror.timeout) before transitioning from STARTING to RUNNING. This is a kludge to help ensure all eligible, boot-time available mirror components are tasted before RUNNING a gmirror. When we are instructed to forget mirror components, bump the generation id to avoid confusion with such stale components later. Add a basic test case for STARTING -> RUNNING startup behavior around stale genids. PR: 232671, 232835 Submitted by: Cindy Yang <cyang AT isilon.com> (previous version) Reviewed by: markj (kernel portions) Discussed with: asomers, Cindy Yang Tested by: pho Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D18062 Changes: head/sys/geom/mirror/g_mirror.c head/sys/geom/mirror/g_mirror.h head/tests/sys/geom/class/mirror/Makefile head/tests/sys/geom/class/mirror/component_selection.sh head/tests/sys/geom/class/mirror/conf.sh
Fixed by (1) ignoring md_all in check_metadata and (2) evicting the stale mirror / refreshing metadata during STARTING. This particular scenario isn't exactly covered by the committed test, so that would be room for improvement. But it is believed to fix the issue.
A commit references this bug: Author: cem Date: Fri Dec 7 02:44:05 UTC 2018 New revision: 341674 URL: https://svnweb.freebsd.org/changeset/base/341674 Log: gmirror: Evaluate mirror components against newest metadata copy Re-apply r341665 with format strings fixed. If we happen to taste a stale mirror component first, don't reject valid, newer components that have differing metadata from the stale component (during STARTING). Instead, update our view of the most recent metadata as we taste components. Like mediasize beforehand, remove some checks from g_mirror_check_metadata which would evict valid components due to metadata that can change over a mirror's lifetime. g_mirror_check_metadata is invoked long before we check genid/syncid and decide which component(s) are newest and whether or not we have quorum. Before checking if we can enter RUNNING (i.e., we have quorum) after a NEW component is added, first remove any known stale or inconsistent disks from the mirrorset, rather than removing them *after* deciding we have quorum. Check if we have quorum after removing these components. Additionally, add a knob, kern.geom.mirror.launch_mirror_before_timeout, to force gmirrors to wait out the full timeout (kern.geom.mirror.timeout) before transitioning from STARTING to RUNNING. This is a kludge to help ensure all eligible, boot-time available mirror components are tasted before RUNNING a gmirror. Add a basic test case for STARTING -> RUNNING startup behavior around stale genids. PR: 232671, 232835 Submitted by: Cindy Yang <cyang AT isilon.com> (previous version) Reviewed by: markj (kernel portions) Discussed with: asomers, Cindy Yang Tested by: pho Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D18062 Changes: head/sys/geom/mirror/g_mirror.c head/sys/geom/mirror/g_mirror.h head/tests/sys/geom/class/mirror/Makefile head/tests/sys/geom/class/mirror/component_selection.sh head/tests/sys/geom/class/mirror/conf.sh