Bug 230246 - GPT label mishandling
Summary: GPT label mishandling
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-geom mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-07-31 18:38 UTC by Mason Loring Bliss
Modified: 2019-09-06 20:30 UTC (History)
3 users (show)

See Also:


Attachments
Midway through... (36.67 KB, text/plain)
2019-09-06 02:02 UTC, Mason Loring Bliss
no flags Details
confdot from previous comment, as file to avoid munging (14.67 KB, text/plain)
2019-09-06 02:10 UTC, Mason Loring Bliss
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mason Loring Bliss 2018-07-31 18:38:32 UTC
Observed:

root@:~ # gmirror label -v efi gpt/freebsd-efi0 gpt/freebsd-efi1
Metadata value stored on gpt/freebsd-efi0.
Metadata value stored on gpt/freebsd-efi1.
Done.
root@:~ # ls /dev/gpt
ls: /dev/gpt: No such file or directory
root@:~ # gmirror destroy efi
root@:~ # ls /dev/gpt
freebsd-boot0	freebsd-efi1	freebsd-swap0	gentoo-efi1
freebsd-boot1	freebsd-root0	freebsd-swap1	gentoo-root0
freebsd-efi0	freebsd-root1	gentoo-efi0	gentoo-root1
root@:~ # gmirror label -v efi gpt/freebsd-efi0 gpt/freebsd-efi1
Metadata value stored on gpt/freebsd-efi0.
Metadata value stored on gpt/freebsd-efi1.
Done.
root@:~ # ls /dev/gpt
ls: /dev/gpt: No such file or directory
root@:~ #

with kern.geom.mirror.debug=1:

GEOM_MIRROR: Device mirror/efi launched (2/2).
GEOM_MIRROR: Device efi: provider destroyed.
GEOM_MIRROR: Device efi destroyed.
GEOM_MIRROR: Device mirror/efi launched (2/2).
GEOM_MIRROR: Device efi: provider destroyed.
GEOM_MIRROR: Device efi destroyed.
GEOM_MIRROR: Device mirror/efi launched (2/2).
GEOM_MIRROR: Device efi: provider destroyed.
GEOM_MIRROR: Device efi destroyed.
GEOM_MIRROR: Device mirror/efi launched (2/2).
GEOM_MIRROR: Device efi: provider destroyed.
GEOM_MIRROR: Device efi destroyed.
GEOM_MIRROR: Device mirror/efi launched (2/2).
GEOM_MIRROR: Device efi: provider destroyed.
GEOM_MIRROR: Device efi destroyed.
GEOM_MIRROR[1]: Creating device efi (id=426299166).
GEOM_MIRROR[1]: Device efi created (2 components, id=426299166).
GEOM_MIRROR[1]: root_mount_hold 0xfffff80008aa8540
GEOM_MIRROR[1]: Adding disk diskid/DISK-CVDA5066005L1207GNp1 to efi.
GEOM_MIRROR[1]: Disk diskid/DISK-CVDA5066005L1207GNp1 state changed from NONE to NEW (device efi).
GEOM_MIRROR[1]: Device efi: provider diskid/DISK-CVDA5066005L1207GNp1 detected.
GEOM_MIRROR[1]: Adding disk diskid/DISK-PNY20162178980100273p1 to efi.
GEOM_MIRROR[1]: Disk diskid/DISK-PNY20162178980100273p1 state changed from NONE to NEW (device efi).
GEOM_MIRROR[1]: Device efi: provider diskid/DISK-PNY20162178980100273p1 detected.
GEOM_MIRROR[1]: Device efi state changed from STARTING to RUNNING.
GEOM_MIRROR[1]: Disk diskid/DISK-PNY20162178980100273p1 state changed from NEW to ACTIVE (device efi).
GEOM_MIRROR[1]: Device efi: provider diskid/DISK-PNY20162178980100273p1 activated.
GEOM_MIRROR[1]: Disk diskid/DISK-CVDA5066005L1207GNp1 state changed from NEW to ACTIVE (device efi).
GEOM_MIRROR[1]: Device efi: provider diskid/DISK-CVDA5066005L1207GNp1 activated.
GEOM_MIRROR[0]: Device mirror/efi launched (2/2).
GEOM_MIRROR[1]: root_mount_rel[2570] 0xfffff80008aa8540

markj on efnet suggested adding the partitions without referencing gpt labels, and on doing that I could make the mirror without losing /dev/gpt.
Comment 1 Mason Loring Bliss 2018-08-26 19:37:50 UTC
Something I've observed is that I don't see this on fresh disks. I only see this when I'm installing to a disk that formerly had gmirrors on it.
Comment 2 Andrey V. Elsukov freebsd_committer 2018-09-11 16:18:21 UTC
For me all looks like I expected. What do you expected to see after these commands?
Comment 3 Mason Loring Bliss 2019-07-27 00:55:51 UTC
Persists on 12. Typical encounter for me is a server that was running FreeBSD and
subsequently moved to GNU/Linux. Redeploying FreeBSD, evidently spoor from the
old install remains, and when I kldload geom_mirror, a degraded mirror is raised
from the dead. Evidently there's some sort of in-memory corruption after this as removing the remnants gives me assertion errors, and I'm not able to proceed
until I clear memory with a reboot.

I haven't read or traced the code yet, but at a guess something's seeing enough
indication of the old mirror to assume there's a mirror component there, and
then trusting that additional on-disk data is valid when it's not. This would
be pretty consistent with an assertion firing.

If life permits, I'll learn enough to track down just what's happening.
Comment 4 Mason Loring Bliss 2019-09-03 16:06:28 UTC
Still here today, still biting me:

# ls /dev/gpt
efi0    efi1    swap0   swap1   tank0   tank1
# gmirror label -v -b prefer efi gpt/efi0 gpt/efi1
Metadata value stored on gpt/efi0.
Metadata value stored on gpt/efi1.
Done.
# ls /dev/gpt
tank0   tank1
# gmirror destroy efi
# ls /dev/gpt
efi0    efi1    swap0   swap1   tank0   tank1
Comment 5 andrew 2019-09-05 11:38:07 UTC
Can you attach the sysctl kern.geom.confdot output at all three stages? also the output of geom part show  on all relevant physical disks
Comment 6 Mason Loring Bliss 2019-09-05 14:56:09 UTC
I'll gather this, but it'll take some churn, as I work past it when I hit it by
dd'ing zeroes onto the disks (or, the effected partitions, anyway) and
restarting. So I'll have to get the system back into the right state.

Amusingly, I thought maybe I could save time and reproduce this on a VM, and that
provided a quick reminder that I need to open a bug about the graphical console
going away with the 12.0 install media. (It looks like a freeze until you check
the serial console, and see that it's switched over to it. This appears under
libvirt.)

I'll try to have the requested debugging data by this evening, EST.
Comment 7 Mason Loring Bliss 2019-09-06 02:02:43 UTC
Created attachment 207220 [details]
Midway through...

Midway through...
Comment 8 Mason Loring Bliss 2019-09-06 02:08:08 UTC
And then:

# sysctl kern.geom.confdot
kern.geom.confdot: digraph geom {
z0xfffff80008ee6d00 [shape=box,label="MIRROR\nswap.sync\nr#1"];
z0xfffff80003635c00 [shape=box,label="MIRROR\nswap\nr#4"];
z0xfffff80003653580 [label="r1w1e1"];
z0xfffff80003653580 -> z0xfffff80008ee5100;
z0xfffff80003635c00 -> z0xfffff80003653580;
z0xfffff80008c63300 [shape=hexagon,label="mirror/swap\nr0w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff80008c63300 -> z0xfffff80003635c00;
z0xfffff80008c60000 [shape=box,label="PART\ndiskid/DISK-PNY20162178980100273\nr#3"];
z0xfffff800088d4c00 [label="r1w1e2"];
z0xfffff800088d4c00 -> z0xfffff8000336c400;
z0xfffff80008c60000 -> z0xfffff800088d4c00;
z0xfffff80008ee4e00 [shape=hexagon,label="diskid/DISK-PNY20162178980100273p3\nr0w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff80008ee4e00 -> z0xfffff80008c60000;
z0xfffff80008ee5100 [shape=hexagon,label="diskid/DISK-PNY20162178980100273p2\nr1w1e1\nerr#0\nsector=512\nstripe=0"];
z0xfffff80008ee5100 -> z0xfffff80008c60000;
z0xfffff80008c62c00 [shape=hexagon,label="diskid/DISK-PNY20162178980100273p1\nr0w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff80008c62c00 -> z0xfffff80008c60000;
z0xfffff8005704e900 [shape=box,label="PART\ndiskid/DISK-CVDA5066005L1207GN\nr#3"];
z0xfffff8005752f100 [label="r0w0e0"];
z0xfffff8005752f100 -> z0xfffff800087f3c00;
z0xfffff8005704e900 -> z0xfffff8005752f100;
z0xfffff80008886700 [shape=hexagon,label="diskid/DISK-CVDA5066005L1207GNp3\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff80008886700 -> z0xfffff8005704e900;
z0xfffff800088e3b00 [shape=hexagon,label="diskid/DISK-CVDA5066005L1207GNp2\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff800088e3b00 -> z0xfffff8005704e900;
z0xfffff8000336c800 [shape=hexagon,label="diskid/DISK-CVDA5066005L1207GNp1\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff8000336c800 -> z0xfffff8005704e900;
z0xfffff80008886000 [shape=box,label="PART\nada0\nr#2"];
z0xfffff80008827c00 [label="r0w0e0"];
z0xfffff80008827c00 -> z0xfffff8000336c200;
z0xfffff80008886000 -> z0xfffff80008827c00;
z0xfffff800087f3900 [shape=hexagon,label="ada0p3\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff800087f3900 -> z0xfffff80008886000;
z0xfffff80008886500 [shape=hexagon,label="ada0p2\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff80008886500 -> z0xfffff80008886000;
z0xfffff8000889b900 [shape=hexagon,label="ada0p1\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff8000889b900 -> z0xfffff80008886000;
z0xfffff800088e5800 [shape=box,label="PART\ndiskid/DISK-110074973765s2\nr#4"];
z0xfffff80003654880 [label="r0w0e0"];
z0xfffff80003654880 -> z0xfffff800088e4000;
z0xfffff800088e5800 -> z0xfffff80003654880;
z0xfffff8000888a100 [shape=hexagon,label="diskid/DISK-110074973765s2a\nr0w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff8000888a100 -> z0xfffff800088e5800;
z0xfffff800088e6100 [shape=box,label="PART\ndiskid/DISK-110074973765\nr#3"];
z0xfffff800088d7980 [label="r0w0e0"];
z0xfffff800088d7980 -> z0xfffff800088e4200;
z0xfffff800088e6100 -> z0xfffff800088d7980;
z0xfffff800088e4000 [shape=hexagon,label="diskid/DISK-110074973765s2\nr0w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff800088e4000 -> z0xfffff800088e6100;
z0xfffff800088e5e00 [shape=hexagon,label="diskid/DISK-110074973765s1\nr0w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff800088e5e00 -> z0xfffff800088e6100;
z0xfffff800087f4800 [shape=box,label="PART\nda0s2\nr#3"];
z0xfffff80003363d80 [label="r1w0e1"];
z0xfffff80003363d80 -> z0xfffff800035ac500;
z0xfffff800087f4800 -> z0xfffff80003363d80;
z0xfffff80008898c00 [shape=hexagon,label="da0s2a\nr1w0e1\nerr#0\nsector=512\nstripe=0"];
z0xfffff80008898c00 -> z0xfffff800087f4800;
z0xfffff800088d2e00 [shape=box,label="PART\nda0\nr#2"];
z0xfffff8000335f700 [label="r1w0e1"];
z0xfffff8000335f700 -> z0xfffff80008899a00;
z0xfffff800088d2e00 -> z0xfffff8000335f700;
z0xfffff800035ac500 [shape=hexagon,label="da0s2\nr1w0e1\nerr#0\nsector=512\nstripe=0"];
z0xfffff800035ac500 -> z0xfffff800088d2e00;
z0xfffff80008899700 [shape=hexagon,label="da0s1\nr0w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff80008899700 -> z0xfffff800088d2e00;
z0xfffff80057036100 [shape=box,label="LABEL\nada1\nr#2"];
z0xfffff800088d4e00 [label="r1w1e3"];
z0xfffff800088d4e00 -> z0xfffff80008886900;
z0xfffff80057036100 -> z0xfffff800088d4e00;
z0xfffff8000336c400 [shape=hexagon,label="diskid/DISK-PNY20162178980100273\nr1w1e2\nerr#0\nsector=512\nstripe=0"];
z0xfffff8000336c400 -> z0xfffff80057036100;
z0xfffff80057036700 [shape=box,label="LABEL\nada0\nr#2"];
z0xfffff800088d5380 [label="r0w0e0"];
z0xfffff800088d5380 -> z0xfffff8000336c200;
z0xfffff80057036700 -> z0xfffff800088d5380;
z0xfffff800087f3c00 [shape=hexagon,label="diskid/DISK-CVDA5066005L1207GN\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff800087f3c00 -> z0xfffff80057036700;
z0xfffff8005704eb00 [shape=box,label="LABEL\nada0p3\nr#3"];
z0xfffff80008828200 [label="r0w0e0"];
z0xfffff80008828200 -> z0xfffff800087f3900;
z0xfffff8005704eb00 -> z0xfffff80008828200;
z0xfffff8005704ea00 [shape=hexagon,label="gptid/3f9109f8-d049-11e9-8cd7-0021ccb4f63d\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff8005704ea00 -> z0xfffff8005704eb00;
z0xfffff8005704ee00 [shape=box,label="LABEL\nada0p3\nr#3"];
z0xfffff800088d6000 [label="r0w0e0"];
z0xfffff800088d6000 -> z0xfffff800087f3900;
z0xfffff8005704ee00 -> z0xfffff800088d6000;
z0xfffff8005704ed00 [shape=hexagon,label="gpt/tank0\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff8005704ed00 -> z0xfffff8005704ee00;
z0xfffff80057045800 [shape=box,label="LABEL\nada0p2\nr#3"];
z0xfffff800088d6a80 [label="r0w0e0"];
z0xfffff800088d6a80 -> z0xfffff80008886500;
z0xfffff80057045800 -> z0xfffff800088d6a80;
z0xfffff80057045700 [shape=hexagon,label="gptid/3f882ec4-d049-11e9-8cd7-0021ccb4f63d\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff80057045700 -> z0xfffff80057045800;
z0xfffff80057045b00 [shape=box,label="LABEL\nada0p2\nr#3"];
z0xfffff80003647f00 [label="r0w0e0"];
z0xfffff80003647f00 -> z0xfffff80008886500;
z0xfffff80057045b00 -> z0xfffff80003647f00;
z0xfffff80057045a00 [shape=hexagon,label="gpt/swap0\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff80057045a00 -> z0xfffff80057045b00;
z0xfffff80008c61a00 [shape=box,label="LABEL\nada0p1\nr#3"];
z0xfffff800088d6a00 [label="r0w0e0"];
z0xfffff800088d6a00 -> z0xfffff8000889b900;
z0xfffff80008c61a00 -> z0xfffff800088d6a00;
z0xfffff80057036800 [shape=hexagon,label="gptid/3f64646b-d049-11e9-8cd7-0021ccb4f63d\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff80057036800 -> z0xfffff80008c61a00;
z0xfffff80008c60b00 [shape=box,label="LABEL\nada0p1\nr#3"];
z0xfffff800088d6980 [label="r0w0e0"];
z0xfffff800088d6980 -> z0xfffff8000889b900;
z0xfffff80008c60b00 -> z0xfffff800088d6980;
z0xfffff80008c60200 [shape=hexagon,label="gpt/efi0\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff80008c60200 -> z0xfffff80008c60b00;
z0xfffff8000888a600 [shape=box,label="LABEL\ndiskid/DISK-110074973765s2a\nr#5"];
z0xfffff800088d7800 [label="r0w0e0"];
z0xfffff800088d7800 -> z0xfffff8000888a100;
z0xfffff8000888a600 -> z0xfffff800088d7800;
z0xfffff8000330ec00 [shape=hexagon,label="ufsid/5c0a029708f9d844\nr0w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff8000330ec00 -> z0xfffff8000888a600;
z0xfffff80008899400 [shape=box,label="LABEL\nda0\nr#2"];
z0xfffff8000335c880 [label="r0w0e0"];
z0xfffff8000335c880 -> z0xfffff80008899a00;
z0xfffff80008899400 -> z0xfffff8000335c880;
z0xfffff800088e4200 [shape=hexagon,label="diskid/DISK-110074973765\nr0w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff800088e4200 -> z0xfffff80008899400;
z0xfffff8000888a400 [shape=box,label="LABEL\nda0s2a\nr#4"];
z0xfffff80003654a80 [label="r1w0e1"];
z0xfffff80003654a80 -> z0xfffff80008898c00;
z0xfffff8000888a400 -> z0xfffff80003654a80;
z0xfffff8000888a300 [shape=hexagon,label="ufs/FreeBSD_Install\nr1w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff8000888a300 -> z0xfffff8000888a400;
z0xfffff80008895700 [shape=box,label="LABEL\nda0s1\nr#3"];
z0xfffff80003654b80 [label="r0w0e0"];
z0xfffff80003654b80 -> z0xfffff80008899700;
z0xfffff80008895700 -> z0xfffff80003654b80;
z0xfffff80008895100 [shape=hexagon,label="msdosfs/EFISYS\nr0w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff80008895100 -> z0xfffff80008895700;
z0xfffff800088e5000 [shape=box,label="VFS\nffs.ufs/FreeBSD_Install\nr#5"];
z0xfffff80003363e00 [label="r1w0e0"];
z0xfffff80003363e00 -> z0xfffff8000888a300;
z0xfffff800088e5000 -> z0xfffff80003363e00;
z0xfffff80008899b00 [shape=box,label="DISK\nda0\nr#1"];
z0xfffff80008899a00 [shape=hexagon,label="da0\nr1w0e1\nerr#0\nsector=512\nstripe=0"];
z0xfffff80008899a00 -> z0xfffff80008899b00;
z0xfffff80008886a00 [shape=box,label="DISK\nada1\nr#1"];
z0xfffff80008886900 [shape=hexagon,label="ada1\nr1w1e3\nerr#0\nsector=512\nstripe=0"];
z0xfffff80008886900 -> z0xfffff80008886a00;
z0xfffff8000336c300 [shape=box,label="DISK\nada0\nr#1"];
z0xfffff8000336c200 [shape=hexagon,label="ada0\nr0w0e0\nerr#0\nsector=512\nstripe=4096"];
z0xfffff8000336c200 -> z0xfffff8000336c300;
z0xfffff80057036600 [shape=box,label="DEV\nmirror/swap\nr#5"];
z0xfffff8005752f380 [label="r0w0e0"];
z0xfffff8005752f380 -> z0xfffff80008c63300;
z0xfffff80057036600 -> z0xfffff8005752f380;
z0xfffff80008ee5000 [shape=box,label="DEV\ndiskid/DISK-PNY20162178980100273p3\nr#4"];
z0xfffff8005752e600 [label="r0w0e0"];
z0xfffff8005752e600 -> z0xfffff80008ee4e00;
z0xfffff80008ee5000 -> z0xfffff8005752e600;
z0xfffff80008896000 [shape=box,label="DEV\ndiskid/DISK-PNY20162178980100273p2\nr#4"];
z0xfffff800088d4980 [label="r0w0e0"];
z0xfffff800088d4980 -> z0xfffff80008ee5100;
z0xfffff80008896000 -> z0xfffff800088d4980;
z0xfffff80008ee5300 [shape=box,label="DEV\ndiskid/DISK-PNY20162178980100273p1\nr#4"];
z0xfffff800088d4b00 [label="r0w0e0"];
z0xfffff800088d4b00 -> z0xfffff80008c62c00;
z0xfffff80008ee5300 -> z0xfffff800088d4b00;
z0xfffff80008ee4c00 [shape=box,label="DEV\ndiskid/DISK-PNY20162178980100273\nr#3"];
z0xfffff8005752e900 [label="r0w0e0"];
z0xfffff8005752e900 -> z0xfffff8000336c400;
z0xfffff80008ee4c00 -> z0xfffff8005752e900;
z0xfffff80008898700 [shape=box,label="DEV\ndiskid/DISK-CVDA5066005L1207GNp3\nr#4"];
z0xfffff800088d5100 [label="r0w0e0"];
z0xfffff800088d5100 -> z0xfffff80008886700;
z0xfffff80008898700 -> z0xfffff800088d5100;
z0xfffff8000889a600 [shape=box,label="DEV\ndiskid/DISK-CVDA5066005L1207GNp2\nr#4"];
z0xfffff8005752ec00 [label="r0w0e0"];
z0xfffff8005752ec00 -> z0xfffff800088e3b00;
z0xfffff8000889a600 -> z0xfffff8005752ec00;
z0xfffff80008c62e00 [shape=box,label="DEV\ndiskid/DISK-CVDA5066005L1207GNp1\nr#4"];
z0xfffff8005752ee00 [label="r0w0e0"];
z0xfffff8005752ee00 -> z0xfffff8000336c800;
z0xfffff80008c62e00 -> z0xfffff8005752ee00;
z0xfffff80008ee5200 [shape=box,label="DEV\ndiskid/DISK-CVDA5066005L1207GN\nr#3"];
z0xfffff8005752f000 [label="r0w0e0"];
z0xfffff8005752f000 -> z0xfffff800087f3c00;
z0xfffff80008ee5200 -> z0xfffff8005752f000;
z0xfffff8005704ec00 [shape=box,label="DEV\ngptid/3f9109f8-d049-11e9-8cd7-0021ccb4f63d\nr#4"];
z0xfffff800088d5300 [label="r0w0e0"];
z0xfffff800088d5300 -> z0xfffff8005704ea00;
z0xfffff8005704ec00 -> z0xfffff800088d5300;
z0xfffff800088e3d00 [shape=box,label="DEV\ngpt/tank0\nr#4"];
z0xfffff80003647880 [label="r0w0e0"];
z0xfffff80003647880 -> z0xfffff8005704ed00;
z0xfffff800088e3d00 -> z0xfffff80003647880;
z0xfffff800035acc00 [shape=box,label="DEV\nada0p3\nr#3"];
z0xfffff800088d5b80 [label="r0w0e0"];
z0xfffff800088d5b80 -> z0xfffff800087f3900;
z0xfffff800035acc00 -> z0xfffff800088d5b80;
z0xfffff80057045900 [shape=box,label="DEV\ngptid/3f882ec4-d049-11e9-8cd7-0021ccb4f63d\nr#4"];
z0xfffff80003361680 [label="r0w0e0"];
z0xfffff80003361680 -> z0xfffff80057045700;
z0xfffff80057045900 -> z0xfffff80003361680;
z0xfffff80003635b00 [shape=box,label="DEV\ngpt/swap0\nr#4"];
z0xfffff80003653180 [label="r0w0e0"];
z0xfffff80003653180 -> z0xfffff80057045a00;
z0xfffff80003635b00 -> z0xfffff80003653180;
z0xfffff8000336a600 [shape=box,label="DEV\nada0p2\nr#3"];
z0xfffff80003653380 [label="r0w0e0"];
z0xfffff80003653380 -> z0xfffff80008886500;
z0xfffff8000336a600 -> z0xfffff80003653380;
z0xfffff80008c60400 [shape=box,label="DEV\ngptid/3f64646b-d049-11e9-8cd7-0021ccb4f63d\nr#4"];
z0xfffff800088d6500 [label="r0w0e0"];
z0xfffff800088d6500 -> z0xfffff80057036800;
z0xfffff80008c60400 -> z0xfffff800088d6500;
z0xfffff8000336a900 [shape=box,label="DEV\ngpt/efi0\nr#4"];
z0xfffff800088d6780 [label="r0w0e0"];
z0xfffff800088d6780 -> z0xfffff80008c60200;
z0xfffff8000336a900 -> z0xfffff800088d6780;
z0xfffff800087f4000 [shape=box,label="DEV\nada0p1\nr#3"];
z0xfffff800088d6800 [label="r0w0e0"];
z0xfffff800088d6800 -> z0xfffff8000889b900;
z0xfffff800087f4000 -> z0xfffff800088d6800;
z0xfffff80008898500 [shape=box,label="DEV\nufsid/5c0a029708f9d844\nr#6"];
z0xfffff80003654780 [label="r0w0e0"];
z0xfffff80003654780 -> z0xfffff8000330ec00;
z0xfffff80008898500 -> z0xfffff80003654780;
z0xfffff800088e5d00 [shape=box,label="DEV\ndiskid/DISK-110074973765s2a\nr#5"];
z0xfffff800088d7880 [label="r0w0e0"];
z0xfffff800088d7880 -> z0xfffff8000888a100;
z0xfffff800088e5d00 -> z0xfffff800088d7880;
z0xfffff800088e6000 [shape=box,label="DEV\ndiskid/DISK-110074973765s2\nr#4"];
z0xfffff8000335c580 [label="r0w0e0"];
z0xfffff8000335c580 -> z0xfffff800088e4000;
z0xfffff800088e6000 -> z0xfffff8000335c580;
z0xfffff800088e4100 [shape=box,label="DEV\ndiskid/DISK-110074973765s1\nr#4"];
z0xfffff80003654980 [label="r0w0e0"];
z0xfffff80003654980 -> z0xfffff800088e5e00;
z0xfffff800088e4100 -> z0xfffff80003654980;
z0xfffff800088e6200 [shape=box,label="DEV\ndiskid/DISK-110074973765\nr#3"];
z0xfffff80003363580 [label="r0w0e0"];
z0xfffff80003363580 -> z0xfffff800088e4200;
z0xfffff800088e6200 -> z0xfffff80003363580;
z0xfffff8000888a500 [shape=box,label="DEV\nufs/FreeBSD_Install\nr#5"];
z0xfffff8000335cf00 [label="r0w0e0"];
z0xfffff8000335cf00 -> z0xfffff8000888a300;
z0xfffff8000888a500 -> z0xfffff8000335cf00;
z0xfffff80008898d00 [shape=box,label="DEV\nda0s2a\nr#4"];
z0xfffff8000335f200 [label="r0w0e0"];
z0xfffff8000335f200 -> z0xfffff80008898c00;
z0xfffff80008898d00 -> z0xfffff8000335f200;
z0xfffff8000330eb00 [shape=box,label="DEV\nmsdosfs/EFISYS\nr#4"];
z0xfffff80003363b80 [label="r0w0e0"];
z0xfffff80003363b80 -> z0xfffff80008895100;
z0xfffff8000330eb00 -> z0xfffff80003363b80;
z0xfffff80008898a00 [shape=box,label="DEV\nda0s2\nr#3"];
z0xfffff8000335f480 [label="r0w0e0"];
z0xfffff8000335f480 -> z0xfffff800035ac500;
z0xfffff80008898a00 -> z0xfffff8000335f480;
z0xfffff80008899600 [shape=box,label="DEV\nda0s1\nr#3"];
z0xfffff8000335f600 [label="r0w0e0"];
z0xfffff8000335f600 -> z0xfffff80008899700;
z0xfffff80008899600 -> z0xfffff8000335f600;
z0xfffff80008899200 [shape=box,label="DEV\nda0\nr#2"];
z0xfffff8000335f680 [label="r0w0e0"];
z0xfffff8000335f680 -> z0xfffff80008899a00;
z0xfffff80008899200 -> z0xfffff8000335f680;
z0xfffff80008897e00 [shape=box,label="DEV\nada1\nr#2"];
z0xfffff80003655500 [label="r0w0e0"];
z0xfffff80003655500 -> z0xfffff80008886900;
z0xfffff80008897e00 -> z0xfffff80003655500;
z0xfffff800087f3d00 [shape=box,label="DEV\nada0\nr#2"];
z0xfffff80008828500 [label="r0w0e0"];
z0xfffff80008828500 -> z0xfffff8000336c200;
z0xfffff800087f3d00 -> z0xfffff80008828500;
}
Comment 9 Mason Loring Bliss 2019-09-06 02:09:27 UTC
And at this point, the labels for ada1 are missing:

# ls /dev/gpt
efi0	swap0	tank0
Comment 10 Mason Loring Bliss 2019-09-06 02:10:46 UTC
Created attachment 207221 [details]
confdot from previous comment, as file to avoid munging

confdot from previous comment, as file to avoid munging
Comment 11 Mason Loring Bliss 2019-09-06 02:17:56 UTC
# ls /dev/ada
/dev/ada0   /dev/ada0p1 /dev/ada0p2 /dev/ada0p3 /dev/ada1  
# gpart show ada1
gpart: No such geom: ada1.
# gpart show /dev/ada1
gpart: No such geom: /dev/ada1.
Comment 12 Mason Loring Bliss 2019-09-06 02:18:33 UTC
# gmirror destroy swap
# ls /dev/ada*
/dev/ada0	/dev/ada0p2	/dev/ada1	/dev/ada1p2
/dev/ada0p1	/dev/ada0p3	/dev/ada1p1	/dev/ada1p3
Comment 13 Mason Loring Bliss 2019-09-06 02:38:57 UTC
# gmirror list
# ls /dev/gpt
efi0	efi1	swap0	swap1	tank0	tank1
# dd if=/dev/zero of=/dev/gpt/efi0 bs=1m
dd: /dev/gpt/efi0: end of device
2+0 records in
1+0 records out
1048576 bytes transferred in 0.171360 secs (6119140 bytes/sec)
# dd if=/dev/zero of=/dev/gpt/efi1 bs=1m
dd: /dev/gpt/efi1: end of device
2+0 records in
1+0 records out
1048576 bytes transferred in 0.012050 secs (87022279 bytes/sec)
# dd if=/dev/zero of=/dev/gpt/swap0 bs=1m
dd: /dev/gpt/swap0: Operation not supported
Comment 14 Mason Loring Bliss 2019-09-06 02:40:56 UTC
More interesting:

# gmirror list
# ls /dev/gpt
efi0	efi1	swap0	swap1	tank0	tank1
# dd if=/dev/zero of=/dev/gpt/efi0 bs=1m
dd: /dev/gpt/efi0: end of device
2+0 records in
1+0 records out
1048576 bytes transferred in 0.171360 secs (6119140 bytes/sec)
# dd if=/dev/zero of=/dev/gpt/efi1 bs=1m
dd: /dev/gpt/efi1: end of device
2+0 records in
1+0 records out
1048576 bytes transferred in 0.012050 secs (87022279 bytes/sec)
# dd if=/dev/zero of=/dev/gpt/swap0 bs=1m
dd: /dev/gpt/swap0: Operation not supported
# gmirror list
Geom name: swap
State: DEGRADED
Components: 2
Balance: load
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 3028984319
Type: AUTOMATIC
Providers:
1. Name: mirror/swap
   Mediasize: 8589934080 (8.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
Consumers:
1. Name: diskid/DISK-CVDA5066005L1207GNp2
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: (null)
   GenID: 0
   SyncID: 1
   ID: 1191147779

#
Comment 15 andrew 2019-09-06 20:30:50 UTC
Just to wrap this up for the record (I explained all this to the OP in another place):

There's no bug in the code as such, the odd behavior is just an artifact of how geom tasting works, and the fact that it works rather unpredictably when geoms are being created and removed while the system is up (rather than at boot time).

Specifically, the sort of thing that happens is that a partition might become visible via many different provider paths, such as ada0p1, diskid/DISK-NNNNp1, gpt/partname, gptid/someuuid, and then if there is existing metadata at the partition's last sector, in this case for gmirror, the mirror geom might attach to any of the providers. If it attaches (say) diskid/DISK-NNNNp1, then the exclusive open of the underlying disk causes the gpart instances providing ada0p*, gpt/*, etc. to detach, so those labels go away.

This shows up as more of an issue with GPT because the combination of GPT and glabel produces many providers, often inconveniently many. (For example you can end up with "gmirror stop" failing to work, because as soon as the mirror closes, it becomes visible via another provider and immediately starts again.)

So this is ultimately, in my view, a "user experience" issue.