kernel: GEOM_MIRROR (error=45) while trying to use ZFS volumes (ZVOL) geoms as a geom_mirror component. How-To-Repeat: # zfs create -V10G tank0/vol0 # diskinfo -v /dev/zvol/tank0/vol0 /dev/zvol/tank0/vol0 512 # sectorsize 10737418240 # mediasize in bytes (10G) 20971520 # mediasize in sectors 0 # stripesize 0 # stripeoffset # gmirror label mirror0 /dev/zvol/tank0/vol0 GEOM_MIRROR: Cannot add disk zvol/tank0/vol0 to mirror0 (error=45). GEOM_MIRROR: Device mirror0 destroyed. gmirror list is empty and gmirror list -a shows: Geom name: mirror0 Consumers: 1. Name: zvol/tank0/vol0 Mediasize: 10737418240 (10G) Sectorsize: 512 Mode: r1w1e1 Right after the sequence above, the block device zvol/tank0/vol0 is not any more available for writing it (seems to be locked by geom_mirror) # dd if=/dev/zero of=/dev/zvol/tank0/vol0 dd: /dev/zvol/tank0/vol0: Operation not permitted And i was not able to release it. # gmirror forget mirror0 gmirror: No such device: mirror0. # gmirror clear /dev/zvol/tank0/vol0 Can't clear metadata on /dev/zvol/tank0/vol0: Operation not permitted. gmirror: Not fully done. # zfs destroy -f tank0/vol0 cannot destroy 'tank0/vol0': dataset is busy The only way to destroy ZVOL is temporarily block geaom_morror loading during the boot process, reboot and destroy zvols.
In advance, i was able to do that on 9.0-RELEASE and earlier ZFS aware kernels.
Here is the shell commands sequence and kernel log with kern.geom.mirror.debug=65535 # zfs create -V1G tank0/vol0 Jan 15 17:17:11 fresh-inst kernel: GEOM_MIRROR[2]: Tasting zvol/tank0/vol0. # gmirror label mirror0 /dev/zvol/tank0/vol0 Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[2]: Tasting zvol/tank0/vol0. Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[2]: Tasting zvol/tank0/vol0. Jan 15 17:19:54 fresh-inst kernel: magic: GEOM::MIRROR Jan 15 17:19:54 fresh-inst kernel: version: 4 Jan 15 17:19:54 fresh-inst kernel: name: mirror0 Jan 15 17:19:54 fresh-inst kernel: mid: 2502644825 Jan 15 17:19:54 fresh-inst kernel: did: 1157435422 Jan 15 17:19:54 fresh-inst kernel: all: 1 Jan 15 17:19:54 fresh-inst kernel: genid: 0 Jan 15 17:19:54 fresh-inst kernel: syncid: 1 Jan 15 17:19:54 fresh-inst kernel: priority: 0 Jan 15 17:19:54 fresh-inst kernel: slice: 4096 Jan 15 17:19:54 fresh-inst kernel: balance: load Jan 15 17:19:54 fresh-inst kernel: mediasize: 1073741312 Jan 15 17:19:54 fresh-inst kernel: sectorsize: 512 Jan 15 17:19:54 fresh-inst kernel: syncoffset: 0 Jan 15 17:19:54 fresh-inst kernel: mflags: NONE Jan 15 17:19:54 fresh-inst kernel: dflags: NONE Jan 15 17:19:54 fresh-inst kernel: hcprovider: Jan 15 17:19:54 fresh-inst kernel: provsize: 1073741824 Jan 15 17:19:54 fresh-inst kernel: MD5 hash: 8af33e03530d3297e18ee99e9d402feb Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[1]: Creating device mirror0 (id=2502644825). Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRRORGEOM_MIRROR[1][5] Jan 15 17:19:54 fresh-inst kernel: : : Jan 15 17:19:54 fresh-inst kernel: Device mirror0 created (1 components, id=2502644825). Jan 15 17:19:54 fresh-inst kernel: g_mirror_worker: Let's see... Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRRORGEOM_MIRROR[1][4]: : root_mount_hold 0 Jan 15 17:19:54 fresh-inst kernel: g_mirror_worker: Sleeping 0xfffffe000e918600. Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[1]: Adding disk zvol/tank0/vol0 to mirror0. Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[2]: Adding disk zvol/tank0/vol0. Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[2]: Disk zvol/tank0/vol0 connected. Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[0]: Cannot add disk zvol/tank0/vol0 to mirror0 (error=45). Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[4]: g_mirror_destroy: Waking up 0xfffffe000e918600. Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRRORGEOM_MIRROR[4][4]: : g_mirror_destroy: Sleeping 0xfffffe000e9186a0. Jan 15 17:19:54 fresh-inst kernel: g_mirror_worker: Woken up 0xfffffe000e918600. Jan 15 17:19:55 fresh-inst kernel: GEOM_MIRROR[5]: Jan 15 17:19:55 fresh-inst kernel: g_mirror_worker: I'm here 4. Jan 15 17:19:55 fresh-inst kernel: GEOM_MIRROR[5]: g_mirror_worker: Let's see... Jan 15 17:19:55 fresh-inst kernel: GEOM_MIRROR[4]: g_mirror_try_destroy: Waking up 0xfffffe000e9186a0. Jan 15 17:19:55 fresh-inst kernel: GEOM_MIRRORGEOM_MIRROR[1][4]: : Thread exiting. Jan 15 17:19:55 fresh-inst kernel: g_mirror_destroy: Woken up 0xfffffe000e9186a0. Jan 15 17:19:55 fresh-inst kernel: GEOM_MIRROR[0]: Device mirror0 destroyed.
This > GEOM_MIRROR: Cannot add disk zvol/tank0/vol0 to mirror0 (error=45). seems to be triggered by the following code in sys/geom/mirror/g_mirror.c: error = g_getattr("GEOM::candelete", disk->d_consumer, &i); if (error != 0) goto fail; plus the fact that ZFS zvol does the following: case BIO_GETATTR: case BIO_DELETE: default: g_io_deliver(bp, EOPNOTSUPP); break; Perhaps, the gmirror code should be more flexible with respect to EOPNOTSUPP from g_getattr? -- Andriy Gapon
Andriy, many thanks for the suggestion. I am trying to rebuild and test the kernel with the following patch: --- sys/geom/mirror/g_mirror.c (revision 245741) +++ sys/geom/mirror/g_mirror.c (working copy) @@ -456,7 +456,7 @@ disk->d_flags = md->md_dflags; error = g_getattr("GEOM::candelete", disk->d_consumer, &i); if (error != 0) - goto fail; + i=0; if (i) disk->d_flags |= G_MIRROR_DISK_FLAG_CANDELETE; if (md->md_provider[0] != '\0') It seems to correct if assume G_MIRROR_DISK_FLAG_CANDELETE is always off unless successfully reported by GEOM::candelete attribute. -- Best regards 21.01.2013 13:47, Andriy Gapon пиÑеÑ: > This >> GEOM_MIRROR: Cannot add disk zvol/tank0/vol0 to mirror0 (error=45). > seems to be triggered by the following code in sys/geom/mirror/g_mirror.c: > > error = g_getattr("GEOM::candelete", disk->d_consumer, &i); > if (error != 0) > goto fail; > > plus the fact that ZFS zvol does the following: > case BIO_GETATTR: > case BIO_DELETE: > default: > g_io_deliver(bp, EOPNOTSUPP); > break; > > Perhaps, the gmirror code should be more flexible with respect to EOPNOTSUPP > from g_getattr? >
Eventually this patch works for me just fine. --- sys/geom/mirror/g_mirror.c (revision 245741) +++ sys/geom/mirror/g_mirror.c (working copy) @@ -456,7 +456,7 @@ disk->d_flags = md->md_dflags; error = g_getattr("GEOM::candelete", disk->d_consumer, &i); if (error != 0) - goto fail; + i=0; if (i) disk->d_flags |= G_MIRROR_DISK_FLAG_CANDELETE; if (md->md_provider[0] != '\0')
Author: avg Date: Sat Jan 26 10:50:04 2013 New Revision: 245946 URL: http://svnweb.freebsd.org/changeset/base/245946 Log: g_mirror: g_getattr() failure should not be fatal This allows to use gmirror e.g. on top of ZVOLs. PR: kern/175323 Submitted by: Alexei.Volkov@softlynx.ru, mav Reported by: Alexei.Volkov@softlynx.ru Tested by: Alexei.Volkov@softlynx.ru Reviewed by: ae, mav, pjd MFC after: 1 week Modified: head/sys/geom/mirror/g_mirror.c Modified: head/sys/geom/mirror/g_mirror.c ============================================================================== --- head/sys/geom/mirror/g_mirror.c Sat Jan 26 10:34:17 2013 (r245945) +++ head/sys/geom/mirror/g_mirror.c Sat Jan 26 10:50:04 2013 (r245946) @@ -457,9 +457,7 @@ g_mirror_init_disk(struct g_mirror_softc disk->d_priority = md->md_priority; disk->d_flags = md->md_dflags; error = g_getattr("GEOM::candelete", disk->d_consumer, &i); - if (error != 0) - goto fail; - if (i) + if (error == 0 && i != 0) disk->d_flags |= G_MIRROR_DISK_FLAG_CANDELETE; if (md->md_provider[0] != '\0') disk->d_flags |= G_MIRROR_DISK_FLAG_HARDCODED; _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
Responsible Changed From-To: freebsd-bugs->avg I am handling this one.
State Changed From-To: open->patched Fixed in head. This fix makes g_getattr("GEOM::candelete") failure non-fatal and that also fixes a GEOM access bits leak.
Author: avg Date: Sat Feb 2 11:30:53 2013 New Revision: 246241 URL: http://svnweb.freebsd.org/changeset/base/246241 Log: MFC r245946: g_mirror: g_getattr() failure should not be fatal PR: kern/175323 Modified: stable/9/sys/geom/mirror/g_mirror.c Directory Properties: stable/9/sys/ (props changed) Modified: stable/9/sys/geom/mirror/g_mirror.c ============================================================================== --- stable/9/sys/geom/mirror/g_mirror.c Sat Feb 2 11:25:10 2013 (r246240) +++ stable/9/sys/geom/mirror/g_mirror.c Sat Feb 2 11:30:53 2013 (r246241) @@ -456,9 +456,7 @@ g_mirror_init_disk(struct g_mirror_softc disk->d_priority = md->md_priority; disk->d_flags = md->md_dflags; error = g_getattr("GEOM::candelete", disk->d_consumer, &i); - if (error != 0) - goto fail; - if (i) + if (error == 0 && i != 0) disk->d_flags |= G_MIRROR_DISK_FLAG_CANDELETE; if (md->md_provider[0] != '\0') disk->d_flags |= G_MIRROR_DISK_FLAG_HARDCODED; _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
State Changed From-To: patched->closed Fixed in stable/9. 8 is not affected.