Bug 86388 - [geom] [geom_part] periodic(8) daily should backup gpart(8) partition labels
Summary: [geom] [geom_part] periodic(8) daily should backup gpart(8) partition labels
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: Unspecified
Hardware: Any Any
: Normal Affects Only Me
Assignee: Allan Jude
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2005-09-21 03:20 UTC by Seklecki
Modified: 2020-11-07 04:20 UTC (History)
3 users (show)

See Also:


Attachments
smime.p7s (2.69 KB, application/x-pkcs7-signature)
2006-06-20 14:37 UTC, Brian A. Seklecki
no flags Details
Backup of disk partitions layout (2.57 KB, text/plain)
2017-11-14 17:17 UTC, Miroslav Lachman
no flags Details
Backup of gmirror list detailed info (1.53 KB, text/plain)
2017-11-14 17:18 UTC, Miroslav Lachman
no flags Details
Backup of zpool info, zfs list and zfs properties (1.98 KB, text/plain)
2017-11-14 17:19 UTC, Miroslav Lachman
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Seklecki freebsd_committer 2005-09-21 03:20:05 UTC
      Open/NetBSD have periodic(8) (well, actually /etc/perdiodic) scripts to backup fdisk(8) and disklabel(8) output.  This is exceptionally useful in the event that you need to recover your system from an offsite storage tape backup to another facility on new (identical) hardware.  Without disk partition / slice layout information, you have no idea how your system was originally partitioned.

Of course, depending on whether you're using software or hardware RAID, you might not even know how your RAID was setup.  We also need a way to extract software RAID configurations (vinnum, geom, gmirror, CCD ex.) to a config file reference.  For example, RAIDFrame's raidctl(8)  has a nice "-G" flag:          
                                                                                                                                                                                                                            "-G dev  
Generate the configuration of the RAIDframe device in a                                                             format suitable for use with the -c or -C options."  

There is one caveat: If you're using software RAID CCD, you have your ccd.conf(5) and the output of the component disks` bsdlabel(8) using my solution to determine where on the disk the CCD slice began.

There's no way to backup hardware RAID configuration layouts until the OS has a unified RAID API (see OpenBSD's msic@ list for details on something coming in the next release there).

Fix: 

I wrote an ugly little script for periodic to accomplish this here initially, until someone commits a higher quality one to the fbsd src tree:

The script is viewable at the following URL as well:

http://digitalfreaks.org/~lavalamp/220.backup-bsdlabels


-------------
#!/bin/sh
#
# $FreeBSD: src/etc/periodic/daily/220.backup-bsdlablels******
#

# If there is a global system configuration file, suck it in.
#
if [ -r /etc/defaults/periodic.conf ]
then
    . /etc/defaults/periodic.conf
    source_periodic_confs
fi

case "$daily_backup_bsdlabels_enable" in
    [Yy][Ee][Ss])

    bak=/var/backups

    disks=`sysctl -n kern.disks`

    if [ -z "$disks"  ]; then
        echo '$daily_backup_disklabels_enable" is set but no disk probed by kernel.' \
        "perhaps NFS diskless client."
        rc = 2
    else

        for i in $disks; do

                # first order of business is to check for an existing backup-backup
                if [ -f $bak/fdisk.$i.bak ] ; then
                        rc=1
                        echo "rotating $bak/fdisk.$i.bak"
                        cp -p $bak/fdisk.$i.bak $bak/fdisk.$i.bak2 || rc=3
                fi

                echo "backing up fdisk for $i"

                fdisk $i > "$bak/fdisk.$i.bak" 2>/dev/null || rc=3

                # again exept now we have to get a list of patitions/slices
                # sparc64 can have...9 hopefully slices on a sunlabel?
                part_slices=$(echo /dev/${i}s[0-9])

                for j in $(echo "$part_slices" | sed 's/\/dev\///'); do
                        if [ -f $bak/disklabel.${j}.bak ] ; then
                                rc=1
                                echo "rotating $bak/disklabel.${j}.bak"
                                cp -p $bak/disklabel.${j}.bak $bak/disklabel.${j}.bak2 || rc=3
                        fi

                        echo "backing up disklabel for ${j}"
                        disklabel /dev/${j} > "$bak/disklabel.${j}.bak" 2>/dev/null || rc=3
                done
        done
    fi;;

    *)  rc=0;;
esac
How-To-Repeat:       Sit around for a few weeks every year and brainstorm DRP (Disaster Recovery Planning) scnearios in a 100% FreeBSD environment.
Comment 1 Gleb Smirnoff freebsd_committer 2005-09-21 14:29:37 UTC
On Wed, Sep 21, 2005 at 02:18:51AM +0000, Seklecki@freebsd.org wrote:
S> >How-To-Repeat:
S>       Sit around for a few weeks every year and brainstorm DRP (Disaster Recovery Planning) scnearios in a 100% FreeBSD environment.

I would argue that storing bsdlabel(8), fdisk(8) or other storage configuration
information on the storage itself is the best idea.

-- 
Totus tuus, Glebius.
GLEBIUS-RIPN GLEB-RIPE
Comment 2 Maxim Konovalov 2005-09-21 15:20:24 UTC
[...]
>  I would argue that storing bsdlabel(8), fdisk(8) or other storage
>  configuration information on the storage itself is the best idea.

The submitter said:

% This is exceptionally useful in the event that you need to recover
% your system from an offsite storage tape backup to another facility
% on new (identical) hardware.

Personally I'm always trying to create a copy of fdisk/disklabel and
backup them when install new machine but it is very easy to forget
about that procedure when you add a disk or change its layout.  I
helped to other people to repair their disklabels several times
already and from time to time see this question in FreeBSD maillists.
There are several tools for partitions/slices restore in ports.  Even
one is under src/. Believe me, this information saves a lot of man
hours and nerve and Net|OpenBSD have this functionality for a reason.

-- 
Maxim Konovalov
Comment 3 Gleb Smirnoff freebsd_committer 2005-09-21 15:53:58 UTC
On Wed, Sep 21, 2005 at 06:20:24PM +0400, Maxim Konovalov wrote:
M> >  I would argue that storing bsdlabel(8), fdisk(8) or other storage
M> >  configuration information on the storage itself is the best idea.
M> 
M> The submitter said:
M> 
M> % This is exceptionally useful in the event that you need to recover
M> % your system from an offsite storage tape backup to another facility
M> % on new (identical) hardware.
M> 
M> Personally I'm always trying to create a copy of fdisk/disklabel and
M> backup them when install new machine but it is very easy to forget
M> about that procedure when you add a disk or change its layout.  I
M> helped to other people to repair their disklabels several times
M> already and from time to time see this question in FreeBSD maillists.
M> There are several tools for partitions/slices restore in ports.  Even
M> one is under src/. Believe me, this information saves a lot of man
M> hours and nerve and Net|OpenBSD have this functionality for a reason.

I know. I always save it, but outside the storage in question.

-- 
Totus tuus, Glebius.
GLEBIUS-RIPN GLEB-RIPE
Comment 4 Brian A. Seklecki 2006-06-20 14:37:15 UTC
On Tue, 2006-03-14 at 18:27 -0500, Tom Rhodes wrote:
> On Mon, 13 Mar 2006 23:47:44 -0500
> "Brian A. Seklecki" <bseklecki@mx00.pub.collaborativefusion.com> wrote:


I've updated this to work with $prefix/local/periodic./daily* and use
echo(1) less, where possible, except in conjunction with sed(1).

I've also addressed the "there's no sysctl with geom devices" with
periodic.conf(5) variables:

$daily_backup_bsdlabels_include
$daily_backup_bsdlabels_exclude (regexp passed to sed(1))

So you can do:

# enable
daily_backup_bsdlabels_enable="YES"
# don't backup unmoued CF->USB storage
daily_backup_bsdlabels_exclude="da[0-9]"
# You might die trying!
daily_backup_bsdlabels_xtra_disks="mirror/gm0"

http://digitalfreaks.org/~lavalamp/220.backup-bsdlabels

Still ugly as hell but it works!

~BAS

PS our bourne shell is strange.

part_slices=/dev/ad0s[0-9] = works
disk=ad0
part_slices=/dev/${disk}s[0-9] = doesn't work
so:
part slices=$(echo /dev/${disk}s[0-9]) = ugly, but works.

~BAS

> echo(1) has to go in this code.
> 
> What about printf(1)?
> 

-- 
Brian A. Seklecki <bseklecki@collaborativefusion.com>
Collaborative Fusion, Inc.
Comment 5 Miroslav Lachman 2007-07-09 19:23:07 UTC
Is there any progress to integrate this patch in to base system (6.x or 
7.x)?

It seems useful to me. I was thinking to integrate similar feature in to 
my own backup scripts, but start googling first and find this periodic 
script.

I think there is nothing bad on backuping disk layout on filesystem 
itself, because filesystem is (could be) easily backed up by tar, rsync, 
dump or any other backup system, so one then have disk layout (fdisk + 
bsdlabel) stored in the backup archive with data.

The last note - I can not find updated version of 220.backup-bsdlabels 
with daily_backup_bsdlabels_xtra_disks announced variable (I am using 
gmirror on almost all my servers)

Miroslav Lachman
Comment 6 Brian A. Seklecki 2007-09-27 16:31:02 UTC
NetBSD and OpenBSD have had that code in place for 10 years.  Not sure why 
we dont have it yet.  The advent of an enterprise class F/OSS backup 
system (Bacula) really emphasizes the need for this.

~BAS

On Mon, 9 Jul 2007, Miroslav Lachman wrote:

> Is there any progress to integrate this patch in to base system (6.x or 7.x)?
Comment 7 Maxim Konovalov 2007-09-28 19:11:49 UTC
On Thu, 27 Sep 2007, 15:50-0000, Brian A. Seklecki wrote:

>  NetBSD and OpenBSD have had that code in place for 10 years.  Not
>  sure why we dont have it yet.  The advent of an enterprise class
>  F/OSS backup system (Bacula) really emphasizes the need for this.
>
Could you please point me out where is the code in NetBSD?

-- 
Maxim Konovalov
Comment 8 Alexander Best 2009-03-28 01:31:01 UTC
as discussed in this thread

http://lists.freebsd.org/pipermail/freebsd-current/2009-January/002389.html

the patch needs to be updated and improved in order to get committed to
CURRENT. also netbsd/openbsd don't seem to be using this script any longer.
Comment 9 Mark Linimon freebsd_committer freebsd_triage 2009-11-10 08:15:45 UTC
State Changed
From-To: open->suspended

Script needs to be updated to work on -current.  To submitter: sorry 
that it has sat in GNATS without action for so long.
Comment 10 Alexander Best freebsd_committer 2011-02-15 01:13:44 UTC
State Changed
From-To: suspended->open

Back to open, since with 'gpart backup' there's a new and easy way to backup the 
partition table. 


Comment 11 Alexander Best freebsd_committer 2011-02-15 01:13:44 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-geom

Over to geom folks. Maybe they can come up with a periodic(8) script to take 
care of this ancient PR.
Comment 12 Miroslav Lachman 2017-11-14 17:15:54 UTC
I am giving up on this PR. 12 years are too much. 

I am providing my local scripts for users wanting to have this kind of backups.
You can put them in to /usr/local/etc/periodic/daily/ (or /etc/periodic/daily/)
Then you can set some variables in /etc/periodic.conf

daily_backup_gpart_enable="YES"
daily_backup_gpart_verbose="YES"
daily_backup_gpart_exclude=""   ## regexp pattern for grep -E

daily_backup_gmirror_enable="YES"

daily_backup_zfs_enable="YES"
daily_backup_zfs_verbose="YES"
daily_backup_zfs_props_enable="YES"
Comment 13 Miroslav Lachman 2017-11-14 17:17:48 UTC
Created attachment 187996 [details]
Backup of disk partitions layout

daily_backup_gpart_enable="YES"
daily_backup_gpart_verbose="YES"
daily_backup_gpart_exclude=""   ## regexp pattern for grep -E
Comment 14 Miroslav Lachman 2017-11-14 17:18:37 UTC
Created attachment 187997 [details]
Backup of gmirror list detailed info

daily_backup_gmirror_enable="YES"
Comment 15 Miroslav Lachman 2017-11-14 17:19:28 UTC
Created attachment 187998 [details]
Backup of zpool info, zfs list and zfs properties

daily_backup_zfs_enable="YES"
daily_backup_zfs_verbose="YES"
daily_backup_zfs_props_enable="YES"
Comment 16 Allan Jude freebsd_committer 2017-11-14 20:20:10 UTC
I had never seen this PR before. I'll take it and work on getting it in to the tree.
Comment 17 Eitan Adler freebsd_committer freebsd_triage 2018-05-28 19:46:48 UTC
batch change:

For bugs that match the following
-  Status Is In progress 
AND
- Untouched since 2018-01-01.
AND
- Affects Base System OR Documentation

DO:

Reset to open status.


Note:
I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
Comment 18 Miroslav Lachman 2018-05-28 20:45:47 UTC
Exactly what I was expecting - six month later and nothing happened.
So please pretty please close this PR to not scare potential new submitters that their work will lay untouched for 13 years...
This is really scary example of how things doesn't work in FreeBSD.
Comment 19 Allan Jude freebsd_committer 2020-07-11 19:09:59 UTC
During the July 2020 Bugathon, a volunteer refreshed the first of these patches, extending it to include support for backing up the EFI System Partition.

https://reviews.freebsd.org/D25628

There is some additional feedback in the review, since the man pages and etc/defaults/periodic.conf also need to be updated.


Miroslav: Sorry for the lack of progress, there are 10,000 open PRs, it is far too easy for one to go unnoticed.
Comment 20 commit-hook freebsd_committer 2020-07-11 20:53:52 UTC
A commit references this bug:

Author: allanjude
Date: Sat Jul 11 20:53:32 UTC 2020
New revision: 363110
URL: https://svnweb.freebsd.org/changeset/base/363110

Log:
  Add a periodic script to backup the partition table and boot code

  Optionally, alert you if the contents change from the previous backup

  PR:		86388
  Submitted by:	Rob Fairbanks <rob.fx907@gmail.com>, Miroslav Lachman <000.fbsd@quip.cz> (Original Version)
  MFC after:	4 weeks
  Relnotes:	yes
  Sponsored by:	Klara Inc.
  Event:		July 2020 Bugathon
  Differential Revision:	https://reviews.freebsd.org/D25628

Changes:
  head/share/man/man5/periodic.conf.5
  head/usr.sbin/periodic/etc/daily/221.backup-gpart
  head/usr.sbin/periodic/periodic.conf
Comment 21 Miroslav Lachman 2020-07-12 06:57:54 UTC
(In reply to Allan Jude from comment #19)
Are there any good reason to not include scripts to backup gmirror, ZFS or other layout info too?
Comment 22 Robert Wing freebsd_committer 2020-07-12 19:23:55 UTC
(In reply to Miroslav Lachman from comment #21)

Hey Miroslav,

I have submitted patches that include man page and periodic.conf(5) changes for the gmirror and the zfs scripts.

They can be reviewed at:

https://reviews.freebsd.org/D25631 - gmirror periodic script patch
 
https://reviews.freebsd.org/D25638 - zfs periodic script patch

Any comments are welcome.

Thanks,
Rob
Comment 23 Allan Jude freebsd_committer 2020-07-13 02:10:52 UTC
(In reply to Miroslav Lachman from comment #21)
I expect to have time to review and land the additional 2 patches in the next few days
Comment 24 Miroslav Lachman 2020-07-13 07:20:14 UTC
(In reply to Rob from comment #22)
Thank you! I am glad it will finally find a way in to the base.
Comment 25 Allan Jude freebsd_committer 2020-07-13 13:55:24 UTC
There is a report that the script is not being installed.

It seems we need to add the scripts to the Makefile in the corresponding directory, like: usr.sbin/periodic/etc/daily
Comment 26 Robert Wing freebsd_committer 2020-07-13 19:47:13 UTC
Allan,

Here's the fix for installing 221.backup-gpart properly.

https://reviews.freebsd.org/D25653

I'll update the other two revisions as well.
Comment 27 Allan Jude freebsd_committer 2020-07-14 00:46:43 UTC
I forgot to tag r363169 with the PR
Comment 28 commit-hook freebsd_committer 2020-11-06 22:58:39 UTC
A commit references this bug:

Author: rew
Date: Fri Nov  6 22:58:32 UTC 2020
New revision: 367436
URL: https://svnweb.freebsd.org/changeset/base/367436

Log:
  Add a periodic script to backup output generated from `zfs list`, `zfs get`,
  `zpool list`, and `zpool get` commands.

  Disabled by default.

  PR:             86388
  Submitted by:   Miroslav Lachman <000.fbsd@quip.cz>
  Reviewed by:    allanjude, 0mp
  Approved by:    allanjude (mentor)
  MFC after:      4 weeks
  Event:          July 2020 Bugathon
  Differential Revision:  https://reviews.freebsd.org/D25638

Changes:
  head/share/man/man5/periodic.conf.5
  head/usr.sbin/periodic/etc/daily/223.backup-zfs
  head/usr.sbin/periodic/etc/daily/Makefile
  head/usr.sbin/periodic/periodic.conf
Comment 29 commit-hook freebsd_committer 2020-11-07 04:16:21 UTC
A commit references this bug:

Author: rew
Date: Sat Nov  7 04:15:25 UTC 2020
New revision: 367443
URL: https://svnweb.freebsd.org/changeset/base/367443

Log:
  Add a periodic script to backup output generated from `gmirror list`.

  Disabled by default.

  PR:             86388
  Submitted by:   Miroslav Lachman <000.fbsd@quip.cz>
  Reviewed by:    allanjude, gbe
  Approved by:    allanjude (mentor)
  MFC after:      4 weeks
  Event:          July 2020 Bugathon
  Differential Revision:  https://reviews.freebsd.org/D25631

Changes:
  head/share/man/man5/periodic.conf.5
  head/usr.sbin/periodic/etc/daily/222.backup-gmirror
  head/usr.sbin/periodic/etc/daily/Makefile
  head/usr.sbin/periodic/periodic.conf