After installing a fresh FreeBSD 10.0 Release with ZFS root on my server I created a snapshot with the -r option. Then I created a file in /usr/local and wanted to rollback to the previous state but noticed that nothing happened.
After a bit of investigation I found out that zfs list claims that /usr is a separate volume which gets its own snapshot entries but in reality it is not mounted and instead located in zroot/ROOT/default
So in order to rollback /usr/local I'd have to rollback zroot/Root/default.
zfs get all zroot/usr reports that /usr is not mounted and the canmount option is set to off.
This creates a lot of confusion and is counter intuitive. If you carefully compare the output of zfs list and mount you can see the problem maybe a bit better:
zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var on /var (zfs, local, noatime, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
Mount shows no /usr volume but when you look at zfs list you get:
$> zfs list
NAME USED AVAIL REFER MOUNTPOINT
zroot 2.17G 103G 144K none
zroot/ROOT 857M 103G 144K none
zroot/ROOT/default 857M 103G 857M /
zroot/tmp 192K 103G 192K /tmp
zroot/usr 1.33G 103G 144K /usr
zroot/usr/home 184K 103G 184K /usr/home
zroot/usr/ports 813M 103G 813M /usr/ports
zroot/usr/src 545M 103G 545M /usr/src
zroot/var 1.27M 103G 644K /var
zroot/var/crash 148K 103G 148K /var/crash
zroot/var/log 212K 103G 212K /var/log
zroot/var/mail 144K 103G 144K /var/mail
zroot/var/tmp 152K 103G 152K /var/tmp
when listing the snapshots again you will get an entry for /usr
zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
zroot@eins 0 - 144K -
zroot/ROOT@eins 0 - 144K -
zroot/ROOT/default@eins 8K - 857M -
zroot/tmp@eins 0 - 192K -
zroot/usr@eins 0 - 144K -
zroot/usr/home@eins 0 - 184K -
zroot/usr/ports@eins 0 - 813M -
zroot/usr/src@eins 0 - 545M -
zroot/var@eins 96K - 644K -
zroot/var/crash@eins 0 - 148K -
zroot/var/log@eins 84K - 212K -
zroot/var/mail@eins 0 - 144K -
zroot/var/tmp@eins 0 - 152K -
but in reality /usr and all its sub directories which are not a dedicated volume are under zroot/ROOT/default.
Can you please clean up this mess if there are no real reasons to keep it that way?
How-To-Repeat: Install a fresh FreeBSD 10 with ZFS root filesystem with bsdinstall. Make a snapshot, create a file in /usr and try to rollback to the snapshot with zfs rollback zroot/usr@snapshot - /usr will remain the same. Instead do zfs rollback zroot/ROOT/default and the file in /usr will disappear.
I observe this with a 10.2-RELEASE install too. All the directories under /usr such as /usr/bin are actually part of the zroot/ROOT/default file system. It is very counter-intuitive.
Looking at the bsdinstall(8) zfsboot script,
this is by design,
# Don't mount /usr so that 'base' files go to the BEROOT
I agree with that it might it being counter intuitive, but there are reasons why 'base' files should not go outside the Boot Environment ROOT. Have a look at sysutils/beadm and perhaps this blog post,
For bugs matching the following conditions:
- Status == In Progress
- Assignee == "bugs@FreeBSD.org"
- Last Modified Year <= 2017
- Set Status to "Open"
After a couple of years, I have also come to the conclusion that it "works as intended" but the documentation around it is horrible. Clearly, it is a good layout to work with the beadm utility, and that is a good thing. The rationale of how it is set up should really be better documented in the handbook and possibly man page for hier. Right now they only way you can learn about it is through some old forum or mailing list posts, which is not really documentation.
Maybe discuss this on mailing lists or forums?