Bug 186345 - FreeBSD 10 Installer creates weird ZFS volume layout for /usr
Summary: FreeBSD 10 Installer creates weird ZFS volume layout for /usr
Status: Closed Works As Intended
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: Unspecified
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-bugs (Nobody)
Depends on:
Reported: 2014-02-01 15:40 UTC by John-Paul Bader
Modified: 2018-05-23 14:22 UTC (History)
2 users (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description John-Paul Bader 2014-02-01 15:40:00 UTC
After installing a fresh FreeBSD 10.0 Release with ZFS root on my server I created a snapshot with the -r option. Then I created a file in /usr/local and wanted to rollback to the previous state but noticed that nothing happened. 

After a bit of investigation I found out that zfs list claims that /usr is a separate volume which gets its own snapshot entries but in reality it is not mounted and instead located in zroot/ROOT/default 

So in order to rollback /usr/local I'd have to rollback zroot/Root/default. 

 zfs get all zroot/usr reports that /usr is not mounted and the canmount option is set to off.

This creates a lot of confusion and is counter intuitive. If you carefully compare the output of zfs list and mount you can see the problem maybe a bit better:

$> mount
zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var on /var (zfs, local, noatime, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)

Mount shows no /usr volume but when you look at zfs list you get:

$> zfs list
zroot               2.17G   103G   144K  none
zroot/ROOT           857M   103G   144K  none
zroot/ROOT/default   857M   103G   857M  /
zroot/tmp            192K   103G   192K  /tmp
zroot/usr           1.33G   103G   144K  /usr
zroot/usr/home       184K   103G   184K  /usr/home
zroot/usr/ports      813M   103G   813M  /usr/ports
zroot/usr/src        545M   103G   545M  /usr/src
zroot/var           1.27M   103G   644K  /var
zroot/var/crash      148K   103G   148K  /var/crash
zroot/var/log        212K   103G   212K  /var/log
zroot/var/mail       144K   103G   144K  /var/mail
zroot/var/tmp        152K   103G   152K  /var/tmp

when listing the snapshots again you will get an entry for /usr

zfs list -t snapshot
NAME                      USED  AVAIL  REFER  MOUNTPOINT
zroot@eins                   0      -   144K  -
zroot/ROOT@eins              0      -   144K  -
zroot/ROOT/default@eins     8K      -   857M  -
zroot/tmp@eins               0      -   192K  -
zroot/usr@eins               0      -   144K  -
zroot/usr/home@eins          0      -   184K  -
zroot/usr/ports@eins         0      -   813M  -
zroot/usr/src@eins           0      -   545M  -
zroot/var@eins             96K      -   644K  -
zroot/var/crash@eins         0      -   148K  -
zroot/var/log@eins         84K      -   212K  -
zroot/var/mail@eins          0      -   144K  -
zroot/var/tmp@eins           0      -   152K  -

but in reality /usr and all its sub directories which are not a dedicated volume are under zroot/ROOT/default.

Can you please clean up this mess if there are no real reasons to keep it that way?

How-To-Repeat: Install a fresh FreeBSD 10 with ZFS root filesystem with bsdinstall. Make a snapshot, create a file in /usr and try to rollback to the snapshot with zfs rollback zroot/usr@snapshot - /usr will remain the same. Instead do zfs rollback zroot/ROOT/default and the file in /usr will disappear.
Comment 1 Vick Khera 2015-09-02 18:29:14 UTC
I observe this with a 10.2-RELEASE install too. All the directories under /usr such as /usr/bin are actually part of the zroot/ROOT/default file system. It is very counter-intuitive.
Comment 2 fre.fbsdpr 2015-10-15 16:32:37 UTC
Looking at the bsdinstall(8) zfsboot script,


this is by design,

  # Don't mount /usr so that 'base' files go to the BEROOT
  /usr            mountpoint=/usr,canmount=off

I agree with that it might it being counter intuitive, but there are reasons why 'base' files should not go outside the Boot Environment ROOT.  Have a look at sysutils/beadm and perhaps this blog post,

Comment 3 Eitan Adler freebsd_committer freebsd_triage 2018-05-20 23:54:03 UTC
For bugs matching the following conditions:
- Status == In Progress
- Assignee == "bugs@FreeBSD.org"
- Last Modified Year <= 2017

- Set Status to "Open"
Comment 4 Vick Khera 2018-05-23 13:11:03 UTC
After a couple of years, I have also come to the conclusion that it "works as intended" but the documentation around it is horrible. Clearly, it is a good layout to work with the beadm utility, and that is a good thing. The rationale of how it is set up should really be better documented in the handbook and possibly man page for hier. Right now they only way you can learn about it is through some old forum or mailing list posts, which is not really documentation.
Comment 5 Andriy Gapon freebsd_committer 2018-05-23 14:22:02 UTC
Maybe discuss this on mailing lists or forums?