FreeBSD 11.2-RELEASE, *based on mfsbsd, *not* tested on *vanilla* FreeBSD 11.2-RELEASE*. Files are lost when the following steps are made: Start a jail foo, edit its /etc/fstab.foo, adding a nullfs-rw mount that is not valid (e.g. non-existent mountpoint), jail -r jail foo When you remove the jail, you get an error because of the wrong entry. The main mountpoint of the jail is now still mounted. When you manually umount this remaining main mountpoint, all nullfs-rw mount files are lost. Expected result: files would not be lost Actual result: files are lost Example: # cat /etc/fstab.maygo /usr/jail/basejail /usr/jail/foo nullfs ro /zroot/j/foo.d/foo.homedir /usr/jail/foo/usr/home nullfs rw /zroot/j/foo.d/foo.var /usr/jail/foo/var nullfs rw /zroot/j/foo.d/foo.tmp /usr/jail/foo/tmp nullfs rw /zroot/j/foo.d/foo.etc /usr/jail/foo/etc nullfs rw /zroot/j/foo.d/foo.root /usr/jail/foo/root nullfs rw # --------------------------- # cat /etc/jail.conf # Common configs for all jails mount.fstab = "/etc/fstab.${name}"; exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.consolelog = "/var/log/jail_${name}_console.log"; host.hostname = "${name}"; allow.mount; devfs_ruleset = "4"; mount.devfs; allow.set_hostname = 0; path = "/usr/jail/${name}"; foo { ip4.addr = 10.2.2.4 ; interface = vtnet0; allow.sysvipc = 1; allow.raw_sockets = 1; persist; } # ----------------------------- # jls JID IP Address Hostname Path # jail -c foo foo: created # find /zroot/j/foo.d | wc 100 100 3971 # jls JID IP Address Hostname Path 46 10.2.2.4 foo /usr/jail/foo # echo "/zroot/j/foo.d/foo.root /usr/jail/foo/NONEXISTENTDIR nullfs rw" >> /etc/fstab.foo # tail -3 /etc/fstab.foo /zroot/j/foo.d/foo.root /usr/jail/foo/root nullfs rw /zroot/j/foo.d/foo.root /usr/jail/foo/NONEXISTENTDIR nullfs rw # jail -r foo foo: removed jail: foo: mount.fstab: /usr/jail/foo/NONEXISTENTDIR: No such file or directory # mount | grep foo zroot/j/foo.d on /zroot/j/foo.d (zfs, local, noatime, nfsv4acls) # umount /zroot/j/foo.d # echo $? 0 # find /zroot/j/foo.d | wc 1 1 17 # zfs list | grep foo zroot/j/foo.d 34.0M 8.50G 34.0M /zroot/j/foo.d # zfs get mountpoint zroot/j/foo.d NAME PROPERTY VALUE SOURCE zroot/j/foo.d mountpoint /zroot/j/foo.d inherited from zroot # mount zroot/j/foo.d /zroot/j/foo.d mount: zroot/j/foo.d: No such file or directory
Follow up: reproducing the original bug, I am now able to do a recovery. In both original bug report and this follow-up, zfs was used. Using the same /etc/jail.conf, and a similar /etc/fstab.foo, getting the zfs fileset lost again, here's how it can be recovered: # mkdir /recovery # an arbitrary recovery dir # zfs set mountpoint=/recovery zroot/j/foo.d # cd /recovery # ls # # nothing recovered # zfs set mountpoint="legacy" zroot/j/foo.d # possibly a needed step # zfs list | grep foo zroot/j/foo.d 71.7M 428M 71.7M legacy # zfs still lists usage # zfs set mountpoint="/zroot/j/foo.d" zroot/j/foo.d #point zfs to org mountpoint # cd /zroot/j/foo.d # ls -1 foo.etc foo.root foo.homedir foo.tmp # files recovered