Bug 54884

There appears to be a caching of free space on UFS1 filesystem in the
filesystem superblock.  These caching methods appear to be different in
-current and -stable.

An update free space value for a certain UFS1 filesystem mounted on a
-current kernel will be seen when the filesystem is mounted in -stable.
However, if there is a change of the free space while the filesystem is
mounted in -stable, the change will not reflect in -current.

Please note this is NOT related to the reserved percent of UFS
filesystems (-m flag to tunefs).

Fix: 

Workaround by forcing fsck when booting from one version to the other,
or do not use one UFS1 filesystem under both versions of FreeBSD.
How-To-Repeat: 
Make sure an UFS1 filesystem is clean.  Force fsck both on -current and
-stable (see below).

Mount the filesystem in -current, then create a large file, preferably
amounting to a large percentage of the filesystem size.  Write down the
values returned by df for that filesystem, then reboot to -stable and
remove the file.  Reboot again to -current, and notice the values from
df have not changed.  However, the file can be created again, and
deleted again in -stable.

Another way to abuse this is to create the files on -stable and remove
them in -current.  This will make the used blocks count on the
filesystem to decrease with each repetition.

It is possible to fix the free space issue by forcing fsck.  Otherwise
the filesystem will appear clean.  However, it seems that the -stable
and -current versions of fsck do not check for the same things, and only
fix the issue for their respective OS versions.  Mounting the filesystem
read/write on -current, after the -current fsck is run will cause the
kernel to update both versions of the free space value.

Some command outputs
STABLE:
*on fsck -p
/dev/ad0s2g: clean, -212550 free (79930 frags, -36560 blocks, 3.5% fragmentation)

*on forcing fsck:
** /dev/ad0s2g
** Last Mounted on /usr
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
SUMMARY INFORMATION BAD
SALVAGE? [yn] y
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? [yn] y
208421 files, 1831342 used, 422700 free
(80596 frags, 42763 blocks, 3.6% fragmentation)

CURRENT:
*on fsck -p
/dev/ad0s2g: FILESYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s2g: clean, -212550 free (79930 frags, -36560 blocks, 3.5% fragmentation)
*on forcing fsck
** /dev/ad0s2g
** Last Mounted on /usr
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
SUMMARY BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? [yn] y
208421 files, 1831344 used, 422698 free (80586 frags, 42764
blocks, 3.6% fragmentation)
***** FILE SYSTEM WAS MODIFIED *****


Fun things to do:
(1) Get the filesystem usage figure to over 109% on an UFS1 filesystem,
which has the default reserved percentage for the superuser at 8%, or
get df to report more used blocks than the filesystem size.
Example:
Filesystem  1K-blocks     Used   Avail Capacity  Mounted on
/dev/ad0s2g   4508084  4933184 -785746   119%    /usr2

(2) Get the filesystem usage figure to be negative.
Example:
Filesystem  1K-blocks     Used   Avail Capacity  Mounted on
/dev/ad0s2g   4508084  -282454 4429892    -7%    /usr