Bug 264450 - ufs: Partition recognized on 12.3 not recognized on CURRENT (2573e6ced996): Cannot find file system superblock .. Invalid fstype: Invalid argument
Summary: ufs: Partition recognized on 12.3 not recognized on CURRENT (2573e6ced996): C...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-fs (Nobody)
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2022-06-04 16:14 UTC by Martin Birgmeier
Modified: 2024-03-11 22:12 UTC (History)
6 users (show)

See Also:
koobs: maintainer-feedback? (mckusick)


Attachments
First megabyte of da0s4d (128.83 KB, application/x-compressed-tar)
2022-06-04 16:14 UTC, Martin Birgmeier
no flags Details
proposed fix (4.63 KB, patch)
2022-06-05 05:58 UTC, Kirk McKusick
no flags Details | Diff
First megabyte of da0s1a within the MBR+BSD image (53.10 KB, application/x-xz)
2024-02-14 22:15 UTC, slb
no flags Details
less restrictive UFS1 superblock check (1.21 KB, patch)
2024-02-17 22:19 UTC, Kirk McKusick
no flags Details | Diff
additional proposed fsck_ffs fix (974 bytes, patch)
2024-02-19 20:38 UTC, Kirk McKusick
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Birgmeier 2022-06-04 16:14:16 UTC
Created attachment 234446 [details]
First megabyte of da0s4d

Scenario:
- A disk which, if attached to a 12.3 system, shows up as:

[0]# ll /dev/da0*
crw-r-----  1 root  operator  0x75 Jun  4 17:39 /dev/da0
crw-r-----  1 root  operator  0x79 Jun  4 17:39 /dev/da0s1
crw-r-----  1 root  operator  0x7b Jun  4 17:39 /dev/da0s4
crw-r-----  1 root  operator  0x83 Jun  4 17:39 /dev/da0s4a
crw-r-----  1 root  operator  0x84 Jun  4 17:39 /dev/da0s4b
crw-r-----  1 root  operator  0x85 Jun  4 17:39 /dev/da0s4d
[0]# gpart show da0
=>       63  976773105  da0  MBR  (466G)
         63       1985       - free -  (993K)
       2048  761726968    1  ntfs  (363G)
  761729016  215044152    4  freebsd  [active]  (103G)

[0]# gpart show da0s4
=>        0  215044152  da0s4  BSD  (103G)
          0         16         - free -  (8.0K)
         16    4194304      1  freebsd-ufs  (2.0G)
    4194320   10485760      2  freebsd-swap  (5.0G)
   14680080  200364072      4  freebsd-ufs  (96G)

[0]# 

- The two filesystems can be fsck'd o.k.:

0]# fsck /dev/da0s4a
** /dev/da0s4a
** Last Mounted on /d/26s4a
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
3627 files, 131623 used, 376160 free (3256 frags, 46613 blocks, 0.6% fragmentation)

***** FILE SYSTEM IS CLEAN *****
[0]# fsck /dev/da0s4d
** /dev/da0s4d
** Last Mounted on /d/26s4a/usr
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1269131 files, 10661916 used, 13593544 free (15728 frags, 1697227 blocks, 0.1% fragmentation)

***** FILE SYSTEM IS CLEAN *****
[0]# 

- And they can be mounted:

[0]# mount /dev/da0s4a /mnt
[0]# mount /dev/da0s4d /mnt/usr
[0]# umount /mnt/usr
[0]# umount /mnt    
[0]# 

- Now connecting this same disk to a system running main 2573e6ced99616f2edccf97fb0f9283a84148246

Result:
- Now the da0s4d partition is not recognized as a UFS filesystem.
- The partitions do show up:

[0]# ll /dev/da*
crw-r-----  1 root  operator  0x7b Jun  4 17:08 /dev/da0
crw-r-----  1 root  operator  0x5d Jun  4 18:11 /dev/da0s1
crw-r-----  1 root  operator  0x5f Jun  4 18:11 /dev/da0s4
crw-r-----  1 root  operator  0x7d Jun  4 18:11 /dev/da0s4a
crw-r-----  1 root  operator  0x7e Jun  4 18:11 /dev/da0s4b
crw-r-----  1 root  operator  0x7f Jun  4 18:11 /dev/da0s4d
[0]# gpart show da0
=>       63  976773105  da0  MBR  (466G)
         63       1985       - free -  (993K)
       2048  761726968    1  ntfs  (363G)
  761729016  215044152    4  freebsd  [active]  (103G)

[0]# gpart show da0s4
=>        0  215044152  da0s4  BSD  (103G)
          0         16         - free -  (8.0K)
         16    4194304      1  freebsd-ufs  (2.0G)
    4194320   10485760      2  freebsd-swap  (5.0G)
   14680080  200364072      4  freebsd-ufs  (96G)

[0]# 

- But fsck'ing yields:

[0]# fsck /dev/da0s4a
** /dev/da0s4a

ADD SUPERBLOCK CHECK-HASH PROTECTION? [yn] n


ADD INODE CHECK-HASH PROTECTION? [yn] n

** Last Mounted on /mnt
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
3627 files, 131623 used, 376160 free (3256 frags, 46613 blocks, 0.6% fragmentation)

***** FILE SYSTEM IS CLEAN *****
[0]# fsck /dev/da0s4d
Cannot find file system superblock
Cannot find file system superblock

LOOK FOR ALTERNATE SUPERBLOCKS? no

[8]# 

- And trying to mount yields:

[8]# mount /dev/da0s4a /mnt
[0]# mount /dev/da0s4d /mnt/usr
mount: /dev/da0s4d: Invalid fstype: Invalid argument
[1]# umount /mnt               
[0]# 

Expected result:
- da0s4d should be usable under main as well.

Note:
- The first megabyte of da0s4d is attached. It hopefully suffices to find out why the superblock is not accepted on main.

-- Martin
Comment 1 Graham Perrin freebsd_committer freebsd_triage 2022-06-04 16:56:13 UTC
Notable, although I can not tell whether they are directly relevant: 

<https://github.com/freebsd/freebsd-src/commit/bc218d89200faa021def77732f3d9fde4f4dee13> (2022-01-06) and 076002f (2022-05-27)
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2022-06-05 01:36:06 UTC
@Reporter Are you able to test fsck/mount with a 13 release or stable snapshot image?
Comment 3 Martin Birgmeier 2022-06-05 05:45:30 UTC
(In reply to Graham Perrin from comment #1)

The issue appeared when the system was upgraded to d64f2f42c111009dade9ab7ae4d75a0256630e14 (2022-05-28). It was then upgraded to 2573e6ced996, and the issue persisted.

The disk was recognized when the system was running 398c88c7582a195cbfeb689ceff1400cc717673f (2022-01-01).

-- Martin
Comment 4 Martin Birgmeier 2022-06-05 05:56:02 UTC
(In reply to Kubilay Kocak from comment #2)

13.1 fc952ac2212b121aa6eefc273f5960ec3e0a466d is working:

[0]# ll /dev/da0*
crw-r-----  1 root  operator  0x59 Jun  5 07:48 /dev/da0
crw-r-----  1 root  operator  0x5d Jun  5 07:48 /dev/da0s1
crw-r-----  1 root  operator  0x5e Jun  5 07:48 /dev/da0s4
crw-r-----  1 root  operator  0x62 Jun  5 07:48 /dev/da0s4a
crw-r-----  1 root  operator  0x63 Jun  5 07:48 /dev/da0s4b
crw-r-----  1 root  operator  0x64 Jun  5 07:48 /dev/da0s4d
[0]# fsck /dev/da0s4a
** /dev/da0s4a

ADD SUPERBLOCK CHECK-HASH PROTECTION? [yn] n


ADD INODE CHECK-HASH PROTECTION? [yn] n

** Last Mounted on /mnt
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
3627 files, 131623 used, 376160 free (3256 frags, 46613 blocks, 0.6% fragmentation)

***** FILE SYSTEM IS CLEAN *****
[0]# fsck /dev/da0s4d
** /dev/da0s4d

ADD SUPERBLOCK CHECK-HASH PROTECTION? [yn] n


ADD INODE CHECK-HASH PROTECTION? [yn] n

** Last Mounted on /mnt/usr
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1269131 files, 10661916 used, 13593544 free (15728 frags, 1697227 blocks, 0.1% fragmentation)

***** FILE SYSTEM IS CLEAN *****
[0]# gpart show da0
=>       63  976773105  da0  MBR  (466G)
         63       1985       - free -  (993K)
       2048  761726968    1  ntfs  (363G)
  761729016  215044152    4  freebsd  [active]  (103G)

[0]# gpart show da0s4
=>        0  215044152  da0s4  BSD  (103G)
          0         16         - free -  (8.0K)
         16    4194304      1  freebsd-ufs  (2.0G)
    4194320   10485760      2  freebsd-swap  (5.0G)
   14680080  200364072      4  freebsd-ufs  (96G)

[0]# 

-- Martin
Comment 5 Kirk McKusick freebsd_committer freebsd_triage 2022-06-05 05:58:55 UTC
Created attachment 234455 [details]
proposed fix

Please apply this diff to your kernel and see if it resolves your problem. If it does not resolve your problem please let me know the "Failed code" that it prints. Note that you need to recompile libufs if you want it to apply to filesystem utilities like fsck, dumpfs, etc. And you need to recompile libsa if you want it to apply to your loader.
Comment 6 Martin Birgmeier 2022-06-05 06:00:51 UTC
I had a small hope that after fsck'ing on 13.1, the partition would be recognized on main - alas, this is not the case.

-- Martin
Comment 7 Martin Birgmeier 2022-06-05 07:11:44 UTC
(In reply to Kirk McKusick from comment #5)

I recompile using buildworld buildkernel -DNO_CLEAN. The following files differed from the installation and were reinstalled (I first install into a temporary location and then do a compare):

boot/kernel/kernel
boot/kernel/ufs.ko
boot/loader
boot/loader.efi
boot/loader.kboot
boot/loader_4th
boot/loader_4th.efi
boot/loader_lua
boot/loader_lua.efi
boot/loader_simp
boot/loader_simp.efi
boot/pxeboot
boot/userboot.so
boot/userboot_4th.so
boot/userboot_lua.so
boot/zfsloader
lib/libufs.so.7
(everything in rescue)
sbin/fsdb
usr/lib/debug/boot/kernel/kernel.debug
usr/lib/debug/boot/kernel/ufs.ko.debug
usr/lib/libufs.a
usr/lib/libufs.so
usr/lib32/libufs.a
usr/lib32/libufs.so
usr/lib32/libufs.so.7

It is interesting to note that libsa was not recompiled.

When attaching the disk, the console shows the following:

da1 at iscsi2 bus 0 scbus4 target 0 lun 0
da1: <FREEBSD CTLDISK 0001>  s/n RPI-B_DISK26 detached
(da1:iscsi2:0:0:0): Periph destroyed
da1 at iscsi3 bus 0 scbus4 target 0 lun 0
da1: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
da1: Serial Number HAL_P0026
da1: 150.000MB/s transfers
da1: Command Queueing enabled
da1: 476940MB (976773168 512 byte sectors)
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3
Failed code 3

Fsck'ing is interesting - it now proceeds on da1s4d (it is now da1 because of a different way of booting):

[0]# ll /dev/da1*    
crw-r-----  1 root  operator  0x6c Jun  5 08:59 /dev/da1
crw-r-----  1 root  operator  0x6e Jun  5 08:59 /dev/da1s1
crw-r-----  1 root  operator  0x6f Jun  5 08:59 /dev/da1s4
crw-r-----  1 root  operator  0x73 Jun  5 08:59 /dev/da1s4a
crw-r-----  1 root  operator  0x74 Jun  5 08:59 /dev/da1s4b
crw-r-----  1 root  operator  0x75 Jun  5 08:59 /dev/da1s4d
[0]# fsck /dev/da1s4a
** /dev/da1s4a

ADD SUPERBLOCK CHECK-HASH PROTECTION? [yn] n


ADD INODE CHECK-HASH PROTECTION? [yn] n

** Last Mounted on /mnt
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
3627 files, 131623 used, 376160 free (3256 frags, 46613 blocks, 0.6% fragmentation)

***** FILE SYSTEM IS CLEAN *****
[0]# fsck /dev/da1s4d
** /dev/da1s4d

ADD SUPERBLOCK CHECK-HASH PROTECTION? [yn] n


ADD INODE CHECK-HASH PROTECTION? [yn] n

** Last Mounted on /mnt/usr
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1269131 files, 10661916 used, 13593544 free (15728 frags, 1697227 blocks, 0.1% fragmentation)

***** FILE SYSTEM IS CLEAN *****
fsck /dev/da1s4d  1.65s user 10.11s system 2% cpu 7:56.92 total
[0]# gpart show da1
=>       63  976773105  da1  MBR  (466G)
         63       1985       - free -  (993K)
       2048  761726968    1  ntfs  (363G)
  761729016  215044152    4  freebsd  [active]  (103G)

[0]# gpart show da1s4
=>        0  215044152  da1s4  BSD  (103G)
          0         16         - free -  (8.0K)
         16    4194304      1  freebsd-ufs  (2.0G)
    4194320   10485760      2  freebsd-swap  (5.0G)
   14680080  200364072      4  freebsd-ufs  (96G)

[0]# 

-- Martin
Comment 8 Martin Birgmeier 2022-06-05 07:14:37 UTC
... and it can be mounted:

[0]# mount /dev/da1s4a /mnt
[0]# mount /dev/da1s4d /mnt/usr
[0]# ll /mnt/usr
total 104
drwxrwxr-x   2 root  operator    512 Nov 15  2009 .snap
drwxr-xr-x   3 root  wheel       512 Nov 12  2021 VOL
drwxr-xr-x   2 root  wheel      8192 Aug 25  2018 bin
lrwxr-xr-x   1 root  wheel        10 Aug  7  2010 cvs -> ../vol/cvs
lrwxr-xr-x   1 root  wheel        16 Oct 10  2010 cvs.local -> ../vol/cvs.local
drwxr-xr-x   2 root  wheel       512 May  1  2016 games
drwxr-xr-x  55 root  wheel      6656 Jul  2  2018 include
drwxr-xr-x  10 root  wheel     16384 Aug 25  2018 lib
drwxr-xr-x   5 root  wheel     16384 Aug 25  2018 lib32
drwxr-xr-x   6 root  wheel       512 Dec  6  2014 libdata
drwxr-xr-x   9 root  wheel      1536 Aug 25  2018 libexec
drwxr-xr-x  15 root  wheel       512 Dec  1  2018 local
-rw-r--r--   1 root  wheel         0 Apr 20  2011 local.NODUMP
drwxr-xr-x   2 root  wheel       512 Dec 13  2009 lost+found
lrwxr-xr-x   1 root  wheel        10 Nov  1  2018 obj -> ../vol/obj
lrwxr-xr-x   1 root  wheel        12 Aug  6  2010 ports -> ../vol/ports
drwxr-xr-x   2 root  wheel      5632 Aug 25  2018 sbin
drwxr-xr-x  32 root  wheel      1024 Aug 14  2017 share
lrwxr-xr-x   1 root  wheel        10 Nov  1  2018 src -> ../vol/src
lrwxr-xr-x   1 root  wheel        16 Aug  6  2010 src.local -> ../vol/src.local
drwxr-xr-x   3 root  wheel       512 Dec 29  2010 srcs
drwxr-xr-x   4 root  wheel       512 Mar 23  2013 ss
drwxrwxrwt   3 root  wheel       512 Apr 15 20:14 tmp
-rw-r--r--   1 root  wheel         0 Aug 28  2011 tmp.NODUMP
drwxr-xr-x   4 root  wheel       512 Jan 29  2012 users
[0]# umount /mnt/usr
[0]# umount /mnt    
[0]# 

-- Martin
Comment 9 Martin Birgmeier 2022-06-05 07:15:35 UTC
But why then "Failed code 3"?

-- Martin
Comment 10 Kirk McKusick freebsd_committer freebsd_triage 2022-06-05 15:09:05 UTC
(In reply to Martin Birgmeier from comment #9)
Code 3 is when an attempt is made to read a block that has a bad magic number. When disks are added the tasting code looks in up to four places to try and find a superblock and most of them fail with a bad magic number. So code 3 failures are expected. I probably should not have added that failure code. But the good news is that you are not getting any other failure codes and you are able to mount your filesystem. So I did manage to track down the cause of your failure. Thanks for the 1M image of your filesystem as that is what enabled me to track down the cause of your failure. I will do some cleanup of the code and get it checked in.
Comment 11 Martin Birgmeier 2022-06-05 16:13:42 UTC
(In reply to Kirk McKusick from comment #10)

Thank you very much for the quick solution!

You should know that I am telling my wife and kids who you are and about your contributions to Unix, and that I admire your modest ways which lead you to still contribute to FreeBSD in your area of expertise and - even - fix someone's problems who is sitting 10,000 km away in Vienna, Austria.

-- Martin
Comment 12 Graham Perrin freebsd_committer freebsd_triage 2022-06-05 16:16:20 UTC
(In reply to Martin Birgmeier from comment #11)

… a round of applause, echoes …
Comment 13 Kirk McKusick freebsd_committer freebsd_triage 2022-06-07 00:31:16 UTC
(In reply to Martin Birgmeier from comment #11)
(In reply to Graham Perrin from comment #12)
Thanks to both of you for your kind words. Since I was the one that broke the code, I do feel like it is my responsibility to fix it :-)

Martin, I do hope to meet you at EuroBSD in September since it is being held in Vienna.
Comment 14 Martin Birgmeier 2022-06-10 18:49:47 UTC
Ah, I did not realize this! But I don't think I'll attend EuroBSDCon, that's way over my head. :-)

But I would be honored if you and your accompaniment would accept an invitation to dinner at some nice place!

Best regards, Martin
Comment 15 commit-hook freebsd_committer freebsd_triage 2022-06-11 18:06:41 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=800a53b445e7eb113ba193b1ac98631299178529

commit 800a53b445e7eb113ba193b1ac98631299178529
Author:     Kirk McKusick <mckusick@FreeBSD.org>
AuthorDate: 2022-06-11 18:04:19 +0000
Commit:     Kirk McKusick <mckusick@FreeBSD.org>
CommitDate: 2022-06-11 18:05:14 +0000

    Bug fix to UFS/FFS superblock integrity checks when reading a superblock.

    One of the checks was that the cylinder group size (fs_cgsize)
    matched that calculated by CGSIZE(). The value calculated by CGSIZE()
    has changed over time as the filesystem has evolved. Thus comparing
    the value of CGSIZE() of the current generation filesystem may not
    match the size as computed by CGSIZE() that was in effect at the
    time an older filesystem was created. Therefore the check for
    fs_cgsize is changed to simply ensure that it is not larger than
    the filesystem blocksize (fs_bsize).

    Reported by: Martin Birgmeier
    Tested by:   Martin Birgmeier
    MFC after:   1 month (with 076002f24d35)
    PR:          264450
    Differential Revision: https://reviews.freebsd.org/D35219

 sys/ufs/ffs/ffs_subr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 16 Kirk McKusick freebsd_committer freebsd_triage 2022-06-11 18:15:41 UTC
(In reply to Martin Birgmeier from comment #14)
My 2-day kernel internals tutorial might be over your head. But given your ability to run FreeBSD and analyze superblock load error bugs, the main conference will not be over your head. Plus you will get to meet many other BSD users in your part of the world. You will have no travel expenses or hotel expenses to attend. In short you should head over to eurobsdcon.org and sign up today :-)
Comment 17 Martin Birgmeier 2022-06-12 07:41:08 UTC
O.k., understood!

Thank you for the fix.

-- Martin
Comment 18 Kirk McKusick freebsd_committer freebsd_triage 2022-09-05 05:56:45 UTC
This fix works. No need to MFC as this code appears only in 14.0.
Comment 19 Dennis 2023-07-23 20:23:32 UTC
I stumbled across this problem after a system update from 12.3-STABLE (pfsense 2.6.0) to 14.0-CURRENT (pfsense 2.7.0).
After that the systems cannot mount the root ufs anymore and will not boot.

-----------------------------------

UFS2 superblock failed: CGSIZE(fs) (16388) > fs->fs_bsize (16384)
Attempted recovery for standard superblock: failed
Attempted extraction of recovery data from standard superblock: failed
Attemp to find boot zone recovery data
Finding an alternate superblock failed.
Check for only non-critical errors in standard superblock
UFS2 superblock failed: CGSIZE(fs) (16388) > fs->fs_bsize (16384)
Failed, superblock has critical errors
Mounting from ufs:/dev/da0s1a failed with error 22; retrying for 3 more seconds
[repeating]

-----------------------------------

I have a few identical virtual systems of this kind. So I can reproduce this problem on multiple machines.
They were set up from a common image some years ago, underwent several sucessfull updates since then and worked flawlessly up till 12.3-STABLE.

I'm not sure whether the above fix is already in 14.0-CURRENT (pfsense 2.7.0)?
Apparently I have a difference of 4 in the blocksizes.

When I look at dumpfs before the update (on 12.3), bsize and cgsize match:

-----------------------------------
#dumpfs da0s1a | grep bsize
bsize   16384   shift   14      mask    0xffffc000
maxbsize 16384  maxbpg  2048    maxcontig 8     contigsumsize 8
sbsize  2048    cgsize  16384   csaddr  3000    cssize  4096
-----------------------------------

So 14.0 is calculating this differently?

Can this be corrected somehow without reformatting the drives?
These systems are remote, so this is problematic for me.
Comment 20 Mark Millard 2023-07-23 22:40:28 UTC
(In reply to Dennis from comment #19)

https://docs.netgate.com/pfsense/en/latest/releases/versions.html reports;

2.7.0
fa-check
2023-06-29
22.9
14.0-CURRENT@0c59e0b4e581
RELENG_2_7_0

But, looking, 14.0-CURRENT@0c59e0b4e581 is from 2023-Mar-30, long
predating the fix reported in comment #15 . The issue is a known
problem with the old release involved, one that has been fixed.

My guess here is that pfsense needs to be based on an updated
FreeBSD that has the fix reported in #15 . Time for a 2.7.1 ?
Comment 21 slb 2024-02-14 22:15:13 UTC
Created attachment 248471 [details]
First megabyte of da0s1a within the MBR+BSD image

I have a similar issue with old filesystem that loads on FreeBSD 9.x, but does not load on 13.2 or 14.0.
I have attached a first MB of the UFS in the MBR+BSD label partitioned drive.  This was originally set up on FreeBSD 4.x, and it might be CHS?  I redacted s1 after first 16s thru the 'swap'/s1b and first two sectors of the UFS which contained user data, so if this img appears unusable, I messed something up.  (I believe this drive had latent data at the time of creation, which may explain some of that user data)

UFS1 superblock failed: fs->fs_old_cpg (89) != 1 (1)
UFS1 superblock failed: fs->fs_old_spc (4096) != fs->fs_fpg * fs->fs_old_nspf (364544)
UFS1 superblock failed: fs->fs_old_ncyl (1885) != fs->fs_ncg (22)
Comment 22 Kirk McKusick freebsd_committer freebsd_triage 2024-02-17 22:19:32 UTC
Created attachment 248551 [details]
less restrictive UFS1 superblock check
Comment 23 Kirk McKusick freebsd_committer freebsd_triage 2024-02-17 22:21:40 UTC
(In reply to slb from comment #21)
Could you please apply the patch in Comment 22, recompile libufs, fsck_ffs, and your kernel to see if it resolves your problem.
Comment 24 slb 2024-02-19 16:53:07 UTC
After patch/rebuild, it does mount and work.

FWIW, I ran fsck on it R/O and brief output is below.  Not sure if it needs any further investigation.  Even in R/W, this fsck seems to reoccur.


** /dev/md1s1a (NO WRITE)
** Last Mounted on /mnt2
** Phase 1 - Check Blocks and Sizes
UFS1 cylinder group 21 failed: cgp->cg_old_ncyl ("16") != sblock.fs_old_cpg ("89")
CYLINDER GROUP 21: INTEGRITY CHECK FAILED
UNEXPECTED SOFT UPDATE INCONSISTENCY
REBUILD CYLINDER GROUP? no
-snip-
Comment 25 Kirk McKusick freebsd_committer freebsd_triage 2024-02-19 20:38:43 UTC
Created attachment 248613 [details]
additional proposed fsck_ffs fix

(In reply to slb from comment #24)

Please apply the proposed patch and recheck the run of fsck.
Comment 26 slb 2024-02-19 22:22:41 UTC
Looks good.  Clean R/W fsck, clean mount.

Thanks!
Comment 27 commit-hook freebsd_committer freebsd_triage 2024-02-20 00:18:05 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=b241767f8ef38f9ca7c109fe2fccd11ccbfaa4f0

commit b241767f8ef38f9ca7c109fe2fccd11ccbfaa4f0
Author:     Kirk McKusick <mckusick@FreeBSD.org>
AuthorDate: 2024-02-20 00:16:07 +0000
Commit:     Kirk McKusick <mckusick@FreeBSD.org>
CommitDate: 2024-02-20 00:17:05 +0000

    Eliminate unnecessary UFS1 integrity checks.

    The UFS1 integrity checks added in FreeBSD 14 were too aggressive
    for UFS1 filesystems created in FreeBSD 4 and 9 systems. This patch
    removes those tests which can be done safely since they are not
    relevant to the current implementation of UFS1.

    This is a follow-on report to bug report 264450 (comments 21-28).

    Reported by: slb@sonnet.com
    Tested by:   slb@sonnet.com
    PR:          264450
    MFC after:   1 week

 sbin/fsck_ffs/fsutil.c | 2 --
 sys/ufs/ffs/ffs_subr.c | 4 ----
 2 files changed, 6 deletions(-)
Comment 28 commit-hook freebsd_committer freebsd_triage 2024-03-05 00:09:33 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=fdfb8e783c3e8606d1294f9772fbd1496994b581

commit fdfb8e783c3e8606d1294f9772fbd1496994b581
Author:     Kirk McKusick <mckusick@FreeBSD.org>
AuthorDate: 2024-02-20 00:16:07 +0000
Commit:     Kirk McKusick <mckusick@FreeBSD.org>
CommitDate: 2024-03-05 00:09:10 +0000

    Eliminate unnecessary UFS1 integrity checks.

    The UFS1 integrity checks added in FreeBSD 14 were too aggressive
    for UFS1 filesystems created in FreeBSD 4 and 9 systems. This patch
    removes those tests which can be done safely since they are not
    relevant to the current implementation of UFS1.

    This is a follow-on report to bug report 264450 (comments 21-28).

    Reported by: slb@sonnet.com
    Tested by:   slb@sonnet.com
    PR:          264450

    (cherry picked from commit b241767f8ef38f9ca7c109fe2fccd11ccbfaa4f0)

 sbin/fsck_ffs/fsutil.c | 2 --
 sys/ufs/ffs/ffs_subr.c | 4 ----
 2 files changed, 6 deletions(-)
Comment 29 commit-hook freebsd_committer freebsd_triage 2024-03-05 00:19:37 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=8c6964b779cecbcf42475c896e5c61df192b80b9

commit 8c6964b779cecbcf42475c896e5c61df192b80b9
Author:     Kirk McKusick <mckusick@FreeBSD.org>
AuthorDate: 2024-02-20 00:16:07 +0000
Commit:     Kirk McKusick <mckusick@FreeBSD.org>
CommitDate: 2024-03-05 00:18:58 +0000

    Eliminate unnecessary UFS1 integrity checks.

    The UFS1 integrity checks added in FreeBSD 14 were too aggressive
    for UFS1 filesystems created in FreeBSD 4 and 9 systems. This patch
    removes those tests which can be done safely since they are not
    relevant to the current implementation of UFS1.

    This is a follow-on report to bug report 264450 (comments 21-28).

    Reported by: slb@sonnet.com
    Tested by:   slb@sonnet.com
    PR:          264450

    (cherry picked from commit b241767f8ef38f9ca7c109fe2fccd11ccbfaa4f0)

 sbin/fsck_ffs/fsutil.c | 2 --
 sys/ufs/ffs/ffs_subr.c | 2 --
 2 files changed, 4 deletions(-)
Comment 30 slb 2024-03-08 23:04:37 UTC
(In reply to commit-hook from comment #29)
I haven't built/tested on stable/13, but it looks like a deletion in ffs_subr.c might have been missed, possibly due to the test macro changing from CHK->FCHK between the major releases.

This would be about line 395:

CHK(fs->fs_old_cpg, !=, 1, %jd);
Comment 31 Kirk McKusick freebsd_committer freebsd_triage 2024-03-08 23:26:33 UTC
(In reply to slb from comment #30)
That fifth deletion was in code that was added in 14 and did exist in 13 and earlier. So I think that the commit was correct. But please do check and let me know.
Comment 32 slb 2024-03-11 17:17:29 UTC
(In reply to Kirk McKusick from comment #31)
I built the stable/13 commits against 13.3.  Still wouldn't mount with:

test# mount -o ro /dev/md0s1a /mnt
UFS1 superblock failed: fs->fs_old_cpg (89) != 1 (1)
mount: /dev/md0s1a: Invalid fstype: Invalid argument


Deleted line 395 of ffs_subr.c, rebuilt, and it mounted.

However, there also seem to be some different fsck tests versus 14:

# fsck /dev/md0s1a
UFS1 superblock failed: fs->fs_old_cpg (89) != 1 (1)
UFS1 superblock failed: fs->fs_old_spc (4096) != fs->fs_fpg * fs->fs_old_nspf (364544)
UFS1 superblock failed: fs->fs_old_ncyl (1885) != fs->fs_ncg (22)
Cannot find file system superblock
UFS1 superblock failed: fs->fs_old_cpg (89) != 1 (1)
UFS1 superblock failed: fs->fs_old_spc (4096) != fs->fs_fpg * fs->fs_old_nspf (364544)
UFS1 superblock failed: fs->fs_old_ncyl (1885) != fs->fs_ncg (22)
Cannot find file system superblock
Comment 33 commit-hook freebsd_committer freebsd_triage 2024-03-11 20:51:27 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=9cfb9a34bf154b27d21f9e2dfe8d2704ab664ed4

commit 9cfb9a34bf154b27d21f9e2dfe8d2704ab664ed4
Author:     Kirk McKusick <mckusick@FreeBSD.org>
AuthorDate: 2024-03-11 20:49:57 +0000
Commit:     Kirk McKusick <mckusick@FreeBSD.org>
CommitDate: 2024-03-11 20:50:48 +0000

    Eliminate unnecessary UFS1 integrity checks.

    This was missed in the cited cherry pick.

    This is a follow-on report to bug report 264450 (comments 21-33).

    Reported by: slb@sonnet.com
    Tested by:   slb@sonnet.com
    PR:          264450

    (cherry picked from commit b241767f8ef38f9ca7c109fe2fccd11ccbfaa4f0)

 sys/ufs/ffs/ffs_subr.c | 1 -
 1 file changed, 1 deletion(-)
Comment 34 Kirk McKusick freebsd_committer freebsd_triage 2024-03-11 20:56:03 UTC
(In reply to slb from comment #32)
You are correct that I missed the deletion of line 395 in ffs_subr.c in my MFC of the patch to 13. I have just now corrected that omission.

The complaints from fsck are probably because you failed to rebuild and reinstall libufs which incorporates ffs_subr.c and is used by fsck_ffs.
Comment 35 slb 2024-03-11 22:12:54 UTC
(In reply to Kirk McKusick from comment #34)
Exactly right; I forgot libufs.  After fixing that, fsck is clean too.

Thanks!