The sdcard images distributed by the project contain two level (MBR/BSD) partition table. The root filesystem is mounted using UFS label, "ufs/rootfs". Problem is, the label gets attached to the wrong device: mmcsd0s2 instead of mmcsd0s2a. The result is that on first startup, only the first level of partition table gets resized, which can result in significant confusion when moving the SD card to another system. Looking at the image on another system (using md(4)), it looks like this: [trasz@brick:~]% gpart show [snip] => 63 6291393 md0 MBR (3.0G) 63 961 - free - (481K) 1024 34816 1 !12 [active] (17M) 35840 6255616 2 freebsd (3.0G) => 0 6255616 md0s2 BSD (3.0G) 0 6255616 1 freebsd-ufs (3.0G) => 0 6255616 ufsid/5a570a62577263b8 BSD (3.0G) 0 6255616 1 freebsd-ufs (3.0G) => 0 6255616 ufs/rootfs BSD (3.0G) 0 6255616 1 freebsd-ufs (3.0G) [trasz@brick:~]% glabel list [snip] Geom name: md0s1 Providers: 1. Name: msdosfs/MSDOSBOOT Mediasize: 17825792 (17M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 524288 Mode: r0w0e0 secoffset: 0 offset: 0 seclength: 34816 length: 17825792 index: 0 Consumers: 1. Name: md0s1 Mediasize: 17825792 (17M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 524288 Mode: r0w0e0 Geom name: md0s2 Providers: 1. Name: ufsid/5a570a62577263b8 Mediasize: 3202875392 (3.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 18350080 Mode: r0w0e0 secoffset: 0 offset: 0 seclength: 6255616 length: 3202875392 index: 0 Consumers: 1. Name: md0s2 Mediasize: 3202875392 (3.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 18350080 Mode: r0w0e0 Geom name: md0s2 Providers: 1. Name: ufs/rootfs Mediasize: 3202875392 (3.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 18350080 Mode: r0w0e0 secoffset: 0 offset: 0 seclength: 6255616 length: 3202875392 index: 0 Consumers: 1. Name: md0s2 Mediasize: 3202875392 (3.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 18350080 Mode: r0w0e0 Notice how the last consumer is md0s2 instead of md0s2a.
Forgot one thing: trasz@brick:~]% doas diskinfo -v /dev/md0s2 /dev/md0s2 512 # sectorsize 3202875392 # mediasize in bytes (3.0G) 6255616 # mediasize in sectors 0 # stripesize 18350080 # stripeoffset 775 # Cylinders according to firmware. 128 # Heads according to firmware. 63 # Sectors according to firmware. MD-DEV115-INO80272 # Disk ident. Yes # TRIM/UNMAP support Unknown # Rotation rate in RPM [trasz@brick:~]% doas diskinfo -v /dev/md0s2a /dev/md0s2a 512 # sectorsize 3202875392 # mediasize in bytes (3.0G) 6255616 # mediasize in sectors 0 # stripesize 18350080 # stripeoffset 775 # Cylinders according to firmware. 128 # Heads according to firmware. 63 # Sectors according to firmware. MD-DEV115-INO80272 # Disk ident. Yes # TRIM/UNMAP support Unknown # Rotation rate in RPM Notice how the mediasize for both is of the same size. This matches UFS providersize: [trasz@brick:~]% doas dumpfs /dev/md0s2a| less magic 19540119 (UFS2) time Tue Feb 27 12:26:16 2018 magic 19540119 (UFS2) time Tue Feb 27 12:26:16 2018 superblock location 65536 id [ 5a570a62 577263b8 ] ncg 5 size 781952 blocks 756767 bsize 32768 shift 15 mask 0xffff8000 fsize 4096 shift 12 mask 0xfffff000 frag 8 shift 3 fsbtodb 3 minfree 8% optim time symlinklen 120 maxbsize 32768 maxbpg 4096 maxcontig 4 contigsumsize 4 nbfree 48681 ndir 1701 nifree 380950 nffree 158 bpg 20035 fpg 160280 ipg 80256 unrefs 0 nindir 4096 inopb 128 maxfilesize 2252349704110079 sbsize 4096 cgsize 32768 csaddr 5056 cssize 4096 sblkno 24 cblkno 32 iblkno 40 dblkno 5056 cgrotor 2 fmod 0 ronly 0 clean 1 metaspace 6408 avgfpdir 64 avgfilesize 16384 flags soft-updates fsmnt /media/rootfs volname rootfs swuid 0 providersize 781952 The 781952 * 4096 == 3202875392. This means from the UFS label point of view it can attach to either of those providers.
Is this the standard FreeBSD snapshot images?
Yes, I've just looked at https://download.freebsd.org/ftp/snapshots/arm/armv6/ISO-IMAGES/12.0/FreeBSD-12.0-CURRENT-arm-armv6-RPI-B-20180307-r330606.img.xz and I see the same thing.
Added manu@ as a CC since I think he may have just touched this.
Using this simple script (Which is basically what the release script is doing) : #!/bin/sh truncate -s 1g blah mddev=$(mdconfig -f blah) gpart create -s mbr ${mddev} gpart add -t '!12' -a 512k -s 50M ${mddev} gpart set -a active -i 1 ${mddev} newfs_msdos -L msdosboot -F 16 /dev/${mddev}s1 gpart add -t freebsd ${mddev} gpart create -s bsd ${mddev}s2 gpart add -t freebsd-ufs -a 64k /dev/${mddev}s2 newfs -U -L rootfs /dev/${mddev}s2a I have different result on 11.1-RELEASE : % sudo diskinfo -v /dev/md0s2 /dev/md0s2 512 # sectorsize 1020788736 # mediasize in bytes (974M) 1993728 # mediasize in sectors 0 # stripesize 52953088 # stripeoffset 494 # Cylinders according to firmware. 64 # Heads according to firmware. 63 # Sectors according to firmware. % sudo diskinfo -v /dev/md0s2a /dev/md0s2a 512 # sectorsize 1020723200 # mediasize in bytes (973M) 1993600 # mediasize in sectors 0 # stripesize 52953088 # stripeoffset 494 # Cylinders according to firmware. 64 # Heads according to firmware. 63 # Sectors according to firmware. % glabel list Geom name: md0s2a Providers: 1. Name: ufsid/5ab946bf62c34897 Mediasize: 1020723200 (973M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 52953088 Mode: r0w0e0 secoffset: 0 offset: 0 seclength: 1993600 length: 1020723200 index: 0 Consumers: 1. Name: md0s2a Mediasize: 1020723200 (973M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 52953088 Mode: r0w0e0 Geom name: md0s2a Providers: 1. Name: ufs/rootfs Mediasize: 1020723200 (973M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 52953088 Mode: r0w0e0 secoffset: 0 offset: 0 seclength: 1993600 length: 1020723200 index: 0 Consumers: 1. Name: md0s2a Mediasize: 1020723200 (973M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 52953088 Mode: r0w0e0 I think that this difference is why I did r326278
And yet in your example the providersizes are different - that's why glabel is able to tell which GEOM to attach to. Why do they end up being different for you, and the same for release images?
(In reply to Edward Tomasz Napierala from comment #6) Because 11.1 vs 12
Erm, but this _is_ 12-CURRENT. Besides, the revision you've mentioned is a change to the growfs rc script - and this problem happens earlier; it's visible in SD card images, before even flashing them to a card. Could you show the "gpart show" output for your case?
(In reply to Edward Tomasz Napierala from comment #8) Something changed in 12 which result in the behavior you see, which is why I commited the growfs change. Now the changes have also been mfc'ed and so it's happening on 11-STABLE and 12.
Yes, but what _exactly_ has changed? If you showed the "gpart show" results, I could take a look at how the partition layout differs.
(In reply to Edward Tomasz Napierala from comment #10) => 63 2097089 md0 MBR (1.0G) 63 961 - free - (481K) 1024 102400 1 !12 [active] (50M) 103424 1993728 2 freebsd (974M) => 0 1993728 md0s2 BSD (974M) 0 1993600 1 freebsd-ufs (973M) 1993600 128 - free - (64K) I'm currently trying to find a good 12-CURRENT commit, it's a bit long, I'm already in 2016 ...
See, the starting offsets of s2 and s2a are different in your case (1993728 vs 1993600), and that explains difference in mediasize. For the distribution images they are equal (6255616). I wonder why's that...
s2a should NOT start at zero offset relative to s2, ever. We struggled that enough during pre-GEOM era and finally made sysinstall to make non-zero offset (16) for first BSD partition relative to a slice. We should do the same here.
(In reply to Eugene Grosbein from comment #13) That is simply FUD from the era of dangeriously dedicated disks and the fact that 8k at the front of a, which is already reserved by UFS, might get over written by boot code. I have all my systems newer than 10, which includes 11 and ^/head running with offsets of 0 and there are no ill effects from doing this. Further what the person is seeing here is an offset of 128, which I SUSPECT is alignment to 64K, which may also be an alignment to a fake track boundary. We *may* want to try and improve code to optionally allow certain alignments to suck things as track, cylinder and arbitrary byte count alignments (VM's living in VMFS really want to be 1MB aligned. getting the guest using an alignment that matches the host is often very painful, but extremely important to things like ZFS performance)
(In reply to Rodney W. Grimes from comment #14) > That is simply FUD from the era of dangeriously dedicated disks and the fact that 8k at the front of a, which is already reserved by UFS, might get over written by boot code. It is not and the problem has nothing to do with "dangeriously dedicated". GEOM just now has serious issues resolving conflicts between distinct classes capable to "taste" same provider, like GEOM_PART and GEOM_LABEL, and others. It just have NO general mechanics to resolve them and we are obliged to have hacks like multiple kern.geom.label.*.enable loader tunnables for GEOM_LABEL, and multiple kern.geom.raid*enable for GEOM_RAID to "resolve" such conflicts manually. Many other classes have no such hacks, though (geom_cache etc.) > I have all my systems newer than 10, which includes 11 and ^/head running with offsets of 0 and there are no ill effects from doing this. You are just being lucky and/or using very simple GEOM structures. Non-zero offset for first partition within BSD label is another and simple way to resolve conflict between GEOM_PART and GEOM_LABEL.
(In reply to Eugene Grosbein from comment #15) Your saying the magic 16 offset of partition A is so that GEOM can taste a label? IMHO that is a very serious layering violation, and worse it puts magic values that are not documented into the UFS layer. I pretty much avoid most of GEOM, probably because of things like this.
(In reply to Rodney W. Grimes from comment #16) > Your saying the magic 16 offset of partition A is so that GEOM can taste a label? Opposite. I'm saying that zero offset of partition A produces situation when GEOM_DISK provider can be tasted and claimed by both of GEOM_PART and GEOM_LABEL classes and that's bad. Non-zero offset changes situation to simplier as now only GEOM_PART can open GEOM_DISK and later GEOM_LABEL is created on top of GEOM_PART only. > I pretty much avoid most of GEOM But you cannot. These days we run file systems (or corresponding nodes in /dev) on top of GEOM_PART (/dev/ada0s1a, /dev/ada1p1) or GEOM_LABEL (/dev/ufs/usr) or GEOM_DISK (/dev/ada2) or GEOM_RAID (/dev/raid/r0) or GEOM_MAP (/dev/map/u-boot) only. geom_ctl.c, geom_io.c, geom_kern.c, geom_dev.c, geom_disk.c, geom_event.c etc. are NOT optional kernel components.
Possible source of the difference: With kern.geom.part.mbr.enforce_chs=0 from manu's script in comment 5 I get: # /dev/md0s2: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 1993600 0 4.2BSD 0 0 0 c: 1993728 0 unused 0 0 # "raw" part, don't edit With kern.geom.part.mbr.enforce_chs=1 I get: # /dev/md0s2: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 1993600 41 4.2BSD 0 0 0 c: 1993761 0 unused 0 0 # "raw" part, don't edit The sysctl default changed in r269857. Unfortunately this is a property of the build host so it won't be easy to say with certainty if this is the reason for the change, but we can likely infer it. That said, it doesn't really matter: if this is the reason it's really only by accident that it worked before, and returning to the original behaviour is not really a fix. For reference, on a 10.1 image (FreeBSD-10.1-RELEASE-arm-armv6-RPI-B.img) I see: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 1918080 105 4.2BSD 0 0 0 c: 1918224 0 unused 0 0 # "raw" part, don't edit
Yea, back in the day we'd always offset both s1 and s1a from the start of the volume / partition respectively. s1 was offset by the first cylinder on the belief, half-true, half-cargo cult, that it made a big difference. s1a was offset to start at a good offset for striping, except when it was just hard-wired (depending on how far you go back). The notion of s1 and s1a starting at the same location is fairly new and does cause some tasting confusion, as has been noted (since in the early days of geom, we'd never have partitions that lined up like that). It's a bit of a race in the tasting, but that's a fixable bug should someone want to spend time to do it. It's really glabel's fault for trying to hide gpart's partition, which is where the heart of the race lies. We shouldn't try to do that hiding, but instead fail any opens via multiple paths while the other path is open... Given the nature of geom's tasting, there's likely multiple issues, but this is the one I chased down years ago and decided fixing it was more time than I have available to the problem when it was easy enough to offset partitions. We should fix it, but until we do, manu needs to change his scripts :)
I checked RPI-B 10.4, 11.0, and 11.1 images, and all have a: offset 105. FreeBSD-11.1-STABLE-arm-armv6-RPI-B-20180412-r332428.img 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 2061312 0 4.2BSD 0 0 0 c: 2061312 0 unused 0 0 # "raw" part, don't edit FreeBSD-12.0-CURRENT-arm-armv6-RPI-B-20180408-r332309.img 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 6255616 0 4.2BSD 0 0 0 c: 6255616 0 unused 0 0 # "raw" part, don't edit This needs to be a blocking issue for 11.2
It seems undesirable to me that a zero offset puts the bsdlabel partition table itself inside of the a: partition, with the result that a command like "dd if=/dev/zero of=/dev/md0s2a" will overwrite the bsdlabel.
a: is not magic, *any* partition a-h can start at offset 0, nothing has ever said what order they have to appear on the disk, though traditionally tools do this. BUT it should not be assumed to be true. Would a dd to the c partition be ok to wipe the label? The label has a special space reserved in UFS so that a partition *can* start at 0,and dd'ing over a zero offset ufs file system wiping the label is expected behavior. I actually expect that on most *unix system a dd to /dev/xxx0a wipes the label.
(In reply to Rodney W. Grimes from comment #22) > a: is not magic, *any* partition a-h can start at offset 0, nothing has ever > said what order they have to appear on the disk, though traditionally tools > do this. BUT it should not be assumed to be true. Sure, my comments are in the context of the re@-produced images, which previously had no partition with offset 0 and now have a: with offset 0. To be clear, I'd prefer that no partition have offset 0. > Would a dd to the c partition be ok to wipe the label? Yes, if we exposed the 'c' partition I would expect it to wipe the label, as it is explicitly documented as referring to the raw part. But note that g_part_bsd does not create a device for the 'c' partition, so the question is moot. > The label has a special space reserved in UFS so that a partition *can* start > at 0,and dd'ing over a zero offset ufs file system wiping the label is expected > behavior. The relationship between bsdlabel (disklabel), boot code, and UFS sounds like the real layering violation here, with each of them requiring knowledge of the others. While looking into this now I found that DragonFly tidied it up in a 'disklabel64' scheme that ae@ imported in r267359: Add disklabel64 support to GEOM_PART class. This partitioning scheme is used in DragonFlyBSD. It is similar to BSD disklabel, but has the following improvements: * metadata has own dedicated place and isn't accessible through partitions; * all offsets are 64-bit; * supports 16 partitions by default (has reserved place for more); * has reserved place for backup label (but not yet implemented); * has UUIDs for partitions and partition types;
While partitions *CAN* start at 0, we shouldn't do that by default. We didn't used to except for 'c' and 'd' in various flavors (back when there was no raw disk access apart from going through these hacks). We should start 'a' at some sensible stripe size. In the past this has been 16, but then changed to something weird. The change back to '0' most likely is because mkimg didn't grok the need to have sensible offsets. Back in the day, 16 was sensible. These days, when we growfs the images on NAND media, 256 or 512 is likely a better default for images we create for distribution. Now, whether that's done with the new offset stuff that benno just added to mkimg, or hacks to mkimg is an exercise for the reader. I can make good arguments both ways. disklabel has always been a bit of an odd duck because it was devised on systems without layered partitioning, then adapted to the PC which had layered partitioning. At first, we coped by having 'a' start at offset 16. There was no good reason for '16' other than it was bigger than where the label was stored. It just so happens it's the same size as the boot blocks, but UFS1 already has 8k at the start of the filesystem to cope with boot block issues (UFS2 increased this from 8k to 256k), so the 16 offset for 'a' wasn't strictly needed because boot1 lived in block 16 and boot2 lived in 17-31, offset from the start of the slice. It's an unfortunate layering violation, but one that we've had since the 386BSD days and there's 0 chance of changing it. The ideal location of the offset for the 'a' partition is stripesize (or more accurately stripesize rounded up to next stripe-size multiple in the absolute block of the underlying filesystem). However, for a release image, that value is unknowable, hence my recommendation for 256k. It's a nice tradeoff between what we'd do at install (1MB I hhink) and wasting a lot of space. Though, to be honest, the images are large enough even a 1MB alignment wouldn't be bad, and would give ideal performance in all but the newest SSDs that are available only as samples with NDAs :). And even then 1MB isn't terrible on those drives... I wish we could go to GPT partitioning on arm, but some SoCs read the files they need to bootstrap u-boot off the first FAT partition they find, so that's a non-starter. and no, I'm not a fan of going to disklabel64 from DFBSD, and certainly not in 11.2, but I won't bar the door for aexerpimentation in -current.
'of the underlying filesystem' should read 'of the underlying disk' that is, it should start on an absolute 1M boundary ideally. Second, I just remembered that the 16 offset for slice 'a' was to keep things 8k aligned and skip the MBR part of the disk. Or at least it had that effect. This is the dangerously dedicated case where you wanted to have an invalid MBR so BIOSes wouldn't look at them too closely. There are no doubt a dozen more bodies buried in this landfill than I'm privy too... Suffice to say, I think starting 'a' at absolute LBA 1MB makes the most sense though.
(In reply to Warner Losh from comment #24) Yes, I'd say there are three separate but related issues here, and this specific PR applies only to the last one: 1. what should g_part_bsd allow for offset 2. what should it default to 3. what offset should be used for our embedded sd card images For this PR I like imp's suggestion of aligning the UFS image to 1MB bytes relative to the disk. (For g_part_bsd I have no desire to disallow 0 offset, but agree with imp that we should default to something >= 16, probably 1M bytes as suggested.) For this PR mkimg is out of scope: these images are created using mdconfig + gpart. We should also make sure mkimg has sensible defaults and alignment control, and switch to mkimg, later on. All of that said - do we actually need a disklabel? For these Arm images we could use MBR with a FAT boot fs and UFS directly on slice 2?
(In reply to Ed Maste from comment #26) 1. We have created a lot systems with any offset, so g_part_bsd should continue to allow it. 2. We should default to multiple of stripe size, per my message. 3. We should start 'a' at 1MB. The boot blocks assume bsdlabel, and I think /boot/loader does as well. I don't think we can make just put things on slice 2 w/o a bsdlabel.
I personally always create large enough swap-dedicated 'b' partition at zero offset of the beginning of the slice and 'a' partition next to it: gpart create -s BSD ada0s1 gpart add -i 2 -s 8G ada0s1 # creaye /dev/ada0s1b at start gpart add ada0s1 # create /dev/ada0s1a after In case of HDD, this dedicates quickest parts of rotating media to swap. Also, this completely eliminates the problem we discuss here even if swap partition is significantly less (f.e. 1G one used for mini-dumps only). Also, there is another good "magical" constant 504 in case of non-existing swap partition: - this gives us 504*512 = 258048 bytes offset not being too small nor too large; - it is aligned to both of 4096 and old 63-sector margins and hence, it is compatible with any value of sysctl kern.geom.part.mbr.enforce_chs or any similar trace of ancient times; - 258048 bytes offset avoids layering violations.
A commit references this bug: Author: gjb Date: Wed Apr 18 16:22:23 UTC 2018 New revision: 332731 URL: https://svnweb.freebsd.org/changeset/base/332731 Log: MFC r326278 (manu): growfs: Commit the changes after expanding the partition This fix the problem in arm snapshot present since at least 6 months where growfs was failing at firstboot and dropped you in a single user shell. Note: In addition to this merge, kern.geom.part.mbr.enforce_chs has been enabled on the build machine to mitigate against the issue in the PR referenced. PR: 226536 Sponsored by: The FreeBSD Foundation Changes: _U stable/10/ stable/10/etc/rc.d/growfs _U stable/11/ stable/11/etc/rc.d/growfs
In addition to merging r326278, kern.geom.part.mbr.enforce_chs has been enabled on the build machines to prevent the issue originally described in the PR. I believe this can be closed now.
We should start using -b to configure partition layout with some intent rather than relying on uncontrolled side-effects of host config, but this should at least be back to the way it was before.