Bug 206521 - Can't decrypt disks on ZFS+Geli installation after order of devices changed
Summary: Can't decrypt disks on ZFS+Geli installation after order of devices changed
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 10.2-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-fs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-01-23 13:54 UTC by florian.ermisch
Modified: 2016-03-27 15:15 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description florian.ermisch 2016-01-23 13:54:48 UTC
(Rendered Markdown here: https://gist.github.com/0xf10e/ddefc6fad77d6b51672f)

## Preface

I added a 256GB mSATA SSD to my Lenovo x220
of which I planned to use a nice chunk as L2ARC
for the SATA HDD. Even without a SATA disk 
present the SSD showed up as HDD2 in the BIOS.

Made a fresh installation of FreeBSD 10.2 amd64
with the memstick image and chose ZFS+Geli. 
Booted, worked, everything just fine. I added a 
SATA HDD which the BIOS listed as HDD0 just as 
I expected from the previous SATA-only setup.
When I now tried to boot from the SSD I was 
prompted for the GELI key of a different device
than before, `ada1p3` instead of `ada0p3`.
But my passphrase wasn't accepted. Never.
Re-installed, added HDD, same result. 
Reinstalled with HDD present, the removed the 
HDD, same result (then prompted for `ada0p3` 
instead of `ada1p3`, of course).

When I restored the configuration which
was present during the installation process
(either remove the HDD or add it back in) 
everything worked fine again.

## Using bhyve to reproduce

Created a VM, added more disks:
```
floh@fuchi-cyber220:~:1305% iohyve info
Name           Size  RAM  CPU  OS       Loader
fbsd102        -     2G   1    default  bhyveload
fbsd102/disk0  2G    2G   1    default  bhyveload
fbsd102/disk1  2G    2G   1    default  bhyveload
fbsd102/disk2  4G    2G   1    default  bhyveload
```

Install FreeBSD 10.2 (the one I used for my laptop):

  ```
  floh@fuchi-cyber220:~:1305% sudo iohyve install fbsd102 FreeBSD-10.2-RELEASE-amd64-disc1.iso
  Installing fbsd102...
  floh@fuchi-cyber220:~:1307% sudo iohyve console fbsd102
  Starting console on fbsd102...
  ~~. to escape console [uses cu(1) for console]
  Connected
  ```

* Chose Auto (ZFS), added only ada2/disk2 to the rootpool.

```
┌────────────────ZFS Configuration───────────────────┐
│ Configure Options:                                 │
│ ┌────────────────────────────────────────────────┐ │
│ │ >>> Install          Proceed with Installation │ │
│ │ T Pool Type/Disks:   stripe: 1 disk            │ │
│ │ - Rescan Devices     *                         │ │
│ │ - Disk Info          *                         │ │
│ │ N Pool Name          zroot                     │ │
│ │ 4 Force 4K Sectors?  YES                       │ │
│ │ E Encrypt Disks?     YES                       │ │
│ │ P Partition Scheme   GPT                       │ │
│ │ S Swap Size          256m                      │ │
│ │ M Mirror Swap?       NO                        │ │
│ │ W Encrypt Swap?      YES                       │ │
│ └────────────────────────────────────────────────┘ │
├────────────────────────────────────────────────────┤
│             <Select>       <Cancel>                │
└────────────────────────────────────────────────────┘
```

* proceed with installation.
* remove disk1 from the VM:
  ```
  floh@fuchi-cyber220:~:1313% sudo iohyve remove fbsd102 disk1
  Are you sure you want to remove disk1 from fbsd102 [Y/N]? y
  floh@fuchi-cyber220:~:1314% iohyve info
  Name           Size  RAM  CPU  OS       Loader
  fbsd102        -     2G   1    default  bhyveload
  fbsd102/disk0  2G    2G   1    default  bhyveload
  fbsd102/disk2  4G    2G   1    default  bhyveload
  ```

* shuffle disks around:
  ```
  floh@fuchi-cyber220:~:1340% sudo zfs rename zroot/iohyve/fbsd102/disk{0,1}
  floh@fuchi-cyber220:~:1342% sudo zfs rename zroot/iohyve/fbsd102/disk{2,0}
  floh@fuchi-cyber220:~:1342% sudo zfs rename zroot/iohyve/fbsd102/disk{1,2}
  ```
* now the 4GB one is `disk0`:
  ```
  floh@fuchi-cyber220:~:1343% iohyve info
  Name           Size  RAM  CPU  OS       Loader
  fbsd102        -     2G   1    default  bhyveload
  fbsd102/disk0  4G    2G   1    default  bhyveload
  fbsd102/disk2  2G    2G   1    default  bhyveload
  ```

* boot the VM and try to get the rootpool imported:
```
floh@fuchi-cyber220:~:1344% sudo iohyve start fbsd102
Starting fbsd102... (Takes 15 seconds for FreeBSD guests)
floh@fuchi-cyber220:~:1344%
floh@fuchi-cyber220:~:1345% sudo iohyve console fbsd102
Starting console on fbsd102...
~~. to escape console [uses cu(1) for console]
Connected
  /
  ______               ____   _____ _____
 |  ____|             |  _ \ / ____|  __ \
 | |___ _ __ ___  ___ | |_) | (___ | |  | |
 |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
 | |   | | |  __/  __/| |_) |____) | |__| |
 | |   | | |    |    ||     |      |      |
 |_|   |_|  \___|\___||____/|_____/|_____/    ```                        `
                                             s` `.....---.......--.```   -/
 +============Welcome to FreeBSD===========+ +o   .--`         /y:`      +.
 |                                         |  yo`:.            :o      `+-
 |  1. Boot Multi User [Enter]             |   y/               -/`   -o/
 |  2. Boot [S]ingle User                  |  .-                  ::/sy+:.
 |  3. [Esc]ape to loader prompt           |  /                     `--  /
 |  4. Reboot                              | `:                          :`
 |                                         | `:                          :`
 |  Options:                               |  /                          /
 |  5. [K]ernel: kernel (1 of 2)           |  .-                        -.
 |  6. Configure Boot [O]ptions...         |   --                      -.
 |                                         |    `:`                  `:`
 |                                         |      .--             `--.
 |                                         |         .---.....----.
 +=========================================+


/boot/kernel/kernel text=0xfc8de8 data=0x1283b0+0x207880 syms=[0x8+0x145350+0x8+0x15fe20]
/boot/kernel/zfs.ko size 0x2f9b00 at 0x199e000
loading required module 'opensolaris'
/boot/kernel/opensolaris.ko size 0x6048 at 0x1c98000
/boot/kernel/geom_eli.ko size 0x21568 at 0x1c9f000
loading required module 'crypto'
/boot/kernel/crypto.ko size 0x35318 at 0x1cc1000
/boot/encryption.key size=0x1000
/boot/kernel/aesni.ko size 0x5a30 at 0x1cf8000
/boot/zfs/zpool.cache size=0x8bc
Booting...
Copyright (c) 1992-2015 The FreeBSD Project.

[... lots kernel messages ...]

ada1: <BHYVE SATA DISK 001> ACS-2 ATA SATA 3.x device
ada1: Serial Number BHYVE-D12E-A75C-27F5
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2048MB (4194304 512 byte sectors: 16H 63S/T 4161C)
ada1: Previously was known as ad6
random: unblocking device.
Timecounter "TSC-low" frequency 1345403547 Hz quality 1000
Enter passphrase for ada0p4:
GEOM_ELI: Wrong key for ada0p4. Tries left: 2.
Enter passphrase for ada0p4:
GEOM_ELI: Wrong key for ada0p4. Tries left: 1.
Enter passphrase for ada0p4:
GEOM_ELI: Wrong key for ada0p4. No tries left.
Enter passphrase for diskid/DISK-BHYVE-F485-E5AF-7C59p4:
GEOM_ELI: Wrong key for diskid/DISK-BHYVE-F485-E5AF-7C59p4. Tries left: 2.
Enter passphrase for diskid/DISK-BHYVE-F485-E5AF-7C59p4:
GEOM_ELI: Wrong key for diskid/DISK-BHYVE-F485-E5AF-7C59p4. Tries left: 1.
Enter passphrase for diskid/DISK-BHYVE-F485-E5AF-7C59p4:
GEOM_ELI: Wrong key for diskid/DISK-BHYVE-F485-E5AF-7C59p4. No tries left.
Enter passphrase for gpt/zfs0:
GEOM_ELI: Wrong key for gpt/zfs0. Tries left: 2.
Enter passphrase for gpt/zfs0:
GEOM_ELI: Wrong key for gpt/zfs0. Tries left: 1.
Enter passphrase for gpt/zfs0:
GEOM_ELI: Wrong key for gpt/zfs0. No tries left.
Trying to mount root from zfs:zroot/ROOT/default []...
Mounting from zfs:zroot/ROOT/default failed with error 2.

Loader variables:
  vfs.root.mountfrom=zfs:zroot/ROOT/default

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:tank
        cd9660:/dev/acd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/acd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>
panic: mountroot: unable to (re-)mount root.
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff80984e30 at kdb_backtrace+0x60
#1 0xffffffff809489e6 at vpanic+0x126
#2 0xffffffff809488b3 at panic+0x43
#3 0xffffffff809e7f5f at vfs_mountroot+0x1eaf
#4 0xffffffff808f03b3 at start_init+0x53
#5 0xffffffff8091244a at fork_exit+0x9a
#6 0xffffffff80d30d2e at fork_trampoline+0xe
Uptime: 4m16s
```

And, no, srsly, I don't need more than 9 tries to type "foobar".

Changing the order of disks back would fix the problem.
As there's no BIOS menu to choose the boot device in bhyve
having no bootcode in (the original) disk0 makes this a
tiny bit difficult with bhyve.
Comment 1 florian.ermisch 2016-01-23 14:02:58 UTC
PS: Would affect other people if they manage to change to order in which FreeBSD enumerates their disks.
Comment 2 Fabian Keil 2016-03-27 14:37:14 UTC
If I don't misinterpret the code, bsdinstall puts the plain
disk name into loader.conf when specifying the keyfile.

Updating the keyfile entry/entries should solve the problem.

To prevent the issue in the first place, bsdinstall could
use GPT labels.

The ElectroBSD installer cloudiatr does this and reordering
disks doesn't seem to cause any problems.

Example:
geli_gpt_rpool-ada0_keyfile0_load="YES"
geli_gpt_rpool-ada0_keyfile0_type="gpt/rpool-ada0:geli_keyfile0"
geli_gpt_rpool-ada0_keyfile0_name="/boot/rpool.key"
[...]
geli_gpt_rpool-ada3_keyfile0_load="YES"
geli_gpt_rpool-ada3_keyfile0_type="gpt/rpool-ada3:geli_keyfile0"
geli_gpt_rpool-ada3_keyfile0_name="/boot/rpool.key"

Putting the original disk name into the label is a bit
confusing when the disk name changes, though, so I'm considering
using generic names like disk1, disk2 etc. in the future.

CC'in Alan who worked on the relevant bsdinstall code in the past.
Comment 3 Allan Jude freebsd_committer 2016-03-27 15:15:58 UTC
(In reply to Fabian Keil from comment #2)
The bit of this related to the GELIBoot commit, was meant to quickly be followed by a patch to the installer to create a single encrypted pool, without using key files (because they are not supported by GELIBoot yet)

This change was delayed due to problems encountered, and my personal lack of time to fix them. I hope to have a patch for the installer soon that will not use key files if the configuration allows a GELIBoot style install (GPT, ZFS, single pool)

The reason GPT labels were not used was:
A) support for MBR
B) GPT labels disappear if the disk_ident label is used first (this caused issues when GPT labels were used for swap in previous versions)