Bug 251583 - FreeBSD/EC2 breakage w/ encrypted EBS volumes
Summary: FreeBSD/EC2 breakage w/ encrypted EBS volumes
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.2-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-virtualization (Nobody)
Depends on:
Reported: 2020-12-04 17:07 UTC by Alan Cummings
Modified: 2021-04-09 16:11 UTC (History)
6 users (show)

See Also:

output of AWS Console "Get instance screenshot" (366.71 KB, image/jpeg)
2020-12-04 17:07 UTC, Alan Cummings
no flags Details
4 snapshot images in order (148.77 KB, application/x-zip-compressed)
2020-12-07 21:27 UTC, Alan Cummings
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alan Cummings 2020-12-04 17:07:13 UTC
Created attachment 220253 [details]
output of AWS Console "Get instance screenshot"

I have been unable to successfully start the FreeBSD 12 AMI --
AMI id: ami-0cd0a66e446e9fa78
AMI Name: FreeBSD 12.2-RELEASE-amd64-f5af2713-6d84-41f0-95ea-2eaee78f2af4-ami-00be86d9bba30a7b3.4
AMI Location: aws-marketplace/FreeBSD 12.2-RELEASE-amd64-f5af2713-6d84-41f0-95ea-2eaee78f2af4-ami-00be86d9bba30a7b3.4

This is the x86 version of the FreeBSD owned AMI. The ARM version was successful on my first attempt (but I need the x86 architecture)

I am running in the us-west-1 region but I got the same results in the us-east-1 region

I have attached the output of selecting "Get instance screenshot" in the AWS console. 
There is no output when selecting "Get system log"
The console status shows "Instance reachability check failed"

I have tried these instance types:
- t2.micro
- t3.medium
- t3a.large
- c4.large

The results were the same in all instance types, with a small exception for t3.medium -- the instance screenshot seemed to be continually re-booting, but never got past the end of the attached screenshot
Comment 1 Colin Percival freebsd_committer 2020-12-07 19:01:52 UTC
Can you confirm that the instances are rebooting *repeatedly*?  It's normal for FreeBSD to reboot once, since upon booting for the first time it downloads and installs security updates.

What size of root disk are you using?

Are you using an encrypted disk?

Have you tried the non-Marketplace AMIs listed in the release announcement? https://lists.freebsd.org/pipermail/freebsd-announce/2020-October/001993.html
Comment 2 Alan Cummings 2020-12-07 21:27:14 UTC
Created attachment 220358 [details]
4 snapshot images in order

series of screenshots showing apparent repeated reboots. Images should be viewed in order *--1 to *--4
Comment 3 Alan Cummings 2020-12-07 21:29:20 UTC
I have tried the AMI ami-0e54f016b55b7f6ce for us-west-1 -- I get the same results

I have always used encrypted disks, sizes from 10G to 30G
Comment 4 Colin Percival freebsd_committer 2020-12-07 22:10:05 UTC
Thanks, there seems to be an issue with encrypted disks -- this isn't the first report we've had like this.
Comment 5 myBeNext 2021-01-12 10:31:06 UTC
We're getting the same. Other than using non-encrypted disks, is there a known work around for this?
Comment 6 Billy 2021-01-12 23:45:02 UTC
My workaround is as follows:

1. Launch an instance with an unencrypted disk.

2. Attached an encrypted disk to the instance.

3. Run bsdinstall to set up the encrypted disk with a fresh installation of FreeBSD.

4. Detach both disks from the instance.

5. Attach the encrypted disk to the instance as /dev/sda1 so that the instance boots from the fresh installation.

I personally like this approach because it lets me customize the install rather than use the settings selected in the AMI.

That said, if you like the AMI version, you can probably skip the bsdinstall step and instead just dd from a working unencrypted disk to an attached encrypted disk.
Comment 7 myBeNext 2021-01-13 14:29:10 UTC
Thank you for your quick response I will try this and confirm asap!
Comment 8 myBeNext 2021-01-21 17:02:42 UTC
(In reply to Billy from comment #6)
Works like a charm! Thank you very much!
Comment 9 Konstantin Pavlov 2021-02-12 12:08:42 UTC

We're hitting the same issue on eu-central-1 using ami-0d94faf9636228402, and unfortunately workaround is not feasible for us since all volumes we run must be encrypted as per AWS org policy.
Comment 10 Jim Pirzyk freebsd_committer 2021-04-05 16:16:23 UTC
I will add that I see this issue with the ami-01e92f8c87be32ba5 (us-west-2) which is 12.2-STABLE-2021-04-01
Comment 11 Alan Cummings 2021-04-09 16:11:41 UTC
(In reply to Billy from comment #6)
The workaround works well for me. thanks!
A couple other things to note on the workaround:
1. You can save the workaround as a new image, and then deploy new *encrypted* instances from that image.
2. You can also deploy *unencrypted* instances from the encrypted workaround image.