Created attachment 220253 [details] output of AWS Console "Get instance screenshot" I have been unable to successfully start the FreeBSD 12 AMI -- AMI id: ami-0cd0a66e446e9fa78 AMI Name: FreeBSD 12.2-RELEASE-amd64-f5af2713-6d84-41f0-95ea-2eaee78f2af4-ami-00be86d9bba30a7b3.4 AMI Location: aws-marketplace/FreeBSD 12.2-RELEASE-amd64-f5af2713-6d84-41f0-95ea-2eaee78f2af4-ami-00be86d9bba30a7b3.4 This is the x86 version of the FreeBSD owned AMI. The ARM version was successful on my first attempt (but I need the x86 architecture) I am running in the us-west-1 region but I got the same results in the us-east-1 region I have attached the output of selecting "Get instance screenshot" in the AWS console. There is no output when selecting "Get system log" The console status shows "Instance reachability check failed" I have tried these instance types: - t2.micro - t3.medium - t3a.large - c4.large The results were the same in all instance types, with a small exception for t3.medium -- the instance screenshot seemed to be continually re-booting, but never got past the end of the attached screenshot
Can you confirm that the instances are rebooting *repeatedly*? It's normal for FreeBSD to reboot once, since upon booting for the first time it downloads and installs security updates. What size of root disk are you using? Are you using an encrypted disk? Have you tried the non-Marketplace AMIs listed in the release announcement? https://lists.freebsd.org/pipermail/freebsd-announce/2020-October/001993.html
Created attachment 220358 [details] 4 snapshot images in order series of screenshots showing apparent repeated reboots. Images should be viewed in order *--1 to *--4
I have tried the AMI ami-0e54f016b55b7f6ce for us-west-1 -- I get the same results I have always used encrypted disks, sizes from 10G to 30G
Thanks, there seems to be an issue with encrypted disks -- this isn't the first report we've had like this.
We're getting the same. Other than using non-encrypted disks, is there a known work around for this?
My workaround is as follows: 1. Launch an instance with an unencrypted disk. 2. Attached an encrypted disk to the instance. 3. Run bsdinstall to set up the encrypted disk with a fresh installation of FreeBSD. 4. Detach both disks from the instance. 5. Attach the encrypted disk to the instance as /dev/sda1 so that the instance boots from the fresh installation. I personally like this approach because it lets me customize the install rather than use the settings selected in the AMI. That said, if you like the AMI version, you can probably skip the bsdinstall step and instead just dd from a working unencrypted disk to an attached encrypted disk.
Thank you for your quick response I will try this and confirm asap!
(In reply to Billy from comment #6) Works like a charm! Thank you very much!
Hello, We're hitting the same issue on eu-central-1 using ami-0d94faf9636228402, and unfortunately workaround is not feasible for us since all volumes we run must be encrypted as per AWS org policy.
I will add that I see this issue with the ami-01e92f8c87be32ba5 (us-west-2) which is 12.2-STABLE-2021-04-01
(In reply to Billy from comment #6) The workaround works well for me. thanks! A couple other things to note on the workaround: 1. You can save the workaround as a new image, and then deploy new *encrypted* instances from that image. 2. You can also deploy *unencrypted* instances from the encrypted workaround image.
(In reply to Konstantin Pavlov from comment #9) I have the EBS volume encryption being enforced by AWS Organizations in my production AWS account too, but not in my test AWS account, which I used to identify EBS encryption as the issue. Since this problem didn't appear in the 12.1-RELEASE AMI, I took the path of: (1) Launching an EC2 instance with an encrypted EBS volume using a 12.1-RELEASE AMI. (2) Logging into the instance and upgrading it to 12.2-RELEASE with freebsd-update. (3) Creating a custom AMI from the instance with which to launch other 12.2 instances. A 12.1-RELEASE AMI was made available in the eu-central-1 region, as documented in that release's announcement: https://www.freebsd.org/releases/12.1R/announce/ I haven't found a way to search for and launch an EC2 instance from a particular AMI in the AWS Console, but I was able to launch an instance of the AMI in my region with the AWS CLI, using: aws ec2 run-instances --image-id <ami in your region> I'll warn that I haven't been able to get user data scripts to run properly when launching an EC2 instance from a custom AMI, as the standard work-around for Amazon Linux-derived custom AMIs (making "#cloud-boothook" to the first line of your user data script) doesn't seem to work. I take that as a sign I should switch to using Ansible to bootstrap my instances. My thanks to Alan Cummings for reporting this, Billy for pointing out a work-around, and to the FreeBSD team for investigating.
I'm seeing the same problem with the latest FreeBSD 12.3 RELEASE AMIs in the AWS Marketplace. Is there any official work around for this? I looked for a 12.1 AMI to upgrade but I don't see one available.
I think this is a bug in EC2. I'm poking some people to see if we can track down what's going wrong here.
Thanks for the info. Greatly appreciated. One more data point, I tried a 13 release instance with an encrypted volume and it works fine. It only appears to be the 12 instances that hang at boot time. Let me know if there's anything I can help test.
My contacts at Amazon tell me that they believe this is related to an internal AWS bug they're currently working on. I don't have a firm timeline for a fix being rolled out, but they've promised to keep me in the loop and when they get their bug fixed they're going to verify that the FreeBSD issue is resolved.
^Triage: is this aging PR still relevant?
I do not know, I am no longer at the job where I was using FreeBSD on AWS. Maybe someone else can confirm?
I don't think this was fixed, but my understanding is that newer FreeBSD releases haven't been affected by it. So I'm going to close as "overcome by events" -- if anyone can reproduce this on FreeBSD 13, please let me know and I'll start kicking some Amazonians about it again.