Bug 234538 - freebsd-update hangs on upgrade from 11.2p7 to 12.0 on Google Compute Engine
Summary: freebsd-update hangs on upgrade from 11.2p7 to 12.0 on Google Compute Engine
Status: Closed Unable to Reproduce
Alias: None
Product: Base System
Classification: Unclassified
Component: conf (show other bugs)
Version: 11.2-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2019-01-01 06:30 UTC by Morgan Vandagriff
Modified: 2022-12-14 16:47 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Morgan Vandagriff 2019-01-01 06:30:33 UTC
freebsd-update fails to successfully upgrade a Google Compute Engine instance running 11.2p7 to 12.0-RELEASE. Hangs on the second freebsd-update install step after rebooting into the 12.0 kernel.

Replication steps using unmodified official image:

1.) Launch a Google Compute Engine instance using official 11.2 image: freebsd-11-2-release-amd64 from the freebsd-org-cloud-dev project. Retain all default instance settings. Note that, on first boot, the image successfully updates itself to the latest patch level.
2.) Login to the instance using method of your choosing (serial console, SSH, etc.)
3.) Run freebsd-update upgrade -r 12.0
4.) Run freebsd-update install
5.) Reboot as instructed
5.) Run freebsd-update install again
6.) Observe that the process hangs, with instance disk and CPU levels going to idle after a moment or two.

Note that I also tried running the second freebsd-update install step without first rebooting, to the same effect.
Comment 1 Colin Percival freebsd_committer freebsd_triage 2019-01-03 02:44:29 UTC
I don't think this is related to FreeBSD Update -- except that it does stress the disk a fair bit so it may be exposing an underlying issue.

Are these Xen nodes?  What disks are being used?  And what filesystem?
Comment 2 Morgan Vandagriff 2019-01-03 03:15:39 UTC
(In reply to Colin Percival from comment #1)

It appears that Google Compute Engine instances run on KVM [1]. The image uses a 32GB HDD-backed "standard persistent disk", however I was able to replicate the issue on another instance running on a 1TB SSD-backed boot volume.

The default image is provisioned with 1 vCPU and 3.75GB of RAM. All of my tests have been performed using UFS.


[1] https://cloud.google.com/compute/docs/faq
Comment 3 Colin Percival freebsd_committer freebsd_triage 2019-01-03 03:22:25 UTC
The only thing that comes to mind is that there was a UFS issue which exhibited symptoms like this a while back... but I think that was in 11.0 and was fixed long before 11.2...
Comment 4 Morgan Vandagriff 2019-06-13 13:42:48 UTC
I discovered a workaround which may be of interest: the issue observed with freebsd-update is apparently related to behavior of the py27-google-compute-engine package installed by default on the FreeBSD GCE images. If one uninstalls the package prior to upgrading, all proceeds normally.

I have a theory that the issue may proceed from py-google-compute-engine's tendency to overwrite certain files in /etc every few seconds whenever it is running. While I'm lacking an understanding of the internals of freebsd-update, it seems plausible that this could somehow give rise to the hangs I observed.
Comment 5 Graham Perrin freebsd_committer freebsd_triage 2021-12-19 16:44:28 UTC
Please, is anything like this still an issue?
Comment 6 Morgan Vandagriff 2021-12-20 18:54:35 UTC
It's been long enough that I no longer remember the details, but I believe the issue was resolved by stopping the Google Compute Engine management service before running freebsd-update. I'm not sure why, but it appeared that there was a conflict between the two processes.
Comment 7 Ed Maste freebsd_committer freebsd_triage 2022-12-14 16:47:09 UTC
Submitter doesn't have original details handy, and 11.x is after EOL in any case. If the same issue is reproducible in the future please reopen or submit a new PR.