Bug 234538 - freebsd-update hangs on upgrade from 11.2p7 to 12.0 on Google Compute Engine
Summary: freebsd-update hangs on upgrade from 11.2p7 to 12.0 on Google Compute Engine
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: conf (show other bugs)
Version: 11.2-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-bugs (Nobody)
Keywords: needs-qa
Depends on:
Reported: 2019-01-01 06:30 UTC by Morgan Vandagriff
Modified: 2019-06-13 13:42 UTC (History)
2 users (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Morgan Vandagriff 2019-01-01 06:30:33 UTC
freebsd-update fails to successfully upgrade a Google Compute Engine instance running 11.2p7 to 12.0-RELEASE. Hangs on the second freebsd-update install step after rebooting into the 12.0 kernel.

Replication steps using unmodified official image:

1.) Launch a Google Compute Engine instance using official 11.2 image: freebsd-11-2-release-amd64 from the freebsd-org-cloud-dev project. Retain all default instance settings. Note that, on first boot, the image successfully updates itself to the latest patch level.
2.) Login to the instance using method of your choosing (serial console, SSH, etc.)
3.) Run freebsd-update upgrade -r 12.0
4.) Run freebsd-update install
5.) Reboot as instructed
5.) Run freebsd-update install again
6.) Observe that the process hangs, with instance disk and CPU levels going to idle after a moment or two.

Note that I also tried running the second freebsd-update install step without first rebooting, to the same effect.
Comment 1 Colin Percival freebsd_committer 2019-01-03 02:44:29 UTC
I don't think this is related to FreeBSD Update -- except that it does stress the disk a fair bit so it may be exposing an underlying issue.

Are these Xen nodes?  What disks are being used?  And what filesystem?
Comment 2 Morgan Vandagriff 2019-01-03 03:15:39 UTC
(In reply to Colin Percival from comment #1)

It appears that Google Compute Engine instances run on KVM [1]. The image uses a 32GB HDD-backed "standard persistent disk", however I was able to replicate the issue on another instance running on a 1TB SSD-backed boot volume.

The default image is provisioned with 1 vCPU and 3.75GB of RAM. All of my tests have been performed using UFS.

[1] https://cloud.google.com/compute/docs/faq
Comment 3 Colin Percival freebsd_committer 2019-01-03 03:22:25 UTC
The only thing that comes to mind is that there was a UFS issue which exhibited symptoms like this a while back... but I think that was in 11.0 and was fixed long before 11.2...
Comment 4 Morgan Vandagriff 2019-06-13 13:42:48 UTC
I discovered a workaround which may be of interest: the issue observed with freebsd-update is apparently related to behavior of the py27-google-compute-engine package installed by default on the FreeBSD GCE images. If one uninstalls the package prior to upgrading, all proceeds normally.

I have a theory that the issue may proceed from py-google-compute-engine's tendency to overwrite certain files in /etc every few seconds whenever it is running. While I'm lacking an understanding of the internals of freebsd-update, it seems plausible that this could somehow give rise to the hangs I observed.