Bug 265549

Summary: Vultr Q35 hangs while booting during virtio-random initialization
Product: Base System Reporter: Bob Grant <bgfbsd>
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: New ---    
Severity: Affects Some People CC: alster, emaste, eugen, fred, grahamperrin, j.kelly.hays, mason
Priority: ---    
Version: 13.1-RELEASE   
Hardware: amd64   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253175
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254513
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269823
Attachments:
Description Flags
FreeBSD 13.1 Installer hang at virtio-random
none
Hang on manual load of module at single user prompt
none
CPU graph for 1 vCPU VM hanging for 16 hrs none

Description Bob Grant 2022-08-01 00:10:15 UTC 
Created attachment 235596 [details]
FreeBSD 13.1 Installer hang at virtio-random

Vultr QEMU Q35 hangs while booting during virtio-random initialization on FreeBSD-13.1-RELEASE and FreeBSD-14.0-CURRENT.

Vultr engineering has temporarily worked around it by blacklisting virtio-random on their FreeBSD images  and/or falling back to i440fx virtualization.  This requires that users installing FreeBSD from their own known media have to do one of the following:

* modify their ISO to blacklist virtio-random 
* do the kludge of installing a Vultr FreeBSD image (which defaults to i440fx emulation) and then reinstalling via an attached ISO image.  
* setting 'hint.vtrnd.0.disabled=1’ at the loader prompt and booting.

To quickly spin up a VM I use a raw image from the FreeBSD images -- customized with pre-loaded configuration -- but it has the exact same hang unless I modify it to blacklist virtio-random.  

This bug comment has information from Vultr engineering corroborating the issue.

bug #254513, comment #c32

I’ve attached images of the FreeBSD-13.1 RELEASE via the VNC console, first from the boot installer hang and subsequently from single user with a subsequent kldload virtio-random which causes an immediate hang.  Apologize for the images but the Vultr console doesn’t allow cut and paste of the screen.

Please let me know if I can provide any other information.
Comment 1 Bob Grant 2022-08-01 00:13:45 UTC 
Created attachment 235597 [details]
Hang on manual load of module at single user prompt

Here is the same hang after booting the installer disk single user and then doing a kldload virtio-random

The system is then unresponsive.

I could only take a screenshot because of vultr's video console.
Comment 2 Bob Grant 2022-08-01 18:05:24 UTC 
I read in one related thread about the virtio-random spinning 100% cpu.  All my hangs took place on a 1 vCPU VM.  

I just tested on a 2 vCPU VM and the system boots successfully.  I'm going to test a 1vCPU VM for a couple of hours and see if it eventually gets past the hang.  

It would appear that there is an unfortunate interaction with a single CPU and and the virtio-random kernel module.
Comment 3 Bob Grant 2022-08-01 18:26:58 UTC 
Ok, I had a test boot of FreeBSD 13.1 STABLE that was booted 15 hours ago and is still hung at the same place on a 1 vCPU system.

Thus the hang is more than just something that will resolve in an hour or two.
Comment 4 Bob Grant 2022-08-01 18:34:24 UTC 
Created attachment 235607 [details]
CPU graph for 1 vCPU VM hanging for 16 hrs

This is the CPU utilization as shown by the vultr dashboard during the boot of 13.1 stable ISO where the boot process was hung for 16 hrs.
Comment 5 Mason Loring Bliss freebsd_triage 2022-09-19 21:59:06 UTC 
I'd opened a case with Vultr and they've made a change (different emulated 
chipset? unsure - I don't have details) that allows stock ISOs to boot for 
me. For older VMs having an ISO attached and potentially for datacenters 
that haven't received the fix, the hint noted above seems to reliably work:

1. boot, access console, escape to loader prompt
2. using the lovely "paste" function at the left, set a hint:

    set hint.vtrnd.0.disabled=1

3. type 'boot'