Bug 243692

Summary:	file-backed VMs virtioblock report dev blocksize == underlying ZFS recordsize
Product:	Base System	Reporter:	James Blachly <james.blachly>
Component:	bhyve	Assignee:	freebsd-virtualization (Nobody) <virtualization>
Status:	New ---
Severity:	Affects Some People	CC:	allanjude
Priority:	---
Version:	12.1-RELEASE
Hardware:	amd64
OS:	Any

Description James Blachly 2020-01-29 04:11:17 UTC

Summary:
Briefly, file-backed virtio-block devices in bhyve are exposed differently than zvol backed VMs. This causes ZFS in illumos to segfault when a file-backed virtio-block device is used.

In illumos, the only difference in the block devices, as far as I can tell, is that zvol-backed devices have the ":lba-access-ok" flag set.


Possible cause:
The virtio block driver has a set of flags that expose supposed capabilities:

https://github.com/freebsd/freebsd/blob/0f0a35a04846fc4f4bdb6caa2852336d7de9447d/usr.sbin/bhyve/pci_virtio_block.c#L70-L84

However, these differ between file-backed VMs and zvol-backed VMs, possibly here:

https://github.com/freebsd/freebsd/blob/0f0a35a04846fc4f4bdb6caa2852336d7de9447d/usr.sbin/bhyve/pci_virtio_block.c#L366-L377

(if not specifically there, seems likely in that function, pci_vtblk_init)

This manifests in illumos as a segmentation fault when the command `zpool create` is issued on a bhyve-exposed virtio block device that is backed by a file.


I also filed an issue with illumos: https://www.illumos.org/issues/12237


Thanks in advance for help.

Comment 1 James Blachly 2020-01-29 15:32:32 UTC

On the illumos side, Hans Rosenfeld reports that the actual issue is that for file-backed virtio-block devices, bhyve reports the block size is equal to the recordsize of the underlying ZFS system, i.e. 128k in my case. This is larger than ASHIFT_MAX thus pool creation fails.

The :lba-access-ok flag is then unrelated but still of note.

Comment 2 Allan Jude freebsd_committer

2020-07-16 16:42:04 UTC

(In reply to James Blachly from comment #1)
When you are configuring the bhyve, when you create the virtio-blk device, it can take an optional parameter: sectorsize=logical[/physical]

So you can force the block size exposed to be smaller than that reported by ZFS

Possibly, you might want to set the zfs record size to be smaller, since you likely don't want that degree of write amplification in your file-backed VM images anyway.