Bug 243692 - file-backed VMs virtioblock report dev blocksize == underlying ZFS recordsize
Summary: file-backed VMs virtioblock report dev blocksize == underlying ZFS recordsize
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: 12.1-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-01-29 04:11 UTC by James Blachly
Modified: 2020-07-16 16:42 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description James Blachly 2020-01-29 04:11:17 UTC
Summary:
Briefly, file-backed virtio-block devices in bhyve are exposed differently than zvol backed VMs. This causes ZFS in illumos to segfault when a file-backed virtio-block device is used.

In illumos, the only difference in the block devices, as far as I can tell, is that zvol-backed devices have the ":lba-access-ok" flag set.


Possible cause:
The virtio block driver has a set of flags that expose supposed capabilities:

https://github.com/freebsd/freebsd/blob/0f0a35a04846fc4f4bdb6caa2852336d7de9447d/usr.sbin/bhyve/pci_virtio_block.c#L70-L84

However, these differ between file-backed VMs and zvol-backed VMs, possibly here:

https://github.com/freebsd/freebsd/blob/0f0a35a04846fc4f4bdb6caa2852336d7de9447d/usr.sbin/bhyve/pci_virtio_block.c#L366-L377

(if not specifically there, seems likely in that function, pci_vtblk_init)

This manifests in illumos as a segmentation fault when the command `zpool create` is issued on a bhyve-exposed virtio block device that is backed by a file.


I also filed an issue with illumos: https://www.illumos.org/issues/12237


Thanks in advance for help.
Comment 1 James Blachly 2020-01-29 15:32:32 UTC
On the illumos side, Hans Rosenfeld reports that the actual issue is that for file-backed virtio-block devices, bhyve reports the block size is equal to the recordsize of the underlying ZFS system, i.e. 128k in my case. This is larger than ASHIFT_MAX thus pool creation fails.

The :lba-access-ok flag is then unrelated but still of note.
Comment 2 Allan Jude freebsd_committer 2020-07-16 16:42:04 UTC
(In reply to James Blachly from comment #1)
When you are configuring the bhyve, when you create the virtio-blk device, it can take an optional parameter: sectorsize=logical[/physical]

So you can force the block size exposed to be smaller than that reported by ZFS

Possibly, you might want to set the zfs record size to be smaller, since you likely don't want that degree of write amplification in your file-backed VM images anyway.