Summary: Briefly, file-backed virtio-block devices in bhyve are exposed differently than zvol backed VMs. This causes ZFS in illumos to segfault when a file-backed virtio-block device is used. In illumos, the only difference in the block devices, as far as I can tell, is that zvol-backed devices have the ":lba-access-ok" flag set. Possible cause: The virtio block driver has a set of flags that expose supposed capabilities: https://github.com/freebsd/freebsd/blob/0f0a35a04846fc4f4bdb6caa2852336d7de9447d/usr.sbin/bhyve/pci_virtio_block.c#L70-L84 However, these differ between file-backed VMs and zvol-backed VMs, possibly here: https://github.com/freebsd/freebsd/blob/0f0a35a04846fc4f4bdb6caa2852336d7de9447d/usr.sbin/bhyve/pci_virtio_block.c#L366-L377 (if not specifically there, seems likely in that function, pci_vtblk_init) This manifests in illumos as a segmentation fault when the command `zpool create` is issued on a bhyve-exposed virtio block device that is backed by a file. I also filed an issue with illumos: https://www.illumos.org/issues/12237 Thanks in advance for help.
On the illumos side, Hans Rosenfeld reports that the actual issue is that for file-backed virtio-block devices, bhyve reports the block size is equal to the recordsize of the underlying ZFS system, i.e. 128k in my case. This is larger than ASHIFT_MAX thus pool creation fails. The :lba-access-ok flag is then unrelated but still of note.
(In reply to James Blachly from comment #1) When you are configuring the bhyve, when you create the virtio-blk device, it can take an optional parameter: sectorsize=logical[/physical] So you can force the block size exposed to be smaller than that reported by ZFS Possibly, you might want to set the zfs record size to be smaller, since you likely don't want that degree of write amplification in your file-backed VM images anyway.