Bug 251046 - bhyve PCI passthrough does not work inside jail
Summary: bhyve PCI passthrough does not work inside jail
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.2-RELEASE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-11-11 11:58 UTC by neirac
Modified: 2021-04-11 17:17 UTC (History)
10 users (show)

See Also:


Attachments
Patch to make bhyve pci passthru work inside a jail (423 bytes, patch)
2020-11-11 11:58 UTC, neirac
no flags Details | Diff
devfs.rules use for testing (222 bytes, text/plain)
2020-11-11 11:58 UTC, neirac
no flags Details
jail.conf used for testing (532 bytes, text/plain)
2020-11-11 11:59 UTC, neirac
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description neirac 2020-11-11 11:58:02 UTC
Created attachment 219550 [details]
Patch to make bhyve pci passthru work inside a jail

on FreeBSD 12.2-RELEASE is possible to run bhyve inside a jail, but testing 
passthru revealed that this not work inside a jail.
The problem is that the jail needs r/w access to the following devices:

* /dev/mem
* /dev/io
* /dev/pci

/dev/pci is not a problem with a securelevel < 0. But the rest needs a patch.
I have attached a rough patch to make bhyve pci passthrough work inside a jail,I also have attached jail.conf and devfs.rules used for testing. 
This was tested in jail using vnet.
Comment 1 neirac 2020-11-11 11:58:45 UTC
Created attachment 219551 [details]
devfs.rules use for testing
Comment 2 neirac 2020-11-11 11:59:17 UTC
Created attachment 219552 [details]
jail.conf used for testing
Comment 3 Mark Johnston freebsd_committer 2020-11-23 16:11:27 UTC
PRIV_IO access is not required only by /dev/io, it is also required for sysarch(I386_SET_IOPERM), which is otherwise available to jailed processes.  So the patch definitely should not be committed.  A better solution would be to extend pci(4) so that bhyve can use it to do everything required for PCI passthrough.  Even then I'm not sure why it's useful to jail the bhyve process - what does it buy you?
Comment 4 Shawn Webb 2020-11-23 17:35:02 UTC
(In reply to Mark Johnston from comment #3)
> Even then I'm not sure why it's useful to jail the bhyve process - what does it buy you?

It allows folks to have a production version of bhyve in the host, and to develop and test the userland components of bhyve in a jail.
Comment 5 Peter Wemm freebsd_committer freebsd_triage 2021-04-08 21:03:49 UTC
Perhaps I'm missing the point, but rather than punching a giant hole in the jail security model (ie: by giving unconstrained ring 0 / kernel privileges to jailed processes), would it not be better to run these development/testing bhyve userland components in a simple chroot environment?
Comment 6 neirac 2021-04-09 13:05:37 UTC
(In reply to Peter Wemm from comment #5)

Currently illumos is able to do pci-passthrough with bhyve running inside a zone, that gives you an extra layer of security,if there is escape from the hypervisor then the attacker will land on a jail and not the host system.
 
Here are relevant links on how is used on illumos :

https://movementarian.org/blog/posts/2018-10-26-pci-pass-through-support-with-bhyve-and-smartos/

https://www.cyber-tec.org/2019/05/29/using-bhyve-pci-passthrough-on-omnios/

I think it would be nice to have this feature on FreeBSD jails, as Mark stated  "better solution would be to extend pci(4) so that bhyve can use it to do everything required for PCI passthrough."
I would like to explore this option any pointer on how to start would be really good.