Bug 277225 - emulators/xen-kernel - poor disk performance
Summary: emulators/xen-kernel - poor disk performance
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Roger Pau Monné
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-02-22 17:18 UTC by mgrooms
Modified: 2024-02-23 13:13 UTC (History)
1 user (show)

See Also:
bugzilla: maintainer-feedback? (royger)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description mgrooms 2024-02-22 17:18:08 UTC 
I've been doing some testing with Xen using a FreeBSD as a control domain and have observed a dramatic difference between dom0 disk performance and domU. Here is what I'm using as a testbed:

Xeon 8336 ( 32c/64t )
256G 3200mhz ECC RAM
16x SSD in RAID10

The difference between dom0 and bare metal disk I/O is pretty much the same so I didn't bother including it here. However, a single domU only seems to perform at around 10% the speed of the dom0 ...


FreeBSD 13.2 Dom0 - UFS
--------------------------------------------------------------------------------
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
test2.shrew.lab 32G  745k  99  1.5g  98  1.1g  75 1985k  98  1.3g  48 +++++ +++
Latency             11285us    8916us     539ms    8359us     464ms    4482us
Version  1.98       ------Sequential Create------ --------Random Create--------
test2.shrew.lab     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             39572us      20us      28us   24095us      15us      25us

FreeBSD 13.2 Dom0 - ZFS
--------------------------------------------------------------------------------
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
test2.shrew.lab 32G  301k  99  1.2g  88  842m  94  673k  99  1.8g  98 +++++ +++
Latency             26165us     134ms   11993us   12551us    5559us    4839us
Version  1.98       ------Sequential Create------ --------Random Create--------
test2.shrew.lab     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ 25662.595077  49 +++++ +++ +++++ +++ +++++ +++
Latency              1878us      61us     315ms    1811us      22us    2567us

Rocky Linux 9.3 DomU - UFS img
--------------------------------------------------------------------------------
Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
linux2.shrew.la 31G 1294k  97  112m   5 89.7m   6 2421k  98  159m   7  3009  76
Latency              6158us   12484us   31509us    3993us    9576us   33077us
Version  2.00       ------Sequential Create------ --------Random Create--------
linux2.shrew.lab    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency               678us     561us     533us     737us      15us     587us

Rocky Linux 9.3 DomU - ZFS img
--------------------------------------------------------------------------------
Delete files in random order...done.
Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
linux3.shrew.la 31G 1307k  98  145m   7 98.8m   7 2389k  98  196m   9  2584  66
Latency              6420us   15877us     132ms    4481us    5128us   28223us
Version  2.00       ------Sequential Create------ --------Random Create--------
linux3.shrew.lab    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency               926us     102us     287us     996us     162us     571us

FreeBSD 14 DomU - UFS img
--------------------------------------------------------------------------------
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
freebsd2.shrew. 32G  748k   5  195m   0 87.9m   0 1991k  13  151m   0  3845   0
Latency             11178us   93012us   43601us    8594us   30453us     159ms
Version  1.98       ------Sequential Create------ --------Random Create--------
freebsd2.shrew.lab  -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             38388us      17us      20us   37807us      13us      19us

FreeBSD 14 DomU - ZFS img
--------------------------------------------------------------------------------
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
freebsd3.shrew. 32G  750k   4  189m   0 95.6m   0 2043k   9  188m   0  2446   1
Latency             11275us     211ms     160ms    8840us   28129us     323ms
Version  1.98       ------Sequential Create------ --------Random Create--------
freebsd3.shrew.lab  -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             39562us      14us      19us   36463us      14us      25us

I assumed there would be overhead but 1/10th the performance doesn't feel right. Is there a problem here or are my expectations just way off?
Comment 1 Roger Pau Monné freebsd_committer freebsd_triage 2024-02-22 17:22:20 UTC 
(In reply to mgrooms from comment #0)
I'm afraid no one has been looking at performance, FreeBSD blkback is missing a bunch of performance related work that was done for Linux, like persistent grants , indirect segments or multiqueue support.  I'm afraid I don't have time to work on any of those extensions, patches welcome :).  Netback is in a similar state sadly.
Comment 2 mgrooms 2024-02-22 17:31:28 UTC 
Interesting. I wasn't aware of that. I'll try to fire up a Linux Dom0 and see if I can get some numbers for comparison. Thanks for the useful feedback.