Bug 219355 - Heavy disk activity in bhyve deadlocks host
Summary: Heavy disk activity in bhyve deadlocks host
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.0-STABLE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-virtualization mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-05-17 13:48 UTC by Joseph Mulloy
Modified: 2017-05-20 14:39 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Joseph Mulloy 2017-05-17 13:48:01 UTC
Hello,

I have a somewhat complicated server setup that I am trying to run bhyve on. I did not have this problem on FreeBSD 10, although it may have been due to my zpool not being as full. My main os pool is a small unencrypted mirror. My jails and bhyve VMs are stored on a separate pool named "data" that is made up of 4x 3TB WD Red drives in a pair of mirrors (RAID 10) with each disk being encrypted via geli and the geli device being passed to ZFS. I am using chyves to manage my bhyve VMs. I have found that doing heavy disk activity will reliably deadlock the host system and I will then need to reboot/reset it. The VM disks are stored in zvols. So far I have been able to trigger this condition by attempting to install Windows 7, where the host crashes when Windows starts copying files. I have also managed to trigger it a couple times by trying to assemble a jail by hand in a FreeBSD guest by copying a template directory holding a FreeBSD installation. The host system has 32GB of RAM and I only give 2GB to the VM so I should have plenty of memory. Below is the output of top during one of the deadlocks when trying to install Windows. The state of the bhyve process is kqread. I was able to successfully install Windows 7 by storing the guest on a separate geli encrypted pool. At one point my pool was 85% full. I cleaned it up to be only 50% full but I'm still having this issue. I think I have somehow got my pool in a state where it's going to keep having this problem. I would like to fix it by recreating my pool but I would like to debug it first in case there is some bug that can be fixed. I don't know how to debug this further so if someone could provide me with some instructions or commands to debug this further I would appreciate it.

root@server1:~ # chyves win7 get all
Checking for newer version of chyves on the master branch from https://github.com/chyves/chyves/raw/master/sbin/chyves.
Setting global property 'check_for_updates_last_check' to value: '20170517'
On current version, will check again on: 20170524
Setting global property 'check_for_updates_last_check_status' to value: '0'
Getting all win7's properties...
bargs                                -H -P -S
bhyve_disk_type                      ahci-hd
bhyve_net_type                       virtio-net
bhyveload_flags
chyves_guest_version                 0300
cpu                                  1
creation                             Created on Mon Oct 10 22:06:33 UTC 2016 by chyves v0.2.0 2016/09/11 using __create()
description                          -
eject_iso_on_n_reboot                2
loader                               uefi
net_ifaces                           tap55
notes                                -
os                                   default
ram                                  2G
rcboot                               0
revert_to_snapshot_method            off
revert_to_snapshot
serial                               nmdm55
template                             no
uefi_console_output                  vnc
uefi_firmware                        BHYVE_UEFI_20160704_1.fd
uefi_vnc_ip                          10.2.4.50
uefi_vnc_mouse_type                  ps2
uefi_vnc_pause_until_client_connect  yes
uefi_vnc_port                        5900
uefi_vnc_res                         1024x768
uuid                                 cce87028-8f35-11e6-86a3-94de80a12470

chyves version: chyves v0.2.0 2016/09/11

root@server1:~ # uname -a
FreeBSD server1.jdmulloy.net 11.0-RELEASE-p9 FreeBSD 11.0-RELEASE-p9 #0: Tue Apr 11 08:48:40 UTC 2017     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

483 processes: 2 running, 451 sleeping, 30 waiting
CPU: 16.3% user,  0.0% nice, 83.7% system,  0.0% interrupt,  0.0% idle
Mem: 2357M Active, 864M Inact, 28G Wired, 4520K Free 
ARC: 4978M Total, 3836K MFU, 115M MRU, 2456M Anon, 1602M Header, 800M Other
Swap: 16G Total, 257M Used, 16G Free, 1% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
10662 root         15  52    0 45776K 17136K uwait   2   2:11  72.10% filebeat
10136 root         12  20    0 43344K 16612K uwait   2   3:38  51.58% filebeat
11208 root         12  20    0 43344K 16840K tx->tx  2   2:25  35.52% filebeat
 5373 root         14  20    0 43600K 14824K pfault  1   1:51  27.82% filebeat
  840 root         11  20    0 39120K 11004K pfault  0   0:39  12.06% filebeat
  792 zabbix        1  20    0 30596K  4204K RUN     3   0:32  12.00% zabbix_agentd
13401 root         21  20    0  2127M  1963M kqread  1   3:18   3.23% bhyve
 8940 root          1  20    0 26264K  5000K CPU3    3   0:01   0.19% top
11613 root         11  47    0 41100K 14368K uwait   3   0:00   0.12% filebeat
 7411 root         14  20    0 45100K 17612K tx->tx  3   0:00   0.09% filebeat
 6504 root         10  52    0 39984K 13992K uwait   3   0:01   0.09% filebeat
 1125 root          1  20    0 22004K  3692K select  3   0:02   0.06% tmux
 1105 jdmulloy2     1  20    0 46760K  5688K select  3   0:01   0.06% mosh-server
Comment 1 Fabian Keil 2017-05-18 08:17:52 UTC
Are you using geli for the swap device as well?

This is known to cause deadlocks under memory pressure (which your top output indicates):
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209759
Comment 2 Peter Grehan freebsd_committer 2017-05-18 12:52:00 UTC
Not sure if you've limited your ARC cache - you'll really want to do that or it will try and force out your bhyve guest. Judging by the amount of swap that has been allocated, that may have already happened.
Comment 3 Joseph Mulloy 2017-05-19 06:26:19 UTC
Swap is geli encrypted and I am not limiting my ARC. I could try limiting the ARC, but it doesn't seem like I use much swap before it stalls. I will try limiting the ARC and see if that works.
Comment 4 Joseph Mulloy 2017-05-19 14:53:46 UTC
I just tried again with the ARC limited to 12 GB (out of 32 GB) and it still crashes. This time I got a ton of "swap_pager: indefinite wait buffer" messages on the console. From the output of top, which I have pasted below I can see that the ARC was on 5 GB but the wired memory is up to 30GB. The guest has the -S flag (wire guest memory) on the bhyve command since that is the default in chyves. I have tried without it before since I suspected a memory problem. I will try again without it.

471 processes: 1 running, 447 sleeping, 23 waiting
CPU: 43.8% user,  0.0% nice, 29.5% system,  1.4% interrupt, 25.4% idle
Mem: 1106M Active, 317M Inact, 30G Wired, 55M Cache, 18M Free
ARC: 5135M Total, 2282K MFU, 40M MRU, 2680M Anon, 1510M Header, 903M Other
Swap: 16G Total, 1221M Used, 15G Free, 7% Inuse, 12K In, 141M Out

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 3091     88       25  20    0   460M 21832K select  0   0:19  85.90% mysqld
 3593     88      143  20    0   604M   195M select  3   2:39  83.29% mysqld
39084 root         21  20    0  2127M  1969M kqread  2   3:12   4.43% bhyve
 4256 zabbix        1  20    0   105M  4584K pfault  3   0:16   0.45% zabbix_server
 4264 zabbix        1  20    0   105M  3272K sbwait  2   0:16   0.44% zabbix_server
 4278 zabbix        1  20    0   105M  8292K nanslp  0   0:16   0.38% zabbix_server
 4294 zabbix        1  20    0   105M  4732K nanslp  0   0:16   0.38% zabbix_server
 4295 zabbix        1  20    0   105M  8396K sbwait  2   0:15   0.35% zabbix_server
 4233 zabbix        1  20    0   102M  6948K nanslp  0   0:03   0.34% zabbix_server
 1191 root          1  20    0 26264K  3716K CPU3    3   0:43   0.11% top
 1158 jdmulloy2     1  20    0 51156K  5224K select  3   0:09   0.09% mosh-server
 1170 root          1  20    0 22420K  3340K select  0   0:22   0.08% tmux
 4266 zabbix        1  20    0   105M  4608K sbwait  2   0:15   0.03% zabbix_server
 4240 zabbix        1  20    0   105M  2256K pfault  2   0:16   0.02% zabbix_server
 2090 root         14  52    0 45048K  9464K uwait   1   0:06   0.02% filebeat
 4544 root         11  52    0 43672K  9840K uwait   1   0:06   0.02% filebeat
 3786 root         13  52    0 44984K 10312K uwait   3   0:07   0.02% filebeat
 4257 zabbix        1  20    0   105M  8360K sbwait  1   0:16   0.01% zabbix_server
 1889 root         14  52    0 44056K 10308K uwait   1   0:06   0.01% filebeat
 1487 root         10  52    0 44600K  9372K uwait   3   0:06   0.01% filebeat
 4247 zabbix        1  20    0   105M  4868K sbwait  3   0:16   0.01% zabbix_server
 2367 root         12  20    0 42676K  8788K tx->tx  2   0:06   0.01% filebeat
 4329 zabbix        1  20    0   105M  1952K nanslp  2   0:16   0.01% zabbix_server
  772 zabbix        1  20    0 31004K  2300K nanslp  0   0:02   0.01% zabbix_agentd
 1662 zabbix        1  20    0 31004K  1972K nanslp  2   0:02   0.01% zabbix_agentd
 3373 zabbix        1  20    0 31004K  2412K nanslp  0   0:02   0.01% zabbix_agentd
 2609 zabbix        1  20    0 31004K  2612K tx->tx  1   0:01   0.01% zabbix_agentd
 3405 root         15  52    0 46232K 10464K uwait   1   0:06   0.01% filebeat
  808 root          1  20    0 10436K  1260K select  0   0:03   0.01% powerd
 2605 zabbix        1  20    0 31004K  2520K nanslp  3   0:02   0.01% zabbix_agentd
 1461 zabbix        1  20    0 31004K  1984K nanslp  0   0:02   0.01% zabbix_agentd
 2635 root         15  52    0 45240K 10552K uwait   1   0:07   0.01% filebeat
 3984 root         12  20    0 42808K 10000K uwait   0   0:06   0.01% filebeat