Bug 228557

Summary: 11.2-RC2 excessive swapping (swap on zvol)
Product: Base System Reporter: pascal.guitierrez
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: New ---    
Severity: Affects Only Me Keywords: regression
Priority: ---    
Version: 11.1-STABLE   
Hardware: amd64   
OS: Any   

Description pascal.guitierrez 2018-05-28 07:28:49 UTC
Since upgrading from 11.1-RELEASE p10 to 11.2-BETA3 on amd64 we are seeing excessive swapping, even when the system is idle.

Swap is on zvol, with system constantly swapping in/out for no apparent reason. There is no load on the system:

5:22PM  up 1 day, 7 mins, 1 user, load averages: 0.06, 0.20, 0.18

However the following lines from top confirm that excessive swapping is occurring:
Swap: 4096M Total, 158M Used, 3938M Free, 3% Inuse, 448K In, 136K Out
Swap: 4096M Total, 156M Used, 3940M Free, 3% Inuse, 8K In, 44K Out
Swap: 4096M Total, 157M Used, 3939M Free, 3% Inuse, 332K In, 24K Out


System has 64GB of RAM, 8-core Xeon E5-2620 v4 CPU, all memory and ZFS tunables are at defaults.
ZFS settings that were changed on swap zvol are:
volblocksize          4K
sync                  always
compression           off

The following process seems to constantly active:
pagedaemon{laundry: dom0}

Did not exhibit this behaviour on 11.1-RELEASE with exactly the same configuration.
Comment 1 pascal.guitierrez 2018-06-14 09:33:11 UTC
issue still exists on 11.2-RC2.
have deployed 11.2-RC2 onto another system and are seeing the same behaviour of excessive swapping.
reducing vfs.zfs.arc_max seems to help reduce the swapping.
Comment 2 Martin Birgmeier 2018-07-08 16:01:24 UTC
I have the same issue, albeit with swapping to a gmirror device instead of a zvol.

Bug #224975 may be related.

The following thread seems to be related: https://lists.freebsd.org/pipermail/freebsd-stable/2018-June/089132.html

I am now trying the patch in https://reviews.freebsd.org/D7538 (mentioned in that thread).

In general one can say that there has been a massive regression regarding ZFS + swap between 11.1 and 11.2.

-- Martin
Comment 3 Martin Birgmeier 2018-07-08 17:18:45 UTC
I have now deployed D7538 on three machines that had all been affected by excessive swapping. So far their swap usage stays at zero.

I have yet to check what happens under memory pressure.

-- Martin
Comment 4 Martin Birgmeier 2018-07-14 15:11:34 UTC
It seems that with this patch the ARC still grows until the system starts to swap again.

"swapoff -a" results in "swapoff: /dev/mirror/swap: Cannot allocate memory"

In 11.1 this used to work.

Also, using "sysctl vfs.zfs.arc_max=xxxxxxxxxx" to reduce the size of the ARC cache does not seem to free memory (top shows that wired memory remains unchanged).

ZFS + swap is very broken under 11.2.

-- Martin