Bug 229764

Summary:	Default settings allow system to wire all ram
Product:	Base System	Reporter:	Shane <FreeBSD>
Component:	kern	Assignee:	freebsd-bugs (Nobody) <bugs>
Status:	New ---
Severity:	Affects Only Me	CC:	grahamperrin
Priority:	---
Version:	CURRENT
Hardware:	Any
OS:	Any

Description Shane 2018-07-14 05:31:30 UTC

vfs.zfs.arc_max has a default of "all RAM less 1 GB, or one half of RAM, whichever is more." - from handbook 19.6.1. Memory used by ARC is wired to prevent it being swapped out.

vm.max_wired has a default setting of 30% of physical ram which limits what the kernel allocates as wired.

Currently, these two values that limit the amount of wired ram do not interact in any way. This means the default settings allow more than 100% of physical ram to be allocated as wired. This is an unrecoverable situation.

The default for arc_max should be reduced to less than 70% of physical ram.

I believe that setting arc_max should also be prevented from going higher than physical ram minus max_wired minus a safety buffer.

Comment 1 Andriy Gapon freebsd_committer

2018-10-04 14:37:02 UTC

First, ARC does not have any parallel or magic mechanism to wire memory. It uses uma(9) and those allocations go through the normal / common memory wiring mechanism.

Second, vm.max_wired only affects memory wiring from userland (e.g. mlock(2) calls). It cannot deny kernel memory allocations.

There is a different mechanism to stall kernel memory allocations (M_WAITOK) when the physical memory gets low.

So, I am not sure if this proposal makes much sense.

Comment 2 Shane 2018-10-06 07:59:57 UTC

(In reply to Andriy Gapon from comment #1)

I'm not looking to change anything as memory is being allocated, I'm only looking at the sysctl settings, specifically arc_max being limited. Initially the default settings but also when assigning a new value. Currently we have defaults and allow assigning values that let a system get into a non-responsive state.

On a machine with 8GB of physical RAM, max_wired is 2.4GB and arc_max is 7GB - that's RAM minus 1GB, so out of the box an 8GB system is allowed to wire more than 9GB, well maybe not allowed but two unrelated areas assuming they can allocate. I know from experience that a system with 8GB that has over 7GB wired only responds to the reset button. In this configuration, I expect sysctl should prevent arc_max being more than 5GB.

If max_wired is only user wired allocations, then kernel wiring is another factor to be added into the equation.

So initally, as max_wired is 30% of RAM, the default arc_max should not be set higher than 70% of RAM. When sysctl assigns a new value to arc_max the physical RAM and max_wired (and kernel wiring?) should be considered to check if the new value would allow the total wired RAM to be too high.

Comment 3 Graham Perrin freebsd_committer

2019-03-02 11:25:22 UTC

See also: 

- bug 187594 - [zfs] [patch] ZFS ARC behavior problem and fix

- bug 229670 - Too many vnodes causes ZFS ARC to exceed limit vfs.zfs.arc_max (high ARC "Other" usage)

Comment 4 Graham Perrin freebsd_committer

2021-11-03 01:12:25 UTC

(In reply to Shane from comment #2)

> … a system with 8GB that has over 7GB wired only responds to the 
> reset button. …

Do you find the same with more recent systems? 


Also, FYI <https://old.reddit.com/comments/pvsu2w/-/hecksww/?context=1> re: 

vfs.zfs.arc.sys_free

vfs.zfs.arc_free_target

Comment 5 Shane 2021-11-04 00:50:35 UTC

I've had 16G for some time now, so haven't had the system lock up like that.

Lately I am often running a bhyve instance, I notice with this that the amount of wired ram seems to stay much lower

my sys_free is zero and arc_free_target is 86463 - so about 350M