Bug 138790

Summary:

[zfs] ZFS ceases caching when mem demand is high

Product:

Base System

Reporter:

Peter Much <pmc>

Component:

kern

Assignee:

Andriy Gapon <avg>

Status:

Closed FIXED

Severity:

Affects Only Me

Priority:

Normal

Version:

7.2-STABLE

Hardware:

Any

OS:

Any

Attachments:

Description	Flags
file.diff	none

Description Peter Much 2009-09-13 23:50:05 UTC

The system was originally equipped with 256 MB memory, and runs a lot
of (seldom used) processes, so there was about 350 MB paged out and
some ongoing competition for memory - which was handled well by
the VM-pager.

I activated ZFS for a selected few filesystems where I need
journalling/CoW. The intention was not to increase performance, but 
to increase crash-safety, for a database only.

I noticed that the actual size of the ARCache did shrink down to 
~1 MB and only metadata would be cached. This makes write-performance
incredibly bad, because for every, say, 1kB write a 128kB read has
to happen first.

As a remedy I increased memory to 768 MB, but the only effect was
the ARCache now shrinking to 2 MB instead of 1 MB.

Reading the source showed me that ZFS will try to shrink the ARCache
as soon as Free+Cache mem gets down to someway near the uppermost 
threshold (vm.v_free_target + vm.v_cache_min).

So, some modification seems necessary here, as it appears
inacceptable that one has to increase the installed memory by maybe
4 or 5 times only to enable ZFS on a machine that otherwise would 
function suitably. 
I do currently not know how the behaviour is on big machines
with a couple GB ram - but I think there the free list will also
run low if enough processes compete for memory.

Fix: Since ZFS write performance becomes horribly bad when there is no 
caching at all available, I suggest that a certain mimimum of
caching should be preserved even if the free list is quite low.

Therefore I changed the code in the following way (see attached
patch), and the results are now ok for my purpose.

Experiments showed that there is a certain risk that the machine may 
experience a freeze/lockdown when working on these parameters.
A crashdump analysis then gave me the hint that this seems to
happen when the amount of arc_anon buffers gets too high and requests
for further buffers will be declined. The machine then seems to
block all activity and steadily increase 
kstat.zfs.misc.arcstats.memory_throttle_count until watchdog-reboot.
I have not yet experienced this effect with the now attached patch,
but further evaluation seems necessary.

How-To-Repeat: 
Start some processes that use up the available memory. Best choice
might be ruby processes that do GarbageCollection and therefore
will be considered active and not candidate for swapping (otherwise
"sysctl vm.swap_idle_enabled=1" would be another solution).

Then check the difference of
kstat.zfs.misc.arcstats.size - vfs.zfs.arc_meta_used
(that should be the amount of payload data currently being cached)
going near zero.

Comment 1 Mark Linimon freebsd_committer

2009-09-15 01:46:39 UTC

Responsible Changed
From-To: freebsd-bugs->freebsd-fs

Over to maintainer(s).

Comment 2 dfilter service freebsd_committer

2010-09-17 08:14:16 UTC

Author: avg
Date: Fri Sep 17 07:14:07 2010
New Revision: 212780
URL: http://svn.freebsd.org/changeset/base/212780

Log:
  zfs arc_reclaim_needed: more reasonable threshold for available pages
  
  vm_paging_target() is not a trigger of any kind for pageademon, but
  rather a "soft" target for it when it's already triggered.
  Thus, trying to keep 2048 pages above that level at the expense of ARC
  was simply driving ARC size into the ground even with normal memory
  loads.
  Instead, use a threshold at which a pagedaemon scan is triggered, so
  that ARC reclaiming helps with pagedaemon's task, but the latter still
  recycles active and inactive pages.
  
  PR:		kern/146410, kern/138790
  MFC after:	3 weeks

Modified:
  head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c

Modified: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
==============================================================================
--- head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c	Fri Sep 17 04:55:01 2010	(r212779)
+++ head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c	Fri Sep 17 07:14:07 2010	(r212780)
@@ -2161,10 +2161,10 @@ arc_reclaim_needed(void)
 		return (0);
 
 	/*
-	 * If pages are needed or we're within 2048 pages
-	 * of needing to page need to reclaim
+	 * Cooperate with pagedaemon when it's time for it to scan
+	 * and reclaim some pages.
 	 */
-	if (vm_pages_needed || (vm_paging_target() > -2048))
+	if (vm_paging_need())
 		return (1);
 
 #if 0
_______________________________________________
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"

Comment 3 dfilter service freebsd_committer

2010-09-17 08:34:57 UTC

Author: avg
Date: Fri Sep 17 07:34:50 2010
New Revision: 212783
URL: http://svn.freebsd.org/changeset/base/212783

Log:
  zfs arc_reclaim_needed: fix typo in mismerge in r212780
  
  PR:		kern/146410, kern/138790
  MFC after:	3 weeks
  X-MFC with:	r212780

Modified:
  head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c

Modified: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
==============================================================================
--- head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c	Fri Sep 17 07:20:20 2010	(r212782)
+++ head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c	Fri Sep 17 07:34:50 2010	(r212783)
@@ -2160,7 +2160,7 @@ arc_reclaim_needed(void)
 	 * Cooperate with pagedaemon when it's time for it to scan
 	 * and reclaim some pages.
 	 */
-	if (vm_paging_need())
+	if (vm_paging_needed())
 		return (1);
 
 #if 0
_______________________________________________
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"

Comment 4 Eitan Adler freebsd_committer

2011-03-01 15:16:05 UTC

State Changed
From-To: open->patched

committed in head (r212783)

Comment 5 Eitan Adler freebsd_committer

2011-03-01 15:23:16 UTC

Responsible Changed
From-To: freebsd-fs->avg

same as above

Comment 6 Andriy Gapon freebsd_committer

2011-04-02 09:23:17 UTC

State Changed
From-To: patched->closed

I think that this has been actually resolved for all branches 
where active ZFS development/maintenance takes place.