Bug 141305 - [zfs] FreeBSD ZFS+sendfile severe performance issues (no cache)
Summary: [zfs] FreeBSD ZFS+sendfile severe performance issues (no cache)
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: Unspecified
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-fs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-12-08 22:50 UTC by Michael Loftis
Modified: 2014-11-22 14:21 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Loftis 2009-12-08 22:50:01 UTC
sendfile from ZFS filesystems does not appear to use the read cache, and may not be read combining either.  Serving files streaming media using default lighttpd setup on FreeBSD+ZFS we get 50+mbyte/sec disk I/O and *barely* 300-400mbit output.  Same hardware, same streaming media load, it saturates a gig-e connection with ~2mbyte/sec disk I/O.  This should be easy to reproduce with any HTTP benchmark suite and anything serving with sendfile on ZFS as lighttpd does.
Comment 1 kevin 2009-12-08 23:07:00 UTC
I can confirm similar results.

If the filesystem is ZFS, calling sendfile(2) over and over again on  
the same file produces constant disk read activity correlating exactly  
with the speed in which TCP is sending. Switching to writev(2) or  
using UFS instead results in expected levels of disk activity.
Comment 2 Mark Linimon freebsd_committer freebsd_triage 2009-12-09 14:12:12 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-fs

Over to maintainer(s).
Comment 3 dfilter service freebsd_committer freebsd_triage 2010-09-15 11:31:35 UTC
Author: avg
Date: Wed Sep 15 10:31:27 2010
New Revision: 212650
URL: http://svn.freebsd.org/changeset/base/212650

Log:
  tmpfs, zfs + sendfile: mark page bits as valid after populating it with data
  
  Otherwise, adding insult to injury, in addition to double-caching of data
  we would always copy the data into a vnode's vm object page from backend.
  This is specific to sendfile case only (VOP_READ with UIO_NOCOPY).
  
  PR:		kern/141305
  Reported by:	Wiktor Niesiobedzki <bsd@vink.pl>
  Reviewed by:	alc
  Tested by:	tools/regression/sockets/sendfile
  MFC after:	2 weeks

Modified:
  head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
  head/sys/fs/tmpfs/tmpfs_vnops.c

Modified: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
==============================================================================
--- head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c	Wed Sep 15 10:18:18 2010	(r212649)
+++ head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c	Wed Sep 15 10:31:27 2010	(r212650)
@@ -498,6 +498,8 @@ again:
 				sched_unpin();
 			}
 			VM_OBJECT_LOCK(obj);
+			if (error == 0)
+				vm_page_set_valid(m, off, bytes);
 			vm_page_wakeup(m);
 			if (error == 0)
 				uio->uio_resid -= bytes;

Modified: head/sys/fs/tmpfs/tmpfs_vnops.c
==============================================================================
--- head/sys/fs/tmpfs/tmpfs_vnops.c	Wed Sep 15 10:18:18 2010	(r212649)
+++ head/sys/fs/tmpfs/tmpfs_vnops.c	Wed Sep 15 10:31:27 2010	(r212650)
@@ -562,6 +562,8 @@ lookupvpg:
 		sf_buf_free(sf);
 		sched_unpin();
 		VM_OBJECT_LOCK(vobj);
+		if (error == 0)
+			vm_page_set_valid(m, offset, tlen);
 		vm_page_wakeup(m);
 		VM_OBJECT_UNLOCK(vobj);
 		return	(error);
_______________________________________________
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
Comment 4 Andriy Gapon freebsd_committer freebsd_triage 2013-07-08 18:04:10 UTC
State Changed
From-To: open->closed

This was fixed long ago.
Comment 5 me 2014-11-22 14:21:25 UTC
Is this really fixed? I'm using 10.1 and I had enabled Sendfile in Apache 2.4. System load was 10-13 (on a quad core with HT). The moment I disabled sendfile, system load went down to < 1.