Bug 141305

Summary: [zfs] FreeBSD ZFS+sendfile severe performance issues (no cache)
Product: Base System Reporter: Michael Loftis <mloftis>
Component: kernAssignee: freebsd-fs (Nobody) <fs>
Status: Closed FIXED    
Severity: Affects Only Me CC: me
Priority: Normal    
Version: Unspecified   
Hardware: Any   
OS: Any   

Description Michael Loftis 2009-12-08 22:50:01 UTC
sendfile from ZFS filesystems does not appear to use the read cache, and may not be read combining either.  Serving files streaming media using default lighttpd setup on FreeBSD+ZFS we get 50+mbyte/sec disk I/O and *barely* 300-400mbit output.  Same hardware, same streaming media load, it saturates a gig-e connection with ~2mbyte/sec disk I/O.  This should be easy to reproduce with any HTTP benchmark suite and anything serving with sendfile on ZFS as lighttpd does.
Comment 1 kevin 2009-12-08 23:07:00 UTC
I can confirm similar results.

If the filesystem is ZFS, calling sendfile(2) over and over again on  
the same file produces constant disk read activity correlating exactly  
with the speed in which TCP is sending. Switching to writev(2) or  
using UFS instead results in expected levels of disk activity.
Comment 2 Mark Linimon freebsd_committer freebsd_triage 2009-12-09 14:12:12 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-fs

Over to maintainer(s).
Comment 3 dfilter service freebsd_committer freebsd_triage 2010-09-15 11:31:35 UTC
Author: avg
Date: Wed Sep 15 10:31:27 2010
New Revision: 212650
URL: http://svn.freebsd.org/changeset/base/212650

Log:
  tmpfs, zfs + sendfile: mark page bits as valid after populating it with data
  
  Otherwise, adding insult to injury, in addition to double-caching of data
  we would always copy the data into a vnode's vm object page from backend.
  This is specific to sendfile case only (VOP_READ with UIO_NOCOPY).
  
  PR:		kern/141305
  Reported by:	Wiktor Niesiobedzki <bsd@vink.pl>
  Reviewed by:	alc
  Tested by:	tools/regression/sockets/sendfile
  MFC after:	2 weeks

Modified:
  head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
  head/sys/fs/tmpfs/tmpfs_vnops.c

Modified: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
==============================================================================
--- head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c	Wed Sep 15 10:18:18 2010	(r212649)
+++ head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c	Wed Sep 15 10:31:27 2010	(r212650)
@@ -498,6 +498,8 @@ again:
 				sched_unpin();
 			}
 			VM_OBJECT_LOCK(obj);
+			if (error == 0)
+				vm_page_set_valid(m, off, bytes);
 			vm_page_wakeup(m);
 			if (error == 0)
 				uio->uio_resid -= bytes;

Modified: head/sys/fs/tmpfs/tmpfs_vnops.c
==============================================================================
--- head/sys/fs/tmpfs/tmpfs_vnops.c	Wed Sep 15 10:18:18 2010	(r212649)
+++ head/sys/fs/tmpfs/tmpfs_vnops.c	Wed Sep 15 10:31:27 2010	(r212650)
@@ -562,6 +562,8 @@ lookupvpg:
 		sf_buf_free(sf);
 		sched_unpin();
 		VM_OBJECT_LOCK(vobj);
+		if (error == 0)
+			vm_page_set_valid(m, offset, tlen);
 		vm_page_wakeup(m);
 		VM_OBJECT_UNLOCK(vobj);
 		return	(error);
_______________________________________________
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
Comment 4 Andriy Gapon freebsd_committer freebsd_triage 2013-07-08 18:04:10 UTC
State Changed
From-To: open->closed

This was fixed long ago.
Comment 5 me 2014-11-22 14:21:25 UTC
Is this really fixed? I'm using 10.1 and I had enabled Sendfile in Apache 2.4. System load was 10-13 (on a quad core with HT). The moment I disabled sendfile, system load went down to < 1.