Bug 87792 - [patch] very bad performance of cp(1) via NFS, possibly mmap() problem
Summary: [patch] very bad performance of cp(1) via NFS, possibly mmap() problem
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 5.4-STABLE
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2005-10-21 17:40 UTC by Andre.Albsmeier
Modified: 2018-01-03 05:16 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andre.Albsmeier 2005-10-21 17:40:15 UTC
cp uses mmap() to copy files < 8MB. When files are copied
from the local host to a remote box via NFS data is transferred
very slowly. For bigger files, cp uses read() and write()
alternately which performs very well.

Fix: 

I have no idea. FreeBSD-4 uses the same mmap() thingy in utils.c
and here performance is ok so I assume the problem is somewhere
else than in the cp sources.
How-To-Repeat: 
I have two files, one is 6MB in size and the other one is 12MB.
I copy them from the local box to an NFS mounted directory named
/scratch:

andre@bali:/tmp>time cp 12mb /scratch
cp 12mb /scratch  0.00s user 0.12s system 10% cpu 1.142 total

andre@bali:/tmp>time cp 6mb /scratch
cp 6mb /scratch  0.00s user 0.13s system 0% cpu 14.274 total

The 12MB filecopy performs quite well as I would expect it
from a 100MBit network. The 6mb filecopy I won't comment on :-).
(This is an extreme case, other client-server combinations
are faster. The "fastest" one I found was 4 seconds which is
still bad for a 6MB file).

In the mmap() case, the whole source file is mmap'ed and
one write() call is used which tries to write the whole file
all at once. I have modified copy_file() in utils.c in a way
that instead of one "big" write(), several small ones are
being made (this is no fix, just for demonstration):

--- utils.c.ORI	Thu Aug 19 07:04:27 2004
+++ utils.c	Fri Oct 21 18:19:59 2005
@@ -140,7 +140,11 @@
 			wtotal = 0;
 			for (bufp = p, wresid = fs->st_size; ;
 			    bufp += wcount, wresid -= (size_t)wcount) {
+#if 0
 				wcount = write(to_fd, bufp, wresid);
+#else
+				wcount = write(to_fd, bufp, wresid > 65536 ? 65536 : wresid );
+#endif
 				wtotal += wcount;
 				if (info) {
 					info = 0;


65536 is just a value for playing; almost everything below
1M and not ridiculous small will do it:

andre@bali:/tmp>time cp 6mb /scratch 
cp 6mb /scratch  0.00s user 0.05s system 8% cpu 0.618 total
Comment 1 tedm 2012-10-29 06:53:01 UTC
Try the following:

In bin/cp/utils.c (source) there is a check, if the file is less than
8MB or so, it uses mmap, if the file is larger, it will use write()

Modify the source and recompiled to -never- use mmap, only to use
write()

Change line 143:
original:
fs->st_size <= 8 * 1048576) {

New:
fs->st_size <= 8 * 8) {

It will use mmap still if the file is larger than 64bytes (if it uses
bytes there, pretty sure it does).

This is from a response to the freebsd-stable mailing list back in
2009 by Brent Jones he reported a 100 fold increase.
Comment 2 Andre.Albsmeier 2012-10-29 08:01:37 UTC
I even went a step further an simply Undef'ed
VM_AND_BUFFER_CACHE_SYNCHRONIZED in cp's Makefile
so the entire mmap code gets disabled...
Comment 3 Eitan Adler freebsd_committer freebsd_triage 2017-12-31 07:59:43 UTC
For bugs matching the following criteria:

Status: In Progress Changed: (is less than) 2014-06-01

Reset to default assignee and clear in-progress tags.

Mail being skipped