A problem with the current NFS server is that it does not cluster
writes, this in turn leads to really poor sequential-write
performance. 

By enabling write clustering NFS write performance goes from
26.6Mbyte/s to 54.3Mbyte/s or increases by a factor of 2. This is on a
SATA disk with write caching enabled (hw.ata.wc=1).

If write caching is disabled performance still goes up from 1.6Mbyte/s
to 5.8Mbyte/s (or by a factor of 3.6).

The attached patch (relative to current) makes the following changes:

1/ Rearrange the code so that the same code can be used to detect both
   sequential read and write access.

2/ Merge in updates from vfs_vnops.c::sequential_heuristic.

3/ Use double hashing in order to avoid hash-clustering in the nfsheur
   table. This change also makes it possible to reduce "try" from 32
   to 8.

4/ Pack the nfsheur table more efficiently.

5/ Tolerate reordered RPCs to some small amount (initially suggested
   by Ellard and Seltzer).

6/ Back-off from sequential access rather than immediately switching to
   random access (Ellard and Seltzer).

7/ To avoid starvation of the buffer pool call bwillwrite. The call is
   issued after the VOP_WRITE in order to avoid additional reordering
   of write operations.

8/ sysctl variables vfs.nfsrv.cluster_writes and cluster_reads to
   enable or disable clustering. vfs.nfsrv.reordered_io counts the
   number of reordered RPCs.

9/ In nfsrv_commit check for write errors and report them back to the
   client. Also check if the RPC argument count is zero which means
   that we must flush to the end of file according to the RFC.

10/ Two earlier commits broke the write gathering support:

    nfs_syscalls.c:1.71

      This change removed NQNFS stuff but left the NQNFS variable
      notstarted. This resulted in NFS write gathering effectively
      being permanently disabled (regardless if NFSv2 or NFSv3).

    nfs_syscalls.c:1.103

      This change disabled write gathering (again) for NFSv3 although
      this should be controlled by vfs.nfs.nfsrvw_procrastinate_v3 !=
      0.

Write gathering may still be useful with NFSv3 to put reordered write
RPCs into order, perhaps also for other reasons. This is now possible
again.

The attached patch is for current but you will observe similar
improvements with earlier FreeBSD versions.

Fix: Apply this patch
How-To-Repeat:         Apply patch, on server toggle vfs.nfsrv.cluster_writes=0,1.
        On client run (for example)

        dd if=/dev/zero of=/xxx/x.rm bs=32k count=10000