Bug 32141

Mounting a NFS volume with NFS V3/TCP from an FreeBSD 4.3+ server
gives low read transferrates (80 kbyte/s) when mounting with
FreeBSD 4.2/4.3 clients.

The server:

(Same effect with 4.3 Release / 4.4 Release / 4.4 stable)

A FreeBSD 4.3+ machine (x86) exports a filesystem.
The installation is out of the box with no tuning done. One
ethernet card is installed and works properly: tcpblast shows
the throughput one expects for an unloaded network (1 Mbyte/s or
11 Mbyte/s for 10/100 Mbit/s network in both directions).
The ethernet config (10/100 hdx/fdx) is done manually. Connection
to the clients is a) via 10Mbit Ethernet hub or b) via switched VLAN
100Mbit/fdx set manually. Neither the server nor the clients have CPU 
or network load.
At client-side: "wire-test server" shows that the server talks 
v2/v3 and udp/tcp in all 4 variants.


(A) FreeBSD 4.2/4.3 Release Client, mount without options:

Mounting the exported filesystem with

    "mount server:/export/scratch /mnt"

and copying a large file (e.g. 3MB generic kernel) from remote
to local filesystem shows the performace for reading 
(cp /mnt/file /usr/tmp) and writing is as expected: 
about 70% .. 90% of tcpdumps bandwitdh; writing slower
than reading; copying files that were not just copyied before
because of cache effects; no tcpdump running. 
Using ftp instead of cp to get the file gives similar results.


(B) FreeBSD 4.2/4.3 Release Client, mount with options:

Mounting the exported filesystem with

    "mount_nfs -3 -T server:/export/scratch /mnt"

forces NFS v3 over TCP. Now copying a large file shows that
performace for writing is just as before. But performance for
reading breaks down to 80 kbyte/s, regardless if we use 10Mbit
or 100Mbit network. Switching down to nfs v2 tcp brings no change. 
Using ftp instead of cp to get the file gives max. transferrate.
BUT changing the server OS to 4.2 Release gives us the performance 
expected, too.
 
(C) Solaris 2.6 client, mount without options:

mounting the filesystem without options gives performance as
expected (r/w). 

(D) Solaris 2.6 client, mount with options:

Forcing NFS v3 TCP gives no change: good performance. Setting
rsize also gives:
32768 = good performance
smaller = goes downto 20%

Things tried out / checked / not appropriate here:
- odd network routing config: no, our's fine, other protocols
  work 
- wrong fdx/hdx settings: ok here: everything set manually and tested
  with tcpdump and ftp'ing files around
- high cpu / network load, lot of clients: no clients, no load here
- /kernel reports some buffer trouble in /var/log/messages:
  no messages here.
- multiplatform with odd NFS implemenentations: no, FreeBSD only
- black box network equipment: nope, only a switch or stupid
  10 mbit HUB between client/server
- unreliable network (wlan) or dialin network: no, ethernet only
- old HW: no, P500+ with PCI Ethernetcards 3com 3c905ctx
- other HW conflicts: no, VGA, Ethernet, Onboard IDE, ser+par only
- broken ipfw settings: no ipfw here
- rc.conf: tcp_extensions YES/NO plus server reboot: no change
- Changing the Server from 4.4 Release to 4.4 Stable: no change
- make all traffic go via stupid 10mbit hub to avoid vlan/switch: 
  no change
- mismatch because of incomplete cvsup: no, Release setup from CDROM

Please feel free to contact me for detailed debug output, session logs,
tcpdumps, etc...

Fix: 

When mounting the remote FS via NFS v3 TCP also set the rsize 
within the mount command:

    'mount_nfs -3 -T -r 4096 server:/export/filesys /mnt'
How-To-Repeat: A FreeBSD 4.3 Release / 4.4 Release NFS server exports a filesystem, 
a 4.{2,3} Release client mounts the filesystem with 

    'mount_nfs -3 -T server:/export/filesys /mnt'

At the client: copying big files from the server to the client 
gives 80kbyte/s tranfserrate while writing to the server goes fast.

This problem also appears with amd using autonegotiation or forcing
amd to use v3/tcp (this way we discovered the 80kbyte/s reading).