memcached has very big response time when reading stored objects size approximately
from 1400 to 2800 - when reply from server sent in two ip packets.

This caused by badly interaction between TCP delayed acknowledgements
(enabled in FreeBSD by default) and Nagle's algorithm.

Nagle's explanation of why interaction between delayed ACK and Nagle's
algorithm is bad:
http://developers.slashdot.org/comments.pl?sid=174457&threshold=1&commentsort=0&mode=thread&cid=14515105

So solution to improve latency is disable Nagle's algorithm for
memcached via socket option TCP_NODELAY.

This don't done by developers of memcached because on Linux TCP_CORK
(analog of TCP_NOPUSH) not compatible with TCP_NODELAY. And on
TCP_CORK show on Linux better performance than TCP_NODLEAY

From memcached Changelog:

2003-08-12 (Brad Fitzpatrick)
        * use TCP_CORK on Linux or TCP_PUSH on BSD
        * only use TCP_NODELAY when we don't have alternatives
        
2003-08-10
        * disable Nagel's Algorithm (TCP_NODELAY) for better performance (avva)

I run tests and it show, that TCP_NODELAY significantly decrease reading response time:
http://citrin.ru/stuff/memcached/read.png
and practically don't affect writing response time:
http://citrin.ru/stuff/memcached/write.png
Script used for this test:
http://citrin.ru/stuff/memcached/bench.pl
        
In result of using TCP_NODLEAY will be some network overhead, but in
environment where usually used memcached (100Mb or 1Gb LAN) it
negligible.

Fix: Apply this patch:



/*
      * the memset call clears nonstandard fields in some impementations--7rfBa7h8kM0KCcJMTPRyYJJE3buTpXegUTFV7SsIZpoSNQfI
Content-Type: text/plain; name="file.diff"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="file.diff"

--- memcached.c.orig    Thu Mar 23 18:51:47 2006
+++ memcached.c Thu Mar 23 18:57:56 2006
@@ -1146,9 +1146,7 @@
     setsockopt(sfd, SOL_SOCKET, SO_REUSEADDR, &flags, sizeof(flags));
     setsockopt(sfd, SOL_SOCKET, SO_KEEPALIVE, &flags, sizeof(flags));
     setsockopt(sfd, SOL_SOCKET, SO_LINGER, &ling, sizeof(ling));
-#if !defined(TCP_NOPUSH)
     setsockopt(sfd, IPPROTO_TCP, TCP_NODELAY, &flags, sizeof(flags));
-#endif
How-To-Repeat: Measure response time when reading from memcached objets size 1000, 2000 and 3000 bytes
(or in more wide range of sizes), e. g. using this script:
http://citrin.ru/stuff/memcached/test.pl