Bug 76736

Syslogd supports logging subprocesses that communicate with the daemon using 
pipelines, e.g., in /etc/syslog.conf

local0.*        | exec grep -v DUPLEX >> /root/log

If the arrival rate of syslog messages is high enough (above 100 per second), syslogd may 
try to write to the pipeline too fast. The symptom of this problem is a line like

Jan 22 16:48:57 maryann syslogd:  exec grep -v DUPLEX >> /root/log: Resource 
temporarily unavailable

in /var/log/messages. The cause of the problem is the use of non-blocking writes by 
syslogd, which will fail when the pipe is full. In this case, the file descriptor of the pipe is 
closed, and the syslog message quietly discarded. See

http://lists.freebsd.org/pipermail/freebsd-hackers/2005-January/009921.html

for further details.

Fix: The solution is to check whether the pipeline is full, and if so give the
logging subprocess a chance to empty the pipeline. This has to be implemented
in a way that still protects syslogd against hung logging subprocesses. The
following patch worked well in my environment. I also tried it against CURRENT
(syslogd version 1.140) and the patch worked fine.
How-To-Repeat: 
Configure a syslog pipeline in one of the local facilities (see above) and write a program 
that submits numbered syslog messages to the same facility. You should see resource 
temporarily unavailable in /var/log/messages, and gaps in the sequence of logged 
messages. In my environment, syslog messages come from a large number of hosts, so it 
may be that network buffering aggravates (or even creates) this problem.