Bug 197321

Environment:  FreeBSD 10.1 amd64 VM instanced on VWMware ESX.

NIC driver: vmx


VM instances boot as diskless FreeBSD clients.  When attempting to forward all syslog data from the clients to a central syslog server on the same network/subnet we randomly see messages NOT make it to the central syslog server.  Whats strange is the messages (best we can tell) appear to be almost the same type of message content.

/etc/syslog.conf:

*.*                                             @10.0.0.110
*.*                                             /var/spool/exim.log


Running syslogd in debug:  syslogd_flags="-d -ns -v -v"


Client machine runs Exim which periodically runs mail queue every 10 minutes.  This will generate two syslog messages as seen in the debug:


logmsg: pri 26, flags 0, from mxs1, msg Feb  4 16:19:35 exim[98580]: Start queue run: pid=98580
Logging to FORW 10.0.0.110
lsent/l: 59/59
Logging to FILE /var/spool/exim.log
logmsg: pri 26, flags 0, from mxs1, msg Feb  4 16:19:35 exim[98580]: End queue run: pid=98580
Logging to FORW 10.0.0.110
lsent/l: 57/57
Logging to FILE /var/spool/exim.log


and they do appear in the local logfile on the client server (which was setup just for this test)

Feb  4 16:19:35 <mail.info> mxs1 exim[98580]: Start queue run: pid=98580
Feb  4 16:19:35 <mail.info> mxs1 exim[98580]: End queue run: pid=98580


However, we randomly miss the "Start queue ..." message on the central server.  This is completely random, sometimes we get it, sometimes we do not. We always seem to receive the "End queue ..." message. 

After doing a TCP dump with the session above, the TCP dump only shows ONE UDP packet going out, which contained the "End queue..." message, but not the "Start queue..." message.


16:19:24.665116 IP 10.0.0.31.514 > 10.0.0.110.514: SYSLOG mail.info, length: 97
16:19:24.705419 IP 10.0.0.31.514 > 10.0.0.110.514: SYSLOG mail.info, length: 61
16:19:35.342946 IP 10.0.0.31.514 > 10.0.0.110.514: SYSLOG mail.info, length: 57 <==
16:20:24.825348 IP 10.0.0.31.514 > 10.0.0.110.514: SYSLOG mail.info, length: 97


So the message is never generating a packet even though syslogd debug says it it is  "logging to FORW 10.0.0.110"


This happens about ~40-50% of the time after looking through logs over several hours.