Bug 220246

Summary: syslogd does not send RFC3164-conformant messages
Product: Base System Reporter: mikeg
Component: binAssignee: freebsd-bugs (Nobody) <bugs>
Status: Open ---    
Severity: Affects Many People CC: bdrewery, cem, ed, emaste, hrs, kevans, koobs, lwhsu, michal.os, mikeg, ngie, sobomax
Priority: --- Keywords: easy, needs-qa, standards
Version: 10.3-RELEASE   
Hardware: Any   
OS: Any   
URL: https://www.ietf.org/rfc/rfc3164.txt
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200933
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194231
Attachments:
Description Flags
Resolving patch
none
Patch against 14-CURRENT
none
patch with added rfc3164-strict format, against 15-CURRENT none

Description mikeg 2017-06-24 08:50:02 UTC
Created attachment 183758 [details]
Resolving patch

When sending messages to a remote host syslogd omits the hostname field required by RFC 3164. This affects anyone sending logs from a FreeBSD host to a central logging server that expects RFC 3164-conformant messages (Logstash, fluentd) - it breaks the remote server's ability to parse the FreeBSD system's log messages.

This issue is present on 10.3-RELEASE and 11.0-RELEASE.

The attached patch corrects the behavior of syslogd when sending messages. Following the logic laid out in the existing code I preserved the previous hostname for forwarded messages, but made the resulting message conform with the RFC. 

This fix has been verified against Logstash & fluentd. I have not deliberately thrown any pathological input at it so there should be some scrutiny.


NOTE: Related standards bug 200933 deals with receiving RFC 3164-conformant messages. There is a patch there which should be reviewed and either applied or adapted as appropriate.
Comment 1 Kyle Evans freebsd_committer freebsd_triage 2018-08-09 02:38:14 UTC
CC'ing ed@, who has done quite a bit with syslogd over the past year.
Comment 2 Ed Schouten freebsd_committer freebsd_triage 2018-08-09 12:57:16 UTC
Hi there,

Thanks for the patch! The syslogd code has been refactored a lot lately to support the RFC 5424 message format. Looking at the code, the issue still applies. That said, I'm a bit hesitant to change anything for the RFC 3164 format support. People expect that it works in a certain way and changing that seems like a bad idea.

Could you please give the RFC 5424 support that's present in 11-STABLE and HEAD a try? Just add "-O rfc5424" to syslogd_flags in rc.conf. Please let me know whether that addresses the issue for you sufficiently.
Comment 3 mikeg 2018-08-24 21:38:53 UTC
I have not forgotten about this PR :)

We're currently in a QA cycle for 11.2 (which I don't believe includes the RFC5424 support?), but once that's done I'll spin up a test machine on -STABLE and test it against logstash.

Our plan is to switch to RFC 5424 style messages anyway when we move to 11.3 or 12.0 & that eliminates the need for this fix on our end & I'm fine hotfixing our local syslogd until then.


I do see the logic in keeping RFC 3164 support as-is (since most folks will probably move to the new format) or making it a 12.0 item (since it introduces incompatibility with old syslogd).
If it stays as-is that probably merits a note/erratum in the manpage for the next person who stumbles on this when they set up centralized logging in a heterogeneous environment.
Comment 4 Li-Wen Hsu freebsd_committer freebsd_triage 2020-06-19 16:47:15 UTC
Any updates here?
Comment 5 Bryan Drewery freebsd_committer freebsd_triage 2022-05-12 19:20:59 UTC
Created attachment 233882 [details]
Patch against 14-CURRENT

Attached is a patch for latest version of the code. I did not modify the "Forwarded from" case.
Comment 6 Michal Scigocki 2024-01-23 10:29:16 UTC
Created attachment 247881 [details]
patch with added rfc3164-strict format, against 15-CURRENT

Hi,

I had a look at RFC3164 and found a couple of interesting items:

- Section 4.2 of RFC3164 states there are no set requirements for the contents of the syslog packet. So the existing format is technically valid.
- The format described in RFC3164 (i.e. "<PRI>TIMESTAMP HOSTNAME MSG") is only RECOMMENDED to be used. It's not enforced.
- The "Forwarded from HOSTNAME:" field is not mentioned anywhere in RFC3164. I found it originally came about in FreeBSD PR 7055, back in 1998.
- Section 4.3.1 of RFC3164 states syslog relay servers that forward messages MUST do so without making any changes to the packet.

Given this, I think the existing bsd/rfc3164 format should be kept as-is for backward compatibility, but a new optional format should be added that strictly conforms to the recommended format defined in RFC3164.

I have attached a new patch based on Bryan Drewery's patch. It adds the third optional format to the -O flag, "strict" or "rfc3164-strict", which adds the hostname and excludes "Forwarded from" field in the network syslog message, and also updates the man page.
Comment 7 Mark Linimon freebsd_committer freebsd_triage 2024-10-08 04:49:38 UTC
^Triage: clear unneeded flags.  Nothing has yet been committed to be merged.
Comment 8 Michal Scigocki 2024-11-20 01:25:04 UTC
Hi, I took my last patch, added a bunch of tests and created a pull request on github with the changes.

https://github.com/freebsd/freebsd-src/pull/1536