Created attachment 183758 [details] Resolving patch When sending messages to a remote host syslogd omits the hostname field required by RFC 3164. This affects anyone sending logs from a FreeBSD host to a central logging server that expects RFC 3164-conformant messages (Logstash, fluentd) - it breaks the remote server's ability to parse the FreeBSD system's log messages. This issue is present on 10.3-RELEASE and 11.0-RELEASE. The attached patch corrects the behavior of syslogd when sending messages. Following the logic laid out in the existing code I preserved the previous hostname for forwarded messages, but made the resulting message conform with the RFC. This fix has been verified against Logstash & fluentd. I have not deliberately thrown any pathological input at it so there should be some scrutiny. NOTE: Related standards bug 200933 deals with receiving RFC 3164-conformant messages. There is a patch there which should be reviewed and either applied or adapted as appropriate.
CC'ing ed@, who has done quite a bit with syslogd over the past year.
Hi there, Thanks for the patch! The syslogd code has been refactored a lot lately to support the RFC 5424 message format. Looking at the code, the issue still applies. That said, I'm a bit hesitant to change anything for the RFC 3164 format support. People expect that it works in a certain way and changing that seems like a bad idea. Could you please give the RFC 5424 support that's present in 11-STABLE and HEAD a try? Just add "-O rfc5424" to syslogd_flags in rc.conf. Please let me know whether that addresses the issue for you sufficiently.
I have not forgotten about this PR :) We're currently in a QA cycle for 11.2 (which I don't believe includes the RFC5424 support?), but once that's done I'll spin up a test machine on -STABLE and test it against logstash. Our plan is to switch to RFC 5424 style messages anyway when we move to 11.3 or 12.0 & that eliminates the need for this fix on our end & I'm fine hotfixing our local syslogd until then. I do see the logic in keeping RFC 3164 support as-is (since most folks will probably move to the new format) or making it a 12.0 item (since it introduces incompatibility with old syslogd). If it stays as-is that probably merits a note/erratum in the manpage for the next person who stumbles on this when they set up centralized logging in a heterogeneous environment.
Any updates here?
Created attachment 233882 [details] Patch against 14-CURRENT Attached is a patch for latest version of the code. I did not modify the "Forwarded from" case.
Created attachment 247881 [details] patch with added rfc3164-strict format, against 15-CURRENT Hi, I had a look at RFC3164 and found a couple of interesting items: - Section 4.2 of RFC3164 states there are no set requirements for the contents of the syslog packet. So the existing format is technically valid. - The format described in RFC3164 (i.e. "<PRI>TIMESTAMP HOSTNAME MSG") is only RECOMMENDED to be used. It's not enforced. - The "Forwarded from HOSTNAME:" field is not mentioned anywhere in RFC3164. I found it originally came about in FreeBSD PR 7055, back in 1998. - Section 4.3.1 of RFC3164 states syslog relay servers that forward messages MUST do so without making any changes to the packet. Given this, I think the existing bsd/rfc3164 format should be kept as-is for backward compatibility, but a new optional format should be added that strictly conforms to the recommended format defined in RFC3164. I have attached a new patch based on Bryan Drewery's patch. It adds the third optional format to the -O flag, "strict" or "rfc3164-strict", which adds the hostname and excludes "Forwarded from" field in the network syslog message, and also updates the man page.
^Triage: clear unneeded flags. Nothing has yet been committed to be merged.
Hi, I took my last patch, added a bunch of tests and created a pull request on github with the changes. https://github.com/freebsd/freebsd-src/pull/1536
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=8cc248fb92bfae1e6ef4e2b48a0064377c4b3c11 commit 8cc248fb92bfae1e6ef4e2b48a0064377c4b3c11 Author: Michal Scigocki <michal.os@hotmail.com> AuthorDate: 2024-12-23 04:36:57 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-12-23 15:46:14 +0000 syslogd: Add rfc3164-strict format option The syslogd bsd/rfc3164 message format does not strictly conform to the format recommended in RFC 3164 when syslog messages are forwarded from remote hosts. The new rfc3164-strict format generates messages that conform with the RFC 3164 recommended format. The existing format is maintained for backwards compatibility. PR: 220246 Reviewed by: markj MFC after: 3 weeks Pull Request: https://github.com/freebsd/freebsd-src/pull/1536 usr.sbin/syslogd/syslogd.8 | 11 +++++-- usr.sbin/syslogd/syslogd.c | 34 ++++++++++++++++------ .../syslogd/tests/syslogd_basic_format_test.sh | 20 +++++++++++++ .../syslogd/tests/syslogd_forwarded_format_test.sh | 26 +++++++++++++++++ .../syslogd/tests/syslogd_relayed_format_test.sh | 27 +++++++++++++++++ 5 files changed, 107 insertions(+), 11 deletions(-)
Shouldn't this issue be closed?
^Triage: assign to committer. Set flags for possible MFCs (just close if not).
FYI, I found one issue trying to squash/cherry-pick my Github pull request patch commits into 14-STABLE. The testcase "jail_noinet" fails to run. To make it work, I had to manually make this change in the testcase: - syslogd_start -j syslogd_noinet -ss + syslogd_start -j syslogd_noinet -s -s It's the same in CURRENT, it was introduced a few commits before mine: https://reviews.freebsd.org/D47649
I also checked 13-STABLE. Same change required for the jail_noinet testcase ("-ss" to "-s -s"). Also in .../syslogd/tests/Makefile, the two execenv metadata setting need to be removed (no execenv feature in 13) and swapped back to is_exclusive="true" to prevent test failures when kyua.conf parallelism is > 1. TEST_METADATA= is_exclusive="true" \ timeout=20