Created attachment 271998 [details] prometheus_sysctl_exporter assert: fix for non-standard label (by Claude LLM, Maintainers review required) % uname -rUm 16.0-CURRENT amd64 1600019 % prometheus_sysctl_exporter // skipped // sysctl_kern_random_initial_seeding_arc4random_bypassed_before_seeding 0 sysctl_kern_random_initial_seeding_read_random_bypassed_before_seeding 0 sysctl_kern_random_initial_seeding_bypass_before_seeding 1 Assertion failed: (label[strspn(label, "abcdefghijklmnopqrstuvwxyz" "ABCDEFGHIJKLMNOPQRSTUVWXYZ" "0123456789_")] == '\0'), function oid_get_metric, file /usr/jails/src/src_16/src/usr.sbin/prometheus_sysctl_exporter/prometheus_sysctl_exporter.c, line 415. Abort (prometheus_sysctl_e), jid 0, uid 1001: exited on signal 6 The issue was found and a patch was proposed by the Claude LLM model (Maintainers review required!): ``` On FreeBSD 16-CURRENT, `prometheus_sysctl_exporter` exits with SIGABRT after printing the kern.random.initial_seeding.* sysctls. The crash is not caused by those nodes themselves; iteration continues to the next OID, kern.evdev.input.<N>.*, which uses the aggregation label "device index" (contains a space). That label is set in sys/dev/evdev/evdev.c in evdev_sysctl_create(): ``` ev_sysctl_tree = SYSCTL_ADD_NODE_WITH_LABEL(..., ev_unit_str, ..., "device index"); ``` The exporter previously assumed all OID labels are valid Prometheus label names and hit an assert() in oid_get_metric(). This worked on older releases (e.g. 15.1) because those evdev sysctls did not exist yet. There are currently 112 affected OIDs under kern.evdev.input.*. ``` ``` diff --git a/usr.sbin/prometheus_sysctl_exporter/prometheus_sysctl_exporter.c b/usr.sbin/prometheus_sysctl_exporter/prometheus_sysctl_exporter.c index e3182467be..691f7a9381 100644 --- a/usr.sbin/prometheus_sysctl_exporter/prometheus_sysctl_exporter.c +++ b/usr.sbin/prometheus_sysctl_exporter/prometheus_sysctl_exporter.c @@ -371,6 +371,20 @@ oid_get_name(const struct oid *o, struct oidname *on) on->oid = *o; } +/* Appends a Prometheus identifier, mapping unsupported characters. */ +static void +prometheus_append_ident(char *metric, size_t mlen, const char *ident) +{ + char buf[2]; + + while (*ident != '\0') { + snprintf(buf, sizeof(buf), "%c", + isalnum((unsigned char)*ident) ? *ident : '_'); + strlcat(metric, buf, mlen); + ++ident; + } +} + /* Populates the name and labels of an OID to a buffer. */ static void oid_get_metric(const struct oidname *on, const struct oidformat *of, @@ -387,13 +401,7 @@ oid_get_metric(const struct oidname *on, const struct oidformat *of, for (i = 0; i < on->oid.len; ++i) { if (*label == '\0') { strlcat(metric, "_", mlen); - while (*name != '\0') { - /* Map unsupported characters to underscores. */ - snprintf(buf, sizeof(buf), "%c", - isalnum(*name) ? *name : '_'); - strlcat(metric, buf, mlen); - ++name; - } + prometheus_append_ident(metric, mlen, name); } name += strlen(name) + 1; label += strlen(label) + 1; @@ -409,12 +417,10 @@ oid_get_metric(const struct oidname *on, const struct oidformat *of, separator = '{'; for (i = 0; i < on->oid.len; ++i) { if (*label != '\0') { - assert(label[strspn(label, - "abcdefghijklmnopqrstuvwxyz" - "ABCDEFGHIJKLMNOPQRSTUVWXYZ" - "0123456789_")] == '\0'); - snprintf(buf, sizeof(buf), "%c%s=\"", separator, label); + snprintf(buf, sizeof(buf), "%c", separator); strlcat(metric, buf, mlen); + prometheus_append_ident(metric, mlen, label); + strlcat(metric, "=\"", mlen); while (*name != '\0') { /* Escape backslashes and double quotes. */ if (*name == '\\' || *name == '"') ```
(In reply to olevole from comment #0) We seem to use "device_index" elsewhere, and I think it probably makes sense to be consistent here.
(In reply to Kyle Evans from comment #1) I agree. I think we should just change "device index" to "device_index".
(In reply to Alan Somers from comment #2) I pinged the author out-of-band to confirm they didn't use it in other parts of the work (since this was a series across FreeBSD + wulf's repo + another downstream library), but I think they did not.
(In reply to Kyle Evans from comment #3) I misread the context here, let's just fix this
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=cb8bda40695f5d402f334f48795b1ab27b72dce5 commit cb8bda40695f5d402f334f48795b1ab27b72dce5 Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2026-06-30 16:51:57 +0000 Commit: Kyle Evans <kevans@FreeBSD.org> CommitDate: 2026-06-30 16:51:57 +0000 evdev: use a prometheus-safe label for ev_sysctl_tree Prometheus doesn't allow spaces, let's normalize this to what we use elsewhere for consistency. The sysctl exporter could probably do this itself, but let's decouple that from the immediate problem: matching the label between the exported data and in-tree is nice for greppability. PR: 296179 Reviewed by: asomers, wulf Differential Revision: https://reviews.freebsd.org/D57966 sys/dev/evdev/evdev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
I looked briefly at prometheus_sysctl_exporter, and I'm not 100% sure what to do there. My gut instinct is that we probably *do* want labels to be as-written in the kernel rather than transformed into something safe, so we probably want to do a validation pass before we print and zap & warn on invalid labels. Does that mean we drop the entry entirely, or just output it without the label? The latter seems like it could cause ambiguity issue, so probably we just drop the entry entirely and warn so that one can debug it.
("as-written in the kernel" meaning the kernel just needs to use safe names, to be clear)
I've opened https://reviews.freebsd.org/D57983 with the intention of making these easier to debug and continue writing out all of the other well-formed metrics.
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e55aa7b67cb0f7a29a0117231b8764524d1fb971 commit e55aa7b67cb0f7a29a0117231b8764524d1fb971 Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2026-06-30 16:51:57 +0000 Commit: Kyle Evans <kevans@FreeBSD.org> CommitDate: 2026-08-06 22:53:01 +0000 evdev: use a prometheus-safe label for ev_sysctl_tree Prometheus doesn't allow spaces, let's normalize this to what we use elsewhere for consistency. The sysctl exporter could probably do this itself, but let's decouple that from the immediate problem: matching the label between the exported data and in-tree is nice for greppability. PR: 296179 Reviewed by: asomers, wulf (cherry picked from commit cb8bda40695f5d402f334f48795b1ab27b72dce5) sys/dev/evdev/evdev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=240bd2eeb21f959f7340402211c0bf43a58d31b5 commit 240bd2eeb21f959f7340402211c0bf43a58d31b5 Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2026-06-30 16:51:57 +0000 Commit: Kyle Evans <kevans@FreeBSD.org> CommitDate: 2026-08-06 23:36:07 +0000 evdev: use a prometheus-safe label for ev_sysctl_tree Prometheus doesn't allow spaces, let's normalize this to what we use elsewhere for consistency. The sysctl exporter could probably do this itself, but let's decouple that from the immediate problem: matching the label between the exported data and in-tree is nice for greppability. PR: 296179 Reviewed by: asomers, wulf (cherry picked from commit cb8bda40695f5d402f334f48795b1ab27b72dce5) sys/dev/evdev/evdev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)