Bug 296179 - prometheus_sysctl_exporter assert: exited on signal 6: non-standard label in evdev (device index)
Summary: prometheus_sysctl_exporter assert: exited on signal 6: non-standard label in ...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 16.0-CURRENT
Hardware: Any Any
: --- Affects Many People
Assignee: Kyle Evans
URL: https://reviews.freebsd.org/D57966
Keywords:
Depends on:
Blocks:
 
Reported: 2026-06-21 08:54 UTC by olevole
Modified: 2026-08-07 01:49 UTC (History)
2 users (show)

See Also:
kevans: mfc-stable15+
kevans: mfc-stable14+


Attachments
prometheus_sysctl_exporter assert: fix for non-standard label (by Claude LLM, Maintainers review required) (2.03 KB, patch)
2026-06-21 08:54 UTC, olevole
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description olevole 2026-06-21 08:54:34 UTC
Created attachment 271998 [details]
prometheus_sysctl_exporter assert: fix for non-standard label (by Claude LLM, Maintainers review required)

% uname -rUm
16.0-CURRENT amd64 1600019

% prometheus_sysctl_exporter
// skipped //
sysctl_kern_random_initial_seeding_arc4random_bypassed_before_seeding 0
sysctl_kern_random_initial_seeding_read_random_bypassed_before_seeding 0
sysctl_kern_random_initial_seeding_bypass_before_seeding 1
Assertion failed: (label[strspn(label, "abcdefghijklmnopqrstuvwxyz" "ABCDEFGHIJKLMNOPQRSTUVWXYZ" "0123456789_")] == '\0'), function oid_get_metric, file /usr/jails/src/src_16/src/usr.sbin/prometheus_sysctl_exporter/prometheus_sysctl_exporter.c, line 415.
Abort

(prometheus_sysctl_e), jid 0, uid 1001: exited on signal 6

The issue was found and a patch was proposed by the Claude LLM model (Maintainers review required!):
```
On FreeBSD 16-CURRENT, `prometheus_sysctl_exporter` exits with SIGABRT after printing the kern.random.initial_seeding.* sysctls. 

The crash is not caused by those nodes themselves; iteration continues to the next OID, kern.evdev.input.<N>.*, which uses the aggregation label "device index" (contains a space). 

That label is set in sys/dev/evdev/evdev.c in evdev_sysctl_create():
```
ev_sysctl_tree = SYSCTL_ADD_NODE_WITH_LABEL(..., ev_unit_str, ...,
    "device index");
```

The exporter previously assumed all OID labels are valid Prometheus label names and hit an assert() in oid_get_metric(). 
This worked on older releases (e.g. 15.1) because those evdev sysctls did not exist yet. 

There are currently 112 affected OIDs under kern.evdev.input.*.
```



```
diff --git a/usr.sbin/prometheus_sysctl_exporter/prometheus_sysctl_exporter.c b/usr.sbin/prometheus_sysctl_exporter/prometheus_sysctl_exporter.c
index e3182467be..691f7a9381 100644
--- a/usr.sbin/prometheus_sysctl_exporter/prometheus_sysctl_exporter.c
+++ b/usr.sbin/prometheus_sysctl_exporter/prometheus_sysctl_exporter.c
@@ -371,6 +371,20 @@ oid_get_name(const struct oid *o, struct oidname *on)
        on->oid = *o;
 }
 
+/* Appends a Prometheus identifier, mapping unsupported characters. */
+static void
+prometheus_append_ident(char *metric, size_t mlen, const char *ident)
+{
+       char buf[2];
+
+       while (*ident != '\0') {
+               snprintf(buf, sizeof(buf), "%c",
+                   isalnum((unsigned char)*ident) ? *ident : '_');
+               strlcat(metric, buf, mlen);
+               ++ident;
+       }
+}
+
 /* Populates the name and labels of an OID to a buffer. */
 static void
 oid_get_metric(const struct oidname *on, const struct oidformat *of,
@@ -387,13 +401,7 @@ oid_get_metric(const struct oidname *on, const struct oidformat *of,
        for (i = 0; i < on->oid.len; ++i) {
                if (*label == '\0') {
                        strlcat(metric, "_", mlen);
-                       while (*name != '\0') {
-                               /* Map unsupported characters to underscores. */
-                               snprintf(buf, sizeof(buf), "%c",
-                                   isalnum(*name) ? *name : '_');
-                               strlcat(metric, buf, mlen);
-                               ++name;
-                       }
+                       prometheus_append_ident(metric, mlen, name);
                }
                name += strlen(name) + 1;
                label += strlen(label) + 1;
@@ -409,12 +417,10 @@ oid_get_metric(const struct oidname *on, const struct oidformat *of,
        separator = '{';
        for (i = 0; i < on->oid.len; ++i) {
                if (*label != '\0') {
-                       assert(label[strspn(label,
-                           "abcdefghijklmnopqrstuvwxyz"
-                           "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
-                           "0123456789_")] == '\0');
-                       snprintf(buf, sizeof(buf), "%c%s=\"", separator, label);
+                       snprintf(buf, sizeof(buf), "%c", separator);
                        strlcat(metric, buf, mlen);
+                       prometheus_append_ident(metric, mlen, label);
+                       strlcat(metric, "=\"", mlen);
                        while (*name != '\0') {
                                /* Escape backslashes and double quotes. */
                                if (*name == '\\' || *name == '"')

```
Comment 1 Kyle Evans freebsd_committer freebsd_triage 2026-06-30 12:35:42 UTC
(In reply to olevole from comment #0)

We seem to use "device_index" elsewhere, and I think it probably makes sense to be consistent here.
Comment 2 Alan Somers freebsd_committer freebsd_triage 2026-06-30 13:51:22 UTC
(In reply to Kyle Evans from comment #1)
I agree.  I think we should just change "device index" to "device_index".
Comment 3 Kyle Evans freebsd_committer freebsd_triage 2026-06-30 14:02:47 UTC
(In reply to Alan Somers from comment #2)

I pinged the author out-of-band to confirm they didn't use it in other parts of the work (since this was a series across FreeBSD + wulf's repo + another downstream library), but I think they did not.
Comment 4 Kyle Evans freebsd_committer freebsd_triage 2026-06-30 14:16:59 UTC
(In reply to Kyle Evans from comment #3)

I misread the context here, let's just fix this
Comment 5 commit-hook freebsd_committer freebsd_triage 2026-06-30 16:52:13 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=cb8bda40695f5d402f334f48795b1ab27b72dce5

commit cb8bda40695f5d402f334f48795b1ab27b72dce5
Author:     Kyle Evans <kevans@FreeBSD.org>
AuthorDate: 2026-06-30 16:51:57 +0000
Commit:     Kyle Evans <kevans@FreeBSD.org>
CommitDate: 2026-06-30 16:51:57 +0000

    evdev: use a prometheus-safe label for ev_sysctl_tree

    Prometheus doesn't allow spaces, let's normalize this to what we use
    elsewhere for consistency.  The sysctl exporter could probably do this
    itself, but let's decouple that from the immediate problem: matching the
    label between the exported data and in-tree is nice for greppability.

    PR:             296179
    Reviewed by:    asomers, wulf
    Differential Revision:  https://reviews.freebsd.org/D57966

 sys/dev/evdev/evdev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 6 Kyle Evans freebsd_committer freebsd_triage 2026-06-30 17:02:22 UTC
I looked briefly at prometheus_sysctl_exporter, and I'm not 100% sure what to do there.  My gut instinct is that we probably *do* want labels to be as-written in the kernel rather than transformed into something safe, so we probably want to do a validation pass before we print and zap & warn on invalid labels.  Does that mean we drop the entry entirely, or just output it without the label?  The latter seems like it could cause ambiguity issue, so probably we just drop the entry entirely and warn so that one can debug it.
Comment 7 Kyle Evans freebsd_committer freebsd_triage 2026-06-30 17:04:59 UTC
("as-written in the kernel" meaning the kernel just needs to use safe names, to be clear)
Comment 8 Kyle Evans freebsd_committer freebsd_triage 2026-07-01 02:47:23 UTC
I've opened https://reviews.freebsd.org/D57983 with the intention of making these easier to debug and continue writing out all of the other well-formed metrics.
Comment 9 commit-hook freebsd_committer freebsd_triage 2026-08-06 23:22:26 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=e55aa7b67cb0f7a29a0117231b8764524d1fb971

commit e55aa7b67cb0f7a29a0117231b8764524d1fb971
Author:     Kyle Evans <kevans@FreeBSD.org>
AuthorDate: 2026-06-30 16:51:57 +0000
Commit:     Kyle Evans <kevans@FreeBSD.org>
CommitDate: 2026-08-06 22:53:01 +0000

    evdev: use a prometheus-safe label for ev_sysctl_tree

    Prometheus doesn't allow spaces, let's normalize this to what we use
    elsewhere for consistency.  The sysctl exporter could probably do this
    itself, but let's decouple that from the immediate problem: matching the
    label between the exported data and in-tree is nice for greppability.

    PR:             296179
    Reviewed by:    asomers, wulf

    (cherry picked from commit cb8bda40695f5d402f334f48795b1ab27b72dce5)

 sys/dev/evdev/evdev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 10 commit-hook freebsd_committer freebsd_triage 2026-08-07 01:48:48 UTC
A commit in branch stable/15 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=240bd2eeb21f959f7340402211c0bf43a58d31b5

commit 240bd2eeb21f959f7340402211c0bf43a58d31b5
Author:     Kyle Evans <kevans@FreeBSD.org>
AuthorDate: 2026-06-30 16:51:57 +0000
Commit:     Kyle Evans <kevans@FreeBSD.org>
CommitDate: 2026-08-06 23:36:07 +0000

    evdev: use a prometheus-safe label for ev_sysctl_tree

    Prometheus doesn't allow spaces, let's normalize this to what we use
    elsewhere for consistency.  The sysctl exporter could probably do this
    itself, but let's decouple that from the immediate problem: matching the
    label between the exported data and in-tree is nice for greppability.

    PR:             296179
    Reviewed by:    asomers, wulf

    (cherry picked from commit cb8bda40695f5d402f334f48795b1ab27b72dce5)

 sys/dev/evdev/evdev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)