Bug 253862 - prometheus_sysctl_exporter outputs invalid format
Summary: prometheus_sysctl_exporter outputs invalid format
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.2-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
: 258393 (view as bug list)
Depends on:
Blocks:
 
Reported: 2021-02-26 04:56 UTC by Devin Smith
Modified: 2021-09-09 21:25 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Devin Smith 2021-02-26 04:56:21 UTC
It appears that /usr/sbin/prometheus_sysctl_exporter is outputting an invalid prometheus format when used with the "-d" option.

$ prometheus_sysctl_exporter -d | grep HELP | awk -F " " '{print $3}' | sort | uniq -c | sort | grep -v " 1 "
   2 sysctl_vfs_zfs_arc_max
   2 sysctl_vfs_zfs_arc_min
   2 sysctl_vfs_zfs_l2arc_feed_again
   2 sysctl_vfs_zfs_l2arc_feed_min_ms
   2 sysctl_vfs_zfs_l2arc_feed_secs
   2 sysctl_vfs_zfs_l2arc_headroom
   2 sysctl_vfs_zfs_l2arc_noprefetch
   2 sysctl_vfs_zfs_l2arc_norw
   2 sysctl_vfs_zfs_l2arc_write_boost
   2 sysctl_vfs_zfs_l2arc_write_max

"Only one HELP line may exist for any given metric name." - https://github.com/prometheus/docs/blob/33f891f/content/docs/instrumenting/exposition_formats.md#comments-help-text-and-type-information

This is causing a downstream consumer to fail during parsing:

$ grep node_exporter /var/log/messages

Feb 25 20:43:51 prometheus node_exporter[46973]: level=error ts=2021-02-26T04:43:51.646Z caller=textfile.go:209 collector=textfile msg="failed to collect textfile data" file=sysctl.prom err="failed to parse textfile data from \"/var/tmp/node_exporter/sysctl.prom\": text format parsing error in line 1974: second HELP line for metric name \"sysctl_vfs_zfs_l2arc_norw\""

It's easy enough to workaround by just excluding the "-d" flag, but it would be nice for prometheus to gather the additional context that "-d" provides.
Comment 1 Paul Holmanskikh 2021-07-24 12:02:21 UTC
Same problem with 13.0.

Actuall sysctl -a returns 2 entries.

root@feeebsd:~ # sysctl -a | grep log.bucket.size:
vm.uma.tcp_log_bucket.size: 176
vm.uma.tcp_log.bucket_size: 30

Confirmed on multiple hosts.
Comment 2 Paul Holmanskikh 2021-07-24 12:30:38 UTC
Root cause is that default "sysctl" entries name conversion method when "." are replaced by "-" causes metric name clash for 

vm.uma.tcp_log_bucket.size
vm.uma.tcp_log.bucket_size

I do not see any good solution for it. 
Meanwhile workaround is to remove both metrics by running 

/usr/bin/sed -I "" '/sysctl_vm_uma_tcp_log_bucket_size .*$/d' filename.

against prometheus_sysctl_exporter output file.
Comment 3 Lapo Luchini 2021-09-09 18:22:34 UTC
*** Bug 258393 has been marked as a duplicate of this bug. ***
Comment 4 Lapo Luchini 2021-09-09 18:24:06 UTC
This is not worked around by excluding "-d", it happens anyways on values (not only on #HELP lines).
Comment 5 Lapo Luchini 2021-09-09 21:24:49 UTC
I'm using this as a workaround:

  prometheus_sysctl_exporter | sort -k1,1 -u

(of course, )
Comment 6 Lapo Luchini 2021-09-09 21:25:24 UTC
(of course, this will drop any value after the first)