Hi, I recently noticed my vmagent instances are failing to scrape metrics due to frequent crashes. I'm putting the crash message at the end of this comment. Some analysis with a friend (Lapo Luchini) uncovered a related go bug report: https://github.com/VictoriaMetrics/metrics/issues/59 It looks like victoriametrics 1.93.9 uses "github.com/VictoriaMetrics/metrics v1.25.3" while the new 1.93.10 uses v1.25.4 which fixes the issue, adding code to handle the new metrics value. So I think it is urgent to update victoriametrics to the newer version. I see bug #275828 contains an update, but also contains other changes. I'm going to test a more simple update consisting only in a version bump that maybe can be integrated faster. ---- Crash message: panic: BUG: unexpected runtimemetrics.KindBad for sample.Name="/gc/gomemlimit:bytes" goroutine 133 [running]: github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper.func1() /wrkdirs/usr/ports/net-mgmt/vmutils/work/VictoriaMetrics-1.93.9/lib/httpserver/httpserver.go:235 +0x65 panic({0x994520, 0x84c414dc0}) /usr/local/go120/src/runtime/panic.go:890 +0x263 github.com/VictoriaMetrics/metrics.writeRuntimeMetric({0xbb22c0, 0x84c0985d0}, {0xa4260e?, 0x1?}, 0x18?) /wrkdirs/usr/ports/net-mgmt/vmutils/work/VictoriaMetrics-1.93.9/vendor/github.com/VictoriaMetrics/metrics/go_metrics.go:97 +0x1d7 github.com/VictoriaMetrics/metrics.writeRuntimeMetrics({0xbb22c0, 0x84c0985d0}) /wrkdirs/usr/ports/net-mgmt/vmutils/work/VictoriaMetrics-1.93.9/vendor/github.com/VictoriaMetrics/metrics/go_metrics.go:90 +0x139 github.com/VictoriaMetrics/metrics.writeGoMetrics({0xbb22c0, 0x84c0985d0}) /wrkdirs/usr/ports/net-mgmt/vmutils/work/VictoriaMetrics-1.93.9/vendor/github.com/VictoriaMetrics/metrics/go_metrics.go:26 +0x46 github.com/VictoriaMetrics/metrics.WriteProcessMetrics({0xbb22c0, 0x84c0985d0}) /wrkdirs/usr/ports/net-mgmt/vmutils/work/VictoriaMetrics-1.93.9/vendor/github.com/VictoriaMetrics/metrics/metrics.go:213 +0x25 github.com/VictoriaMetrics/metrics.WritePrometheus({0xbb22c0, 0x84c0985d0}, 0x1) /wrkdirs/usr/ports/net-mgmt/vmutils/work/VictoriaMetrics-1.93.9/vendor/github.com/VictoriaMetrics/metrics/metrics.go:88 +0x299 github.com/VictoriaMetrics/VictoriaMetrics/lib/appmetrics.writePrometheusMetrics({0xbb22c0, 0x84c0985d0}) /wrkdirs/usr/ports/net-mgmt/vmutils/work/VictoriaMetrics-1.93.9/lib/appmetrics/appmetrics.go:46 +0x45 github.com/VictoriaMetrics/VictoriaMetrics/lib/appmetrics.WritePrometheusMetrics({0x3efbfa7df1f8, 0x84c0240a0}) /wrkdirs/usr/ports/net-mgmt/vmutils/work/VictoriaMetrics-1.93.9/lib/appmetrics/appmetrics.go:29 +0xba github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper(0x84c3ba760, {0xbb65b0?, 0x84c0240a0}, 0x84c41e000, 0xb21688) /wrkdirs/usr/ports/net-mgmt/vmutils/work/VictoriaMetrics-1.93.9/lib/httpserver/httpserver.go:306 +0xa67 github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.gzipHandler.func1({0xbb65b0?, 0x84c0240a0?}, 0x84c444000?) /wrkdirs/usr/ports/net-mgmt/vmutils/work/VictoriaMetrics-1.93.9/lib/httpserver/httpserver.go:196 +0x37 net/http.HandlerFunc.ServeHTTP(0xee6c00?, {0xbb65b0?, 0x84c0240a0?}, 0x4?) /usr/local/go120/src/net/http/server.go:2122 +0x2f github.com/klauspost/compress/gzhttp.NewWrapper.func1.1({0xbb64c0, 0x84c444000}, 0x9be080?) /wrkdirs/usr/ports/net-mgmt/vmutils/work/VictoriaMetrics-1.93.9/vendor/github.com/klauspost/compress/gzhttp/compress.go:495 +0x35a net/http.HandlerFunc.ServeHTTP(0x0?, {0xbb64c0?, 0x84c444000?}, 0x46b26e?) /usr/local/go120/src/net/http/server.go:2122 +0x2f net/http.serverHandler.ServeHTTP({0x84c52d020?}, {0xbb64c0, 0x84c444000}, 0x84c41e000) /usr/local/go120/src/net/http/server.go:2936 +0x316 net/http.(*conn).serve(0x84c0002d0, {0xbb6928, 0x84c52cf90}) /usr/local/go120/src/net/http/server.go:1995 +0x612 created by net/http.(*Server).Serve /usr/local/go120/src/net/http/server.go:3089 +0x5ed
(In reply to Guido Falsi from comment #0) > I see bug #275828 contains an update, but also contains other changes. Yeah it should be a simple update, those other changes are dubious and unrelated.
(In reply to Alexey Dokuchaev from comment #1) The new rc scripts there could be a useful addition. You should evaluate importing those at least.
(In reply to Guido Falsi from comment #0) > Some analysis with a friend (Lapo Luchini) uncovered a related go bug > report: https://github.com/VictoriaMetrics/metrics/issues/59 [...] > It looks like victoriametrics 1.93.9 uses "github.com/VictoriaMetrics/ > metrics v1.25.3" That's right. > while the new 1.93.10 uses v1.25.4 which fixes the issue, adding code to > handle the new metrics value. Technically 1.93.10 uses v1.31.0 which I assume naturally also contains the fix. (In reply to Guido Falsi from comment #2) > The new rc scripts there could be a useful addition. You should evaluate > importing those at least. I think Lapo had asked me about rc script(s) for vmutils several months ago, but it somehow went off the radar as there were no initial patch to build upon (unlike now). I'll check them and and probably include with the port.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=e44c9efcca8e50a586976ce3b63032ed63208a55 commit e44c9efcca8e50a586976ce3b63032ed63208a55 Author: Alexey Dokuchaev <danfe@FreeBSD.org> AuthorDate: 2024-01-27 10:04:20 +0000 Commit: Alexey Dokuchaev <danfe@FreeBSD.org> CommitDate: 2024-01-27 10:04:20 +0000 net-mgmt/victoria-metrics: update VictoriaMetrics to version 1.93.10 Notably, this fixes the issue of vmagent instances failing to scrape metrics due to frequent crashes, after Golang update to version 1.20 lead to errors (panics) like: panic: BUG: unexpected runtimemetrics.KindBad for sample.Name="/gc/gomemlimit:bytes" PR: 275828, 276639 net-mgmt/victoria-metrics/Makefile | 4 +- net-mgmt/victoria-metrics/Makefile.deps | 65 ++++++++------- net-mgmt/victoria-metrics/distinfo | 136 +++++++++++++++++++------------- 3 files changed, 119 insertions(+), 86 deletions(-)
Closing this as the issue reported here is now fixed. Thanks!