Bug 247776
| Summary: | sysutils/beats7: Filebeat dashboards fail to install on setup | ||
|---|---|---|---|
| Product: | Ports & Packages | Reporter: | ricalo <me> |
| Component: | Individual Port(s) | Assignee: | Juraj Lutter <otis> |
| Status: | Closed FIXED | ||
| Severity: | Affects Some People | CC: | bcr, otis, raul.munoz, sirdice |
| Priority: | --- | Flags: | bugzilla:
maintainer-feedback?
(elastic) |
| Version: | Latest | ||
| Hardware: | amd64 | ||
| OS: | Any | ||
Bumping the importance of this bug. I assume this would affect some people. Does the current version (7.9.1) in ports work for you? (In reply to daniel.engberg.lists from comment #2) No, fails with the same message. Issue remains on 7.9.3, still getting an error 500 response from Kibana when trying to import the dashboards. Copying the dashboards from the corresponding Linux archive does indeed work. Same issue crops up with the dashboards for metricbeat and heartbeat.
2020-11-15T19:06:42.883+0100 ERROR instance/beat.go:951 Exiting: Failed to import dashboard: Failed to load directory /usr/local/share/beats/filebeat/kibana/7/dashboard:
error loading /usr/local/share/beats/filebeat/kibana/7/dashboard/Filebeat-Kafka-overview.json: returned 500 to import file: <nil>.
Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
error loading /usr/local/share/beats/filebeat/kibana/7/dashboard/Filebeat-Mongodb-overview.json: returned 500 to import file: <nil
>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
(and more StatusCode: 500 results for all the other files)
To reproduce:
filebeat --path.config /usr/local/etc/beats --path.home /usr/local/share/beats/filebeat --path.data "/var/db/beats/filebeat" --path.logs "/var/log/beats" setup -e --dashboards
Thanks for noticing this. I'll re-test prior to committing 7.10.0 (that I already have in queue). Still happening with 7.10.0 too. Dug a little deeper and tried to compare the json files with the ones from the Linux archive and I noticed a few differences that might be important.
Picked Filebeat-apache.json but they all appear to have the same kind of differences.
The linux version has this:
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\": []}"
},
But on FreeBSD it's defined like this:
"kibanaSavedObjectMeta": {
"searchSourceJSON": {
"filter": []
}
It looks like the value of "searchSourceJSON" should be taken literally because the value is embedded in quotes. The literal value is a string containing a JSON structure. On FreeBSD the JSON seems to have been parsed as as a structure instead of a literal string value.
Any updates on this issue? I'm seeing the same problems with metricbeat when importing the dashboards, so I guess it affects all beats. I tried copying and using the Linux version 7.15.1, but then I'm seeing the following errors for all of them:
Exiting: Failed to import dashboard: Failed to load directory /usr/local/share/beats/metricbeat/kibana/7/dashboard:
error loading /usr/local/share/beats/metricbeat/kibana/7/dashboard/023d2930-f1a5-11e7-a9ef-93c69af7b129-ecs.json: returned 400 to import file: <nil>. Response: {"statusCode":400,"error":"Bad Request","message":"[request body.objects]: expected value of type [array] but got [undefined]"}
Either the versions need to match exactly with the other installed components (Kibana and Elasticsearch) or the FreeBSD port is using outdated dashboards with JSON errors in them. Can anyone out there build a functional ElasticStack on FreeBSD using the current port version? I can help test patches when needed.
I have already 7.15.1 update in my work queue. Will test and commit in coming few days. Observation: Source code archive downloaded from github has "corrupted"? .json files. Binary pre-built packages from elastic.co have these files with correct contents. Now the best way to replace those improperly escaped json files with correct ones is being investigated. A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=3c5941307404109bdfc8130de42d85a557180ecf commit 3c5941307404109bdfc8130de42d85a557180ecf Author: Juraj Lutter <otis@FreeBSD.org> AuthorDate: 2021-10-24 21:36:34 +0000 Commit: Juraj Lutter <otis@FreeBSD.org> CommitDate: 2021-10-24 22:37:44 +0000 sysutils/beats7: Update to 7.15.1 - Update to 7.15.1 - For breaking changes for 7.15.x see UPDATING - Install kibana dashboard files from binary package (PR 247776) - Install .yml config files into examples directory (PR 258506) - Release notes: https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-7.15.1.html PR: 247776, 258506 UPDATING | 30 + sysutils/beats7/Makefile | 159 ++- sysutils/beats7/distinfo | 92 +- .../files/patch-auditbeat_auditbeat.reference.yml | 11 +- .../files/patch-auditbeat_auditbeat.yml (new) | 13 + .../files/patch-filebeat_filebeat.reference.yml | 13 +- sysutils/beats7/files/patch-filebeat_filebeat.yml | 15 +- .../patch-metricbeat_metricbeat.reference.yml | 11 +- .../beats7/files/patch-metricbeat_metricbeat.yml | 15 +- ...patch-packetbeat_packetbeat.reference.yml (new) | 11 + .../beats7/files/patch-packetbeat_packetbeat.yml | 13 +- sysutils/beats7/files/pkg-message.in (new) | 12 + sysutils/beats7/pkg-plist | 1298 +------------------- 13 files changed, 298 insertions(+), 1395 deletions(-) Fixed. Thanks for the report. |
I configured the following values in /usr/local/etc/beats/filebeat.yml: setup.dashboards.enabled: true setup.dashboards.directory: /usr/local/share/beats/filebeat/kibana After starting the service with service filebeat start, the console throws the following errors for each json file in /usr/local/share/beats/filebeat/kibana/7/dashboard/: error loading /usr/local/share/beats/filebeat/kibana/7/dashboard/Filebeat-logstash-log.json: returned 500 to import file: <nil>. Response {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"} Furthermore, I found the following errors in /var/log/kibana.log: {"type":"log","@timestamp":"2020-07-03T12:49:49Z","tags":["warning","savedobjects-service"],"pid":62625,"message":"Failed to transform document [object Object]. Transform: visualization:7.0.0\nDoc: {\"id\":\"0b1dace0-cbdb-11e7-9852-73e0a9df1bb6-ecs\",\"type\":\"visualization\",\"attributes\":{\"description\":\"\",\"kibanaSavedObjectMeta\":{\"searchSourceJSON\":{\"filter\":[],\"query\":{\"language\":\"kuery\",\"query\":\"\"}}},\"title\":\"Logs Severity [Filebeat Logstash] ECS\",\"uiStateJSON\":{},\"version\":1,\"visState\":{\"aggs\":[{\"enabled\":true,\"id\":\"1\",\"params\":{},\"schema\":\"metric\",\"type\":\"count\"},{\"enabled\":true,\"id\":\"2\",\"params\":{\"field\":\"log.level\",\"order\":\"desc\",\"orderBy\":\"1\",\"size\":5},\"schema\":\"segment\",\"type\":\"terms\"}],\"params\":{\"addLegend\":true,\"addTooltip\":true,\"isDonut\":true,\"legendPosition\":\"right\",\"type\":\"pie\"},\"title\":\"Logs Severity [Filebeat Logstash] ECS\",\"type\":\"pie\"},\"savedSearchRefName\":\"search_0\"},\"migrationVersion\":{\"visualization\":\"6.7.2\"},\"updated_at\":\"2020-07-03T12:49:49.573Z\",\"references\":[{\"type\":\"search\",\"name\":\"search_0\",\"id\":\"cfaba090-cbda-11e7-9852-73e0a9df1bb6-ecs\"}]}"} {"type":"error","@timestamp":"2020-07-03T12:49:49Z","tags":[],"pid":62625,"level":"error","error":{"message":"Failure attempting to migrate saved object 'Logs Severity [Filebeat Logstash] ECS' - SyntaxError: Unexpected token o in JSON at position 1","name":"Error","stack":"Error: Failure attempting to migrate saved object 'Logs Severity [Filebeat Logstash] ECS' - SyntaxError: Unexpected token o in JSON at position 1\n at migrateTableSplits (/usr/local/www/kibana7/src/plugins/visualizations/server/saved_objects/visualization_migrations.js:573:11)\n at /usr/local/www/kibana7/node_modules/lodash/index.js:3375:35\n at tryTransformDoc (/usr/local/www/kibana7/src/core/server/saved_objects/migrations/core/document_migrator.js:224:22)\n at migrateProp (/usr/local/www/kibana7/src/core/server/saved_objects/migrations/core/document_migrator.js:281:11)\n at applyMigrations (/usr/local/www/kibana7/src/core/server/saved_objects/migrations/core/document_migrator.js:180:11)\n at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/local/www/kibana7/src/core/server/saved_objects/migrations/core/document_migrator.js:159:43)\n at DocumentMigrator.doc [as migrate] (/usr/local/www/kibana7/src/core/server/saved_objects/migrations/core/document_migrator.js:52:19)\n at KibanaMigrator.migrateDocument (/usr/local/www/kibana7/src/core/server/saved_objects/migrations/kibana/kibana_migrator.js:142:34)\n at objects.map.object (/usr/local/www/kibana7/src/core/server/saved_objects/service/lib/repository.js:227:74)\n at Array.map (<anonymous>)\n at SavedObjectsRepository.bulkCreate (/usr/local/www/kibana7/src/core/server/saved_objects/service/lib/repository.js:211:37)\n at SavedObjectsClient.bulkCreate (/usr/local/www/kibana7/src/core/server/saved_objects/service/saved_objects_client.js:46:35)\n at EncryptedSavedObjectsClientWrapper.bulkCreate (/usr/local/www/kibana7/x-pack/plugins/encrypted_saved_objects/server/saved_objects/encrypted_saved_objects_client_wrapper.js:80:89)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?exclude=index-pattern&force=true","query":{"exclude":"index-pattern","force":"true"},"pathname":"/api/kibana/dashboards/import","path":"/api/kibana/dashboards/import?exclude=index-pattern&force=true","href":"/api/kibana/dashboards/import?exclude=index-pattern&force=true"},"message":"Failure attempting to migrate saved object 'Logs Severity [Filebeat Logstash] ECS' - SyntaxError: Unexpected token o in JSON at position 1"} I tried replacing the JSON files with the corresponding files from https://github.com/elastic/beats to no avail. Workaround: 1. Downloaded beat from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.1-linux-x86_64.tar.gz 2. Unpackaged tar file (tar -xvf filebeat-7.7.1-linux-x86_64.tar.gz) 3. Copied json files to expected location (cp filebeat-7.7.1-linux-x86_64/kibana/7/dashboard/*.json /usr/local/share/beats/filebeat/kibana/7/dashboard/) 4. Start the service (service filebeat start)