Bug 247776 - sysutils/beats7: Filebeat dashboards fail to install on setup
Summary: sysutils/beats7: Filebeat dashboards fail to install on setup
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Some People
Assignee: Juraj Lutter
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-07-05 13:30 UTC by ricalo
Modified: 2020-12-29 16:49 UTC (History)
4 users (show)

See Also:
bugzilla: maintainer-feedback? (elastic)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description ricalo 2020-07-05 13:30:40 UTC
I configured the following values in /usr/local/etc/beats/filebeat.yml:

setup.dashboards.enabled: true
setup.dashboards.directory: /usr/local/share/beats/filebeat/kibana

After starting the service with service filebeat start, the console throws the following errors for each json file in /usr/local/share/beats/filebeat/kibana/7/dashboard/:

error loading /usr/local/share/beats/filebeat/kibana/7/dashboard/Filebeat-logstash-log.json: returned 500 to import file: <nil>. Response {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

Furthermore, I found the following errors in /var/log/kibana.log:

{"type":"log","@timestamp":"2020-07-03T12:49:49Z","tags":["warning","savedobjects-service"],"pid":62625,"message":"Failed to transform document [object Object]. Transform: visualization:7.0.0\nDoc: {\"id\":\"0b1dace0-cbdb-11e7-9852-73e0a9df1bb6-ecs\",\"type\":\"visualization\",\"attributes\":{\"description\":\"\",\"kibanaSavedObjectMeta\":{\"searchSourceJSON\":{\"filter\":[],\"query\":{\"language\":\"kuery\",\"query\":\"\"}}},\"title\":\"Logs Severity [Filebeat Logstash] ECS\",\"uiStateJSON\":{},\"version\":1,\"visState\":{\"aggs\":[{\"enabled\":true,\"id\":\"1\",\"params\":{},\"schema\":\"metric\",\"type\":\"count\"},{\"enabled\":true,\"id\":\"2\",\"params\":{\"field\":\"log.level\",\"order\":\"desc\",\"orderBy\":\"1\",\"size\":5},\"schema\":\"segment\",\"type\":\"terms\"}],\"params\":{\"addLegend\":true,\"addTooltip\":true,\"isDonut\":true,\"legendPosition\":\"right\",\"type\":\"pie\"},\"title\":\"Logs Severity [Filebeat Logstash] ECS\",\"type\":\"pie\"},\"savedSearchRefName\":\"search_0\"},\"migrationVersion\":{\"visualization\":\"6.7.2\"},\"updated_at\":\"2020-07-03T12:49:49.573Z\",\"references\":[{\"type\":\"search\",\"name\":\"search_0\",\"id\":\"cfaba090-cbda-11e7-9852-73e0a9df1bb6-ecs\"}]}"}
{"type":"error","@timestamp":"2020-07-03T12:49:49Z","tags":[],"pid":62625,"level":"error","error":{"message":"Failure attempting to migrate saved object 'Logs Severity [Filebeat Logstash] ECS' - SyntaxError: Unexpected token o in JSON at position 1","name":"Error","stack":"Error: Failure attempting to migrate saved object 'Logs Severity [Filebeat Logstash] ECS' - SyntaxError: Unexpected token o in JSON at position 1\n    at migrateTableSplits (/usr/local/www/kibana7/src/plugins/visualizations/server/saved_objects/visualization_migrations.js:573:11)\n    at /usr/local/www/kibana7/node_modules/lodash/index.js:3375:35\n    at tryTransformDoc (/usr/local/www/kibana7/src/core/server/saved_objects/migrations/core/document_migrator.js:224:22)\n    at migrateProp (/usr/local/www/kibana7/src/core/server/saved_objects/migrations/core/document_migrator.js:281:11)\n    at applyMigrations (/usr/local/www/kibana7/src/core/server/saved_objects/migrations/core/document_migrator.js:180:11)\n    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/local/www/kibana7/src/core/server/saved_objects/migrations/core/document_migrator.js:159:43)\n    at DocumentMigrator.doc [as migrate] (/usr/local/www/kibana7/src/core/server/saved_objects/migrations/core/document_migrator.js:52:19)\n    at KibanaMigrator.migrateDocument (/usr/local/www/kibana7/src/core/server/saved_objects/migrations/kibana/kibana_migrator.js:142:34)\n    at objects.map.object (/usr/local/www/kibana7/src/core/server/saved_objects/service/lib/repository.js:227:74)\n    at Array.map (<anonymous>)\n    at SavedObjectsRepository.bulkCreate (/usr/local/www/kibana7/src/core/server/saved_objects/service/lib/repository.js:211:37)\n    at SavedObjectsClient.bulkCreate (/usr/local/www/kibana7/src/core/server/saved_objects/service/saved_objects_client.js:46:35)\n    at EncryptedSavedObjectsClientWrapper.bulkCreate (/usr/local/www/kibana7/x-pack/plugins/encrypted_saved_objects/server/saved_objects/encrypted_saved_objects_client_wrapper.js:80:89)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?exclude=index-pattern&force=true","query":{"exclude":"index-pattern","force":"true"},"pathname":"/api/kibana/dashboards/import","path":"/api/kibana/dashboards/import?exclude=index-pattern&force=true","href":"/api/kibana/dashboards/import?exclude=index-pattern&force=true"},"message":"Failure attempting to migrate saved object 'Logs Severity [Filebeat Logstash] ECS' - SyntaxError: Unexpected token o in JSON at position 1"}

I tried replacing the JSON files with the corresponding files from https://github.com/elastic/beats to no avail.

Workaround:
1. Downloaded beat from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.1-linux-x86_64.tar.gz
2. Unpackaged tar file (tar -xvf filebeat-7.7.1-linux-x86_64.tar.gz)
3. Copied json files to expected location (cp filebeat-7.7.1-linux-x86_64/kibana/7/dashboard/*.json /usr/local/share/beats/filebeat/kibana/7/dashboard/)
4. Start the service (service filebeat start)
Comment 1 ricalo 2020-07-18 16:34:58 UTC
Bumping the importance of this bug. I assume this would affect some people.
Comment 2 daniel.engberg.lists 2020-09-13 20:38:57 UTC
Does the current version (7.9.1) in ports work for you?
Comment 3 ricalo 2020-09-15 00:17:00 UTC
(In reply to daniel.engberg.lists from comment #2)

No, fails with the same message.
Comment 4 sirdice 2020-11-19 21:33:06 UTC
Issue remains on 7.9.3, still getting an error 500 response from Kibana when trying to import the dashboards. Copying the dashboards from the corresponding Linux archive does indeed work. Same issue crops up with the dashboards for metricbeat and heartbeat. 

2020-11-15T19:06:42.883+0100    ERROR   instance/beat.go:951    Exiting: Failed to import dashboard: Failed to load directory /usr/local/share/beats/filebeat/kibana/7/dashboard:
  error loading /usr/local/share/beats/filebeat/kibana/7/dashboard/Filebeat-Kafka-overview.json: returned 500 to import file: <nil>.
 Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
  error loading /usr/local/share/beats/filebeat/kibana/7/dashboard/Filebeat-Mongodb-overview.json: returned 500 to import file: <nil
>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}

(and more StatusCode: 500 results for all the other files)

To reproduce:
filebeat --path.config /usr/local/etc/beats --path.home /usr/local/share/beats/filebeat --path.data "/var/db/beats/filebeat" --path.logs "/var/log/beats" setup -e --dashboards
Comment 5 Juraj Lutter freebsd_committer 2020-11-19 21:47:38 UTC
Thanks for noticing this. I'll re-test prior to committing 7.10.0 (that I already have in queue).
Comment 6 sirdice 2020-12-10 21:46:41 UTC
Still happening with 7.10.0 too. Dug a little deeper and tried to compare the json files with the ones from the Linux archive and I noticed a few differences that might be important. 

Picked Filebeat-apache.json but they all appear to have the same kind of differences.

The linux version has this:
                "kibanaSavedObjectMeta": {
                    "searchSourceJSON": "{\"filter\": []}"
                },

But on FreeBSD it's defined like this:
                "kibanaSavedObjectMeta": {
                    "searchSourceJSON": {
                        "filter": []
                    }

It looks like the value of "searchSourceJSON" should be taken literally because the value is embedded in quotes. The literal value is a string containing a JSON structure. On FreeBSD the JSON seems to have been parsed as as a structure instead of a literal string value.