Bug 244849 - [NEW PORT] sysutils/beats7: Collect logs locally and send to remote logstash
Summary: [NEW PORT] sysutils/beats7: Collect logs locally and send to remote logstash
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-elastic mailing list
URL:
Keywords: feature, needs-qa
: 246444 (view as bug list)
Depends on:
Blocks:
 
Reported: 2020-03-16 18:18 UTC by Miroslav Lachman
Modified: 2020-06-04 09:29 UTC (History)
9 users (show)

See Also:
koobs: maintainer-feedback? (elastic)
koobs: maintainer-feedback? (000.fbsd)


Attachments
new port as patch / upgrade from repo copied beats to beats7 (157.21 KB, patch)
2020-03-16 18:18 UTC, Miroslav Lachman
no flags Details | Diff
shar to add beats7 - 7.6.1 (90.45 KB, text/plain)
2020-03-16 18:19 UTC, Miroslav Lachman
no flags Details
poudriere testport build log / plist complaints (19.65 KB, text/plain)
2020-03-16 18:21 UTC, Miroslav Lachman
no flags Details
new port as patch / upgrade from repo copied beats to beats7 (158.02 KB, patch)
2020-03-16 19:50 UTC, Miroslav Lachman
no flags Details | Diff
shar to add beats7 - 7.6.1 (91.26 KB, text/plain)
2020-03-16 19:51 UTC, Miroslav Lachman
no flags Details
fix for packetbeat build (666 bytes, patch)
2020-03-31 16:01 UTC, Ryan Lawrence
bug: maintainer-approval+
Details | Diff
overall/latest changes summarized (100.53 KB, text/plain)
2020-05-19 14:59 UTC, Regis A. Despres
no flags Details
ari contribution included (107.11 KB, text/plain)
2020-05-28 07:44 UTC, Regis A. Despres
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Miroslav Lachman 2020-03-16 18:18:21 UTC
Created attachment 212448 [details]
new port as patch / upgrade from repo copied beats to beats7

As was asked by Greg Lewis here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237374
I am providing the first wild take on beats7.
Again - as I am not using / running Kibana nor Beats I cannot properly test it. It was just about modifying few patches from beats 6 to match beats 7 and modify pkg-plist.
Poudriere testport complaints about few empty dirs left in /var/db/beats but I tested current sysutils/beats and there are even more complaints about empty directories left after pkg delete so I am not sure if I should add those dirs to pkg-plist or not.

====> Checking for pkg-plist issues (check-plist)
===> Parsing plist
===> Checking for items in STAGEDIR missing from pkg-plist
Error: Orphaned: @dir /var/db/beats/filebeat/kibana/7/dashboard
Error: Orphaned: @dir /var/db/beats/filebeat/kibana/7
Error: Orphaned: @dir /var/db/beats/filebeat/kibana
Error: Orphaned: @dir /var/db/beats/heartbeat/kibana
Error: Orphaned: @dir /var/db/beats/metricbeat/kibana/7/dashboard
Error: Orphaned: @dir /var/db/beats/metricbeat/kibana/7
Error: Orphaned: @dir /var/db/beats/metricbeat/kibana
===> Checking for items in pkg-plist which are not in STAGEDIR
===> Error: Plist issues found.
*** Error code 1

Also the pkg-plist does not contain entries for packetbeat because this module is marked as broken and cannot be build. 
https://github.com/elastic/beats/issues/15256
Comment 1 Miroslav Lachman 2020-03-16 18:19:34 UTC
Created attachment 212449 [details]
shar to add beats7 - 7.6.1
Comment 2 Miroslav Lachman 2020-03-16 18:21:32 UTC
Created attachment 212450 [details]
poudriere testport build log / plist complaints

The same issues are in current sysutils/beats port so I am not sure if it is intentional or not
Comment 3 Miroslav Lachman 2020-03-16 19:50:14 UTC
Created attachment 212452 [details]
new port as patch / upgrade from repo copied beats to beats7

plist options fix
Comment 4 Miroslav Lachman 2020-03-16 19:51:25 UTC
Created attachment 212453 [details]
shar to add beats7 - 7.6.1

plist options fix
Comment 5 Greg Lewis freebsd_committer 2020-03-22 17:31:56 UTC
I think that it makes a lot of sense to first move the current beats port to beats6 before we commit a beats7.
Comment 6 Juraj Lutter 2020-03-22 17:34:48 UTC
What is the proper workflow to rename the port? I did quite a bit of a work with beats6 so I'd be interested in renaming it, too.
Comment 7 Miroslav Lachman 2020-03-23 10:13:10 UTC
Can somebody more experienced with packaging take a look at thees issue?
https://lists.freebsd.org/pipermail/freebsd-elastic/2020-March/000563.html

The problem is that install process is trying to create hardlink to file which can be on dofferent partition (hardlink from /usr/local to /var/db) and it failed if those are separate filesystems. But it passed pouderiere testport where everything is on one filesystem.
Comment 8 Greg Lewis freebsd_committer 2020-03-27 04:48:38 UTC
Renaming the port shouldn't be too difficult.  If someone wants to copy it over, make the necessary changes and post a diff or shar I can handle committing the actual changes.
Comment 9 Ryan Lawrence 2020-03-31 16:01:46 UTC
Created attachment 212894 [details]
fix for packetbeat build

I've added attachment patch for fixing the build issue for packetbeat. The Makefile will need to be updated to remove "PACKETBEAT_BROKEN=     An underlying library is currently broken under FreeBSD" which disables it from being built. Tested and builds and runs without any issues. Currently using it in production enjoy!
Comment 10 Patrice 2020-04-25 11:55:42 UTC
Hello,

we tested the shar file and compilation fails. Here are the details:

# pwd
/usr/ports/sysutils/beats7
# make
===>  License APACHE20 accepted by the user
===>   beats7-7.6.1 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by beats7-7.6.1 for building
===>  Extracting for beats7-7.6.1
=> SHA256 Checksum OK for elastic-beats-v7.6.1_GH0.tar.gz.
===>  Patching for beats7-7.6.1
===>  Applying FreeBSD patches for beats7-7.6.1
===>   beats7-7.6.1 depends on executable: gmake - found
===>   beats7-7.6.1 depends on file: /usr/local/bin/go - found
===>  Configuring for beats7-7.6.1
===>  Building for beats7-7.6.1
gmake[2]: Entering directory '/usr/ports/usr/ports/sysutils/beats7/work/beats-7.6.1/filebeat'
go build 
# github.com/elastic/beats/vendor/github.com/DataDog/zstd
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:61:22: could not determine kind of name for C.ZSTD_compressBegin
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:64:22: could not determine kind of name for C.ZSTD_compressBegin_usingDict
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:94:13: could not determine kind of name for C.ZSTD_compressContinue
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:120:13: could not determine kind of name for C.ZSTD_compressEnd
gmake[2]: *** [../libbeat/scripts/Makefile:121: filebeat] Error 2
gmake[2]: Leaving directory '/usr/ports/usr/ports/sysutils/beats7/work/beats-7.6.1/filebeat'
*** Error code 2

Stop.
make[1]: stopped in /usr/ports/sysutils/beats7
*** Error code 1

Stop.
make: stopped in /usr/ports/sysutils/beats7
Comment 11 Regis A. Despres 2020-04-27 23:06:01 UTC
(In reply to Ryan Lawrence from comment #9)
File to patch seems not present anymore =)

Regards,



-- 
Regis A. Despres
Comment 12 Regis A. Despres 2020-04-30 11:27:33 UTC
Hi there,

Letting know your work has been put together in addition w/few things (plist fix, auditbeat addition, fixed packetbeat patch ..) into a repo linked to cirrus-ci for automated poudriere runs and test artefact disposal here : https://github.com/kalw/beats-fbsd-port/releases

Hope this help to get into mainstream soon :)


Regards,



-- 
Regis A. Despres
Comment 13 Patrice 2020-04-30 11:57:18 UTC
(In reply to Regis A. Despres from comment #12)
Hello,

thanks for your work. Still the same compilation problem:

root@ports12x:/usr/ports/sysutils/beats7 # make

===>  License APACHE20 accepted by the user
===>   beats7-7.6.2 depends on file: /usr/local/sbin/pkg - found
=> elastic-beats-v7.6.2_GH0.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch https://codeload.github.com/elastic/beats/tar.gz/v7.6.2?dummy=/elastic-beats-v7.6.2_GH0.tar.gz
fetch: https://codeload.github.com/elastic/beats/tar.gz/v7.6.2?dummy=/elastic-beats-v7.6.2_GH0.tar.gz: size unknown
fetch: https://codeload.github.com/elastic/beats/tar.gz/v7.6.2?dummy=/elastic-beats-v7.6.2_GH0.tar.gz: size of remote file is not known
elastic-beats-v7.6.2_GH0.tar.gz                         74 MB 6547 kBps    12s
===> Fetching all distfiles required by beats7-7.6.2 for building
===>  Extracting for beats7-7.6.2
=> SHA256 Checksum OK for elastic-beats-v7.6.2_GH0.tar.gz.
===>  Patching for beats7-7.6.2
===>  Applying FreeBSD patches for beats7-7.6.2
===>   beats7-7.6.2 depends on executable: gmake - found
===>   beats7-7.6.2 depends on file: /usr/local/bin/go - found
===>  Configuring for beats7-7.6.2
===>  Building for beats7-7.6.2
gmake[2]: Entering directory '/usr/ports/usr/ports/sysutils/beats7/work/beats-7.6.2/filebeat'
go build 
# github.com/elastic/beats/vendor/github.com/DataDog/zstd
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:61:22: could not determine kind of name for C.ZSTD_compressBegin
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:64:22: could not determine kind of name for C.ZSTD_compressBegin_usingDict
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:94:13: could not determine kind of name for C.ZSTD_compressContinue
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:120:13: could not determine kind of name for C.ZSTD_compressEnd
gmake[2]: *** [../libbeat/scripts/Makefile:121: filebeat] Error 2
gmake[2]: Leaving directory '/usr/ports/usr/ports/sysutils/beats7/work/beats-7.6.2/filebeat'
*** Error code 2

Stop.
make[1]: stopped in /usr/ports/sysutils/beats7
*** Error code 1

Stop.
make: stopped in /usr/ports/sysutils/beats7
Comment 14 Regis A. Despres 2020-04-30 12:25:15 UTC
(In reply to Patrice from comment #13)

Hi Patrice,

Did you get into the `portsnap fetch update` step ?
While I don't have any solution yet for the usecase you report, you might leverage the automated build packages https://cirrus-ci.com/task/5367768021204992 if you are in 12x on amd64.


Regards,



-- 
Regis A. Despres
Comment 15 Patrice 2020-05-04 05:19:01 UTC
Thanks. Yes, portsnap update was run, still the same error.

We don't use poudriere for building our ports but portmaster. So the current port doesn't work with portmaster (nor with standard make in /usr/ports directories).

Best regards,
Comment 16 Regis A. Despres 2020-05-06 20:56:19 UTC
(In reply to Patrice from comment #15)
Could share your `uname -a`, `go version` and `gmake -v` ouputs in order to reproduce your case ?

CI matrix actually automatically build 12.1-RELEASE , 11.3-STABLE . 12.1-STABLE manually tested reports also ok , i.e. :

vagrant init freebsd/FreeBSD-12.0-CURRENT
vagrant up
vagrant ssh 
uname -a
    FreeBSD freebsd 12.1-STABLE FreeBSD 12.1-STABLE r359553 GENERIC  amd64
sudo portsnap fetch upgrade
sudo pkg install -y git
git clone https://github.com/kalw/beats-fbsd-port.git
sudo mv beats-fbsd-port/sysutils/beats7 /usr/ports/sysutils/beats7
sudo make -DBATCH=yes -C /usr/ports/sysutils/beats7/
..
====> Compressing man pages (compress-man)
===> Staging rc.d startup script(s)
sudo make -DBATCH=yes -C /usr/ports/sysutils/beats7/ install package
===>  Installing for beats7-7.6.2
===>  Checking if beats7 is already installed
===>   Registering installation for beats7-7.6.2
Installing beats7-7.6.2...
===>  Building package for beats7-7.6.2
% filebeat version
filebeat version 7.6.2 (amd64), libbeat 7.6.2 [unknown built unknown]
% metricbeat version
metricbeat version 7.6.2 (amd64), libbeat 7.6.2 [unknown built unknown]




--
Regis A. Despres
Comment 17 Regis A. Despres 2020-05-06 21:03:48 UTC
(In reply to Regis A. Despres from comment #16)

For the record

go version
go version go1.14.2 freebsd/amd64

gmake -v
GNU Make 4.2.1
Comment 18 Patrice 2020-05-07 07:37:52 UTC
(In reply to Regis A. Despres from comment #17)

Thanks, will try your vagrant stuff. Could you share the content of your vagrantfile?

Output for versions:

root@ports12x:~ # uname -a
FreeBSD ports12x 12.1-RELEASE-p2 FreeBSD 12.1-RELEASE-p2 GENERIC  amd64
root@ports12x:~ # go version
go version go1.14.2 freebsd/amd64
root@ports12x:~ # gmake -v
GNU Make 4.2.1
Built for amd64-portbld-freebsd12.1
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Comment 19 Regis A. Despres 2020-05-07 07:55:58 UTC
(In reply to Patrice from comment #18)
`vagrant init something` create the vagrantfile you should not even opening it
Anyway mine is slightly modified like that:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vagrant.plugins = ["vagrant-disksize"]
config.vm.define "bsd-12.1-STABLE" do |bsd|
    bsd.vm.box = "freebsd/FreeBSD-12.1-STABLE"
    bsd.disksize.size = '50GB'
    bsd.ssh.shell = "sh"
    bsd.vm.provider :virtualbox do |v|
      v.customize ["modifyvm", :id, "--memory", 2048]
      v.customize ["modifyvm", :id, "--name", "bsd-12.1-STABLE"]
    end
  end
end
Comment 20 Patrice 2020-05-07 08:03:47 UTC
(In reply to Regis A. Despres from comment #19)

The FreeBSD-12.0-CURRENT from Vagrant fails at git install:

root@freebsd:/home/vagrant # pkg install -y git
ld-elf.so.1: Shared object "libssl.so.111" not found, required by "pkg"
root@freebsd:/home/vagrant # pkg upgrade
ld-elf.so.1: Shared object "libssl.so.111" not found, required by "pkg"

I am with trying your Vagrantfile now.
Comment 21 Patrice 2020-05-07 08:52:16 UTC
(In reply to Patrice from comment #20)

Ok, I found the issue. Having zstd-1.4.4_1 port installed before performing the make triggers it. Deinstalling zstd-1.4.4_1 makes the compilation works ok.

So, there is still someting weird somewhere.

Regards,
Comment 22 Regis A. Despres 2020-05-07 10:51:33 UTC
(In reply to Patrice from comment #21)

Cool I'll try to add the constraint


-- 
Regis A. Despres
Comment 23 Regis A. Despres 2020-05-10 12:25:00 UTC
added in makefile , I guess once version confirmed it will need to be reflected on zstd side.


Regards,


--
Regis A. Despres
Comment 24 Regis A. Despres 2020-05-19 11:00:02 UTC
Hi there,

How can we help to move this initiative forward ?


Regards,



-- 
Regis A. Despres
Comment 25 Kubilay Kocak freebsd_committer freebsd_triage 2020-05-19 11:08:29 UTC
As of today, it appears the following is needed to progress:

- A *single* patch (atached in this issue that has been QA'd (addressing any issues mentioned in comment 0 and comment 10 for example and pases portlint/poudriere at least)

- Feedback (and approval) from beats maintainer (elastic@) regarding upgrade process from beats 6 -> 7 (comment 5)

@Miroslav/Regis Could you integrate, if it hasn't been already, Ryans 'fix for packetbeat build' patch into your patch please (obsoleting the old one and Ryans during attachment)

^Triage: Since elastic@ maintainers the current beats port, assign this issue to them
Comment 26 Regis A. Despres 2020-05-19 14:59:36 UTC
Created attachment 214643 [details]
overall/latest changes summarized

Hi there,

Thanks for the prompt reply !
Attached the shar that match the last successful build w/poudriere on 11.3-RELEASE and 12.3-RELEASE.
Packetbeat patch included, version updated to 7.6.2, auditbeat added (in define option only), conflict with zstd added
Overall build and test process can be reviewed here https://github.com/kalw/beats-fbsd-port/blob/master/.cirrus.yml
Logs and artefacts can also be found here https://cirrus-ci.com/build/5651205462425600


Regards,



-- 
Regis A. Despres
Comment 27 ari 2020-05-23 03:23:16 UTC
Using that last patch 214643 I get this error:

[1/2] Deleting files for beats-6.8.7_1: 100%
[2/2] Installing beats7-7.6.2...
[2/2] Extracting beats7-7.6.2:  93%
pkg: Fail to create hardlink: /var/db/beats/filebeat/kibana/7/dashboard/.pkgtemp.Filebeat-Kafka-overview.json.Mx6KDFDydX11:Cross-device link
[2/2] Extracting beats7-7.6.2: 100%
Comment 28 ari 2020-05-23 06:56:42 UTC
I'd change the package comment to

Send logs, network, heartbeat and system data to elasticsearch or logstash

----


Also, data like dashboard for Kibana really should go to

/usr/local/share/beats/

rather than /var/db/. That's not the FreeBSD way.

----

In your filebeat.yml file you have

#path.logs: ${path.home}/logs

a more sane default might be

#path.logs: /var/logs/filebeat
Comment 29 Regis A. Despres 2020-05-23 14:48:18 UTC
Hi there,

Thanks for the review.
Comment and configs are updated here https://github.com/kalw/beats-fbsd-port/commit/6f722215b2dda957a8aee719924573195f838331
and there https://github.com/kalw/beats-fbsd-port/commit/2fe15a9648c7594149b380ef0001e00a9d6eb186
Still rebranding file paths.
I'll get back with whole bunch of changes and the associated shar in a few.


Regards,



--
Regis A. Despres
Comment 30 Greg Lewis freebsd_committer 2020-05-23 19:34:04 UTC
*** Bug 246444 has been marked as a duplicate of this bug. ***
Comment 31 ari 2020-05-24 00:04:19 UTC
While you were working on yours I did this:

https://github.com/ari/beats7

The commits start from your previous draft, so it should be easy enough to merge.
Comment 32 ari 2020-05-24 03:16:47 UTC
Time really flies when you are struggling with Makefiles. tabs, spaces, oh my...

Anyhow, I've done a lot more work on this now:

https://github.com/ari/beats7

* passes portlint
* builds, runs and pushes data to my elasticsearch on my system
* improved configuration setup

I have it running on a dozen FreeBSD machines right now.


My port fixes https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244627 and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228785
Comment 33 Regis A. Despres 2020-05-24 12:23:47 UTC
(In reply to ari from comment #32)
Hey Ari,

Thanks for your inputs !
That's a significant rebrand of the original beat =)
Btw, merged your changes referencing your contribution and added a few.
(that would be a lot easier with a PR :) )
waiting for the automated build  & poudriere run to attach the new shar.


Regards,


--
Regis A. Despres
Comment 34 Regis A. Despres 2020-05-28 07:44:51 UTC
Created attachment 214952 [details]
ari contribution included
Comment 35 Regis A. Despres 2020-06-03 10:15:11 UTC
(In reply to ari from comment #28)
(In reply to Kubilay Kocak from comment #25)

Hi there, 

Since last shar include remarks, advices and contribution, how guys want to move forward ?


Regards,



-- 
Regis A. Despres
Comment 36 ari 2020-06-04 01:19:29 UTC
I'm not sure what else is needed, but the github repo I created contains the exact files I've been using in production since 24 May without any issues. Do you need to make further adjustments?

If so, what needs to be fixed?
Comment 37 Regis A. Despres 2020-06-04 08:54:51 UTC
(In reply to ari from comment #36)

A few, like expanding your work to metricbeat for instance.
But don't get wrong, I don't want to take over anything I only wanted to leverage the CI in place there.
Goal is to move this subject forward and happily enjoy beat7 mainstream asap .. [where|whom]ever is it from =)


Regards,



--
Regis A. Despres
Comment 38 Regis A. Despres 2020-06-04 09:29:31 UTC
(In reply to ari from comment #36)


read a bit quick your reply, nothing on our side required, everything is living in the last shar =) 


-- 
Regis A. Despsres