Bug 244849

Created attachment 212449 [details]
shar to add beats7 - 7.6.1

Created attachment 212450 [details]
poudriere testport build log / plist complaints

The same issues are in current sysutils/beats port so I am not sure if it is intentional or not

Created attachment 212452 [details]
new port as patch / upgrade from repo copied beats to beats7

plist options fix

Created attachment 212453 [details]
shar to add beats7 - 7.6.1

plist options fix

Comment 5 Greg Lewis 2020-03-22 17:31:56 UTC

I think that it makes a lot of sense to first move the current beats port to beats6 before we commit a beats7.

Comment 6 Juraj Lutter 2020-03-22 17:34:48 UTC

What is the proper workflow to rename the port? I did quite a bit of a work with beats6 so I'd be interested in renaming it, too.

Can somebody more experienced with packaging take a look at thees issue?
https://lists.freebsd.org/pipermail/freebsd-elastic/2020-March/000563.html

The problem is that install process is trying to create hardlink to file which can be on dofferent partition (hardlink from /usr/local to /var/db) and it failed if those are separate filesystems. But it passed pouderiere testport where everything is on one filesystem.

Comment 8 Greg Lewis 2020-03-27 04:48:38 UTC

Renaming the port shouldn't be too difficult.  If someone wants to copy it over, make the necessary changes and post a diff or shar I can handle committing the actual changes.

Created attachment 212894 [details]
fix for packetbeat build

I've added attachment patch for fixing the build issue for packetbeat. The Makefile will need to be updated to remove "PACKETBEAT_BROKEN=     An underlying library is currently broken under FreeBSD" which disables it from being built. Tested and builds and runs without any issues. Currently using it in production enjoy!

Hello,

we tested the shar file and compilation fails. Here are the details:

# pwd
/usr/ports/sysutils/beats7
# make
===>  License APACHE20 accepted by the user
===>   beats7-7.6.1 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by beats7-7.6.1 for building
===>  Extracting for beats7-7.6.1
=> SHA256 Checksum OK for elastic-beats-v7.6.1_GH0.tar.gz.
===>  Patching for beats7-7.6.1
===>  Applying FreeBSD patches for beats7-7.6.1
===>   beats7-7.6.1 depends on executable: gmake - found
===>   beats7-7.6.1 depends on file: /usr/local/bin/go - found
===>  Configuring for beats7-7.6.1
===>  Building for beats7-7.6.1
gmake[2]: Entering directory '/usr/ports/usr/ports/sysutils/beats7/work/beats-7.6.1/filebeat'
go build 
# github.com/elastic/beats/vendor/github.com/DataDog/zstd
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:61:22: could not determine kind of name for C.ZSTD_compressBegin
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:64:22: could not determine kind of name for C.ZSTD_compressBegin_usingDict
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:94:13: could not determine kind of name for C.ZSTD_compressContinue
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:120:13: could not determine kind of name for C.ZSTD_compressEnd
gmake[2]: *** [../libbeat/scripts/Makefile:121: filebeat] Error 2
gmake[2]: Leaving directory '/usr/ports/usr/ports/sysutils/beats7/work/beats-7.6.1/filebeat'
*** Error code 2

Stop.
make[1]: stopped in /usr/ports/sysutils/beats7
*** Error code 1

Stop.
make: stopped in /usr/ports/sysutils/beats7

(In reply to Ryan Lawrence from comment #9)
File to patch seems not present anymore =)

Regards,



-- 
Regis A. Despres

Hi there,

Letting know your work has been put together in addition w/few things (plist fix, auditbeat addition, fixed packetbeat patch ..) into a repo linked to cirrus-ci for automated poudriere runs and test artefact disposal here : https://github.com/kalw/beats-fbsd-port/releases

Hope this help to get into mainstream soon :)


Regards,



-- 
Regis A. Despres

(In reply to Regis A. Despres from comment #12)
Hello,

thanks for your work. Still the same compilation problem:

root@ports12x:/usr/ports/sysutils/beats7 # make

===>  License APACHE20 accepted by the user
===>   beats7-7.6.2 depends on file: /usr/local/sbin/pkg - found
=> elastic-beats-v7.6.2_GH0.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch https://codeload.github.com/elastic/beats/tar.gz/v7.6.2?dummy=/elastic-beats-v7.6.2_GH0.tar.gz
fetch: https://codeload.github.com/elastic/beats/tar.gz/v7.6.2?dummy=/elastic-beats-v7.6.2_GH0.tar.gz: size unknown
fetch: https://codeload.github.com/elastic/beats/tar.gz/v7.6.2?dummy=/elastic-beats-v7.6.2_GH0.tar.gz: size of remote file is not known
elastic-beats-v7.6.2_GH0.tar.gz                         74 MB 6547 kBps    12s
===> Fetching all distfiles required by beats7-7.6.2 for building
===>  Extracting for beats7-7.6.2
=> SHA256 Checksum OK for elastic-beats-v7.6.2_GH0.tar.gz.
===>  Patching for beats7-7.6.2
===>  Applying FreeBSD patches for beats7-7.6.2
===>   beats7-7.6.2 depends on executable: gmake - found
===>   beats7-7.6.2 depends on file: /usr/local/bin/go - found
===>  Configuring for beats7-7.6.2
===>  Building for beats7-7.6.2
gmake[2]: Entering directory '/usr/ports/usr/ports/sysutils/beats7/work/beats-7.6.2/filebeat'
go build 
# github.com/elastic/beats/vendor/github.com/DataDog/zstd
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:61:22: could not determine kind of name for C.ZSTD_compressBegin
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:64:22: could not determine kind of name for C.ZSTD_compressBegin_usingDict
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:94:13: could not determine kind of name for C.ZSTD_compressContinue
../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:120:13: could not determine kind of name for C.ZSTD_compressEnd
gmake[2]: *** [../libbeat/scripts/Makefile:121: filebeat] Error 2
gmake[2]: Leaving directory '/usr/ports/usr/ports/sysutils/beats7/work/beats-7.6.2/filebeat'
*** Error code 2

Stop.
make[1]: stopped in /usr/ports/sysutils/beats7
*** Error code 1

Stop.
make: stopped in /usr/ports/sysutils/beats7

(In reply to Patrice from comment #13)

Hi Patrice,

Did you get into the `portsnap fetch update` step ?
While I don't have any solution yet for the usecase you report, you might leverage the automated build packages https://cirrus-ci.com/task/5367768021204992 if you are in 12x on amd64.


Regards,



-- 
Regis A. Despres

Thanks. Yes, portsnap update was run, still the same error.

We don't use poudriere for building our ports but portmaster. So the current port doesn't work with portmaster (nor with standard make in /usr/ports directories).

Best regards,

(In reply to Patrice from comment #15)
Could share your `uname -a`, `go version` and `gmake -v` ouputs in order to reproduce your case ?

CI matrix actually automatically build 12.1-RELEASE , 11.3-STABLE . 12.1-STABLE manually tested reports also ok , i.e. :

vagrant init freebsd/FreeBSD-12.0-CURRENT
vagrant up
vagrant ssh 
uname -a
    FreeBSD freebsd 12.1-STABLE FreeBSD 12.1-STABLE r359553 GENERIC  amd64
sudo portsnap fetch upgrade
sudo pkg install -y git
git clone https://github.com/kalw/beats-fbsd-port.git
sudo mv beats-fbsd-port/sysutils/beats7 /usr/ports/sysutils/beats7
sudo make -DBATCH=yes -C /usr/ports/sysutils/beats7/
..
====> Compressing man pages (compress-man)
===> Staging rc.d startup script(s)
sudo make -DBATCH=yes -C /usr/ports/sysutils/beats7/ install package
===>  Installing for beats7-7.6.2
===>  Checking if beats7 is already installed
===>   Registering installation for beats7-7.6.2
Installing beats7-7.6.2...
===>  Building package for beats7-7.6.2
% filebeat version
filebeat version 7.6.2 (amd64), libbeat 7.6.2 [unknown built unknown]
% metricbeat version
metricbeat version 7.6.2 (amd64), libbeat 7.6.2 [unknown built unknown]




--
Regis A. Despres

(In reply to Regis A. Despres from comment #16)

For the record

go version
go version go1.14.2 freebsd/amd64

gmake -v
GNU Make 4.2.1

(In reply to Regis A. Despres from comment #17)

Thanks, will try your vagrant stuff. Could you share the content of your vagrantfile?

Output for versions:

root@ports12x:~ # uname -a
FreeBSD ports12x 12.1-RELEASE-p2 FreeBSD 12.1-RELEASE-p2 GENERIC  amd64
root@ports12x:~ # go version
go version go1.14.2 freebsd/amd64
root@ports12x:~ # gmake -v
GNU Make 4.2.1
Built for amd64-portbld-freebsd12.1
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

(In reply to Patrice from comment #18)
`vagrant init something` create the vagrantfile you should not even opening it
Anyway mine is slightly modified like that:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vagrant.plugins = ["vagrant-disksize"]
config.vm.define "bsd-12.1-STABLE" do |bsd|
    bsd.vm.box = "freebsd/FreeBSD-12.1-STABLE"
    bsd.disksize.size = '50GB'
    bsd.ssh.shell = "sh"
    bsd.vm.provider :virtualbox do |v|
      v.customize ["modifyvm", :id, "--memory", 2048]
      v.customize ["modifyvm", :id, "--name", "bsd-12.1-STABLE"]
    end
  end
end

(In reply to Regis A. Despres from comment #19)

The FreeBSD-12.0-CURRENT from Vagrant fails at git install:

root@freebsd:/home/vagrant # pkg install -y git
ld-elf.so.1: Shared object "libssl.so.111" not found, required by "pkg"
root@freebsd:/home/vagrant # pkg upgrade
ld-elf.so.1: Shared object "libssl.so.111" not found, required by "pkg"

I am with trying your Vagrantfile now.

(In reply to Patrice from comment #20)

Ok, I found the issue. Having zstd-1.4.4_1 port installed before performing the make triggers it. Deinstalling zstd-1.4.4_1 makes the compilation works ok.

So, there is still someting weird somewhere.

Regards,

(In reply to Patrice from comment #21)

Cool I'll try to add the constraint


-- 
Regis A. Despres

added in makefile , I guess once version confirmed it will need to be reflected on zstd side.


Regards,


--
Regis A. Despres

Hi there,

How can we help to move this initiative forward ?


Regards,



-- 
Regis A. Despres

Comment 25 Kubilay Kocak 2020-05-19 11:08:29 UTC

As of today, it appears the following is needed to progress:

- A *single* patch (atached in this issue that has been QA'd (addressing any issues mentioned in comment 0 and comment 10 for example and pases portlint/poudriere at least)

- Feedback (and approval) from beats maintainer (elastic@) regarding upgrade process from beats 6 -> 7 (comment 5)

@Miroslav/Regis Could you integrate, if it hasn't been already, Ryans 'fix for packetbeat build' patch into your patch please (obsoleting the old one and Ryans during attachment)

^Triage: Since elastic@ maintainers the current beats port, assign this issue to them

Created attachment 214643 [details]
overall/latest changes summarized

Hi there,

Thanks for the prompt reply !
Attached the shar that match the last successful build w/poudriere on 11.3-RELEASE and 12.3-RELEASE.
Packetbeat patch included, version updated to 7.6.2, auditbeat added (in define option only), conflict with zstd added
Overall build and test process can be reviewed here https://github.com/kalw/beats-fbsd-port/blob/master/.cirrus.yml
Logs and artefacts can also be found here https://cirrus-ci.com/build/5651205462425600


Regards,



-- 
Regis A. Despres

Using that last patch 214643 I get this error:

[1/2] Deleting files for beats-6.8.7_1: 100%
[2/2] Installing beats7-7.6.2...
[2/2] Extracting beats7-7.6.2:  93%
pkg: Fail to create hardlink: /var/db/beats/filebeat/kibana/7/dashboard/.pkgtemp.Filebeat-Kafka-overview.json.Mx6KDFDydX11:Cross-device link
[2/2] Extracting beats7-7.6.2: 100%

I'd change the package comment to

Send logs, network, heartbeat and system data to elasticsearch or logstash

----


Also, data like dashboard for Kibana really should go to

/usr/local/share/beats/

rather than /var/db/. That's not the FreeBSD way.

----

In your filebeat.yml file you have

#path.logs: ${path.home}/logs

a more sane default might be

#path.logs: /var/logs/filebeat

Hi there,

Thanks for the review.
Comment and configs are updated here https://github.com/kalw/beats-fbsd-port/commit/6f722215b2dda957a8aee719924573195f838331
and there https://github.com/kalw/beats-fbsd-port/commit/2fe15a9648c7594149b380ef0001e00a9d6eb186
Still rebranding file paths.
I'll get back with whole bunch of changes and the associated shar in a few.


Regards,



--
Regis A. Despres

Comment 30 Greg Lewis 2020-05-23 19:34:04 UTC

*** Bug 246444 has been marked as a duplicate of this bug. ***

While you were working on yours I did this:

https://github.com/ari/beats7

The commits start from your previous draft, so it should be easy enough to merge.

Time really flies when you are struggling with Makefiles. tabs, spaces, oh my...

Anyhow, I've done a lot more work on this now:

https://github.com/ari/beats7

* passes portlint
* builds, runs and pushes data to my elasticsearch on my system
* improved configuration setup

I have it running on a dozen FreeBSD machines right now.


My port fixes https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244627 and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228785

(In reply to ari from comment #32)
Hey Ari,

Thanks for your inputs !
That's a significant rebrand of the original beat =)
Btw, merged your changes referencing your contribution and added a few.
(that would be a lot easier with a PR :) )
waiting for the automated build  & poudriere run to attach the new shar.


Regards,


--
Regis A. Despres

Created attachment 214952 [details]
ari contribution included

(In reply to ari from comment #28)
(In reply to Kubilay Kocak from comment #25)

Hi there, 

Since last shar include remarks, advices and contribution, how guys want to move forward ?


Regards,



-- 
Regis A. Despres

I'm not sure what else is needed, but the github repo I created contains the exact files I've been using in production since 24 May without any issues. Do you need to make further adjustments?

If so, what needs to be fixed?

(In reply to ari from comment #36)

A few, like expanding your work to metricbeat for instance.
But don't get wrong, I don't want to take over anything I only wanted to leverage the CI in place there.
Goal is to move this subject forward and happily enjoy beat7 mainstream asap .. [where|whom]ever is it from =)


Regards,



--
Regis A. Despres

(In reply to ari from comment #36)


read a bit quick your reply, nothing on our side required, everything is living in the last shar =) 


-- 
Regis A. Despsres

Comment 39 Greg Lewis 2020-06-07 18:13:28 UTC

I tried to unpack the latest shar, but it seems mangled towards the end of the patch-packetbeat_packetbeat.yml section and doesn't unpack subsequent files correctly.

Comment 40 commit-hook 2020-06-07 21:22:12 UTC

A commit references this bug:

Author: glewis
Date: Sun Jun  7 21:21:35 UTC 2020
New revision: 538190
URL: https://svnweb.freebsd.org/changeset/ports/538190

Log:
  Add a port for version 7.x of the ELK beats platform

  PR:		244849
  Submitted by:	Miroslav Lachman <000.fbsd@quip.cz>
  		Ryan Lawrence <bug@ryan-l.com>
  		Regis A. Despres <regis.despres@gmail.com>
  		Ari Maniatis <ari@ish.com.au>

Changes:
  head/sysutils/Makefile
  head/sysutils/beats7/
  head/sysutils/beats7/Makefile
  head/sysutils/beats7/distinfo
  head/sysutils/beats7/files/auditbeat.in
  head/sysutils/beats7/files/filebeat.in
  head/sysutils/beats7/files/heartbeat.in
  head/sysutils/beats7/files/metricbeat.in
  head/sysutils/beats7/files/packetbeat.in
  head/sysutils/beats7/files/patch-filebeat_filebeat.yml
  head/sysutils/beats7/files/patch-libbeat_scripts_Makefile
  head/sysutils/beats7/files/patch-metricbeat_metricbeat.yml
  head/sysutils/beats7/files/patch-packetbeat_packetbeat.yml
  head/sysutils/beats7/files/patch-vendor_github.com_docker_docker_pkg_system_mknod.go
  head/sysutils/beats7/files/patch-vendor_github.com_insomniacslk_dhcp_dhcpv4_bindtodevice_bsd.go
  head/sysutils/beats7/pkg-plist

Awesome Greg !
(Meanwhile I'll look into the auto-generated shar)


-- 
Regis A. Despres

Comment 42 Greg Lewis 2020-06-07 23:27:56 UTC

(In reply to Regis A. Despres from comment #41)

I think the main thing is just to check that what I committed looks like what you have (save for a few changes in the Makefile).

(In reply to Greg Lewis from comment #42)

Yep is is, except from the makefile diffs you've mentioned https://gist.github.com/kalw/edb1c9ec04c7927c9e19079088fd3874

Thanks again !



-- 
Regis A. Despres