Created attachment 212448 [details] new port as patch / upgrade from repo copied beats to beats7 As was asked by Greg Lewis here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237374 I am providing the first wild take on beats7. Again - as I am not using / running Kibana nor Beats I cannot properly test it. It was just about modifying few patches from beats 6 to match beats 7 and modify pkg-plist. Poudriere testport complaints about few empty dirs left in /var/db/beats but I tested current sysutils/beats and there are even more complaints about empty directories left after pkg delete so I am not sure if I should add those dirs to pkg-plist or not. ====> Checking for pkg-plist issues (check-plist) ===> Parsing plist ===> Checking for items in STAGEDIR missing from pkg-plist Error: Orphaned: @dir /var/db/beats/filebeat/kibana/7/dashboard Error: Orphaned: @dir /var/db/beats/filebeat/kibana/7 Error: Orphaned: @dir /var/db/beats/filebeat/kibana Error: Orphaned: @dir /var/db/beats/heartbeat/kibana Error: Orphaned: @dir /var/db/beats/metricbeat/kibana/7/dashboard Error: Orphaned: @dir /var/db/beats/metricbeat/kibana/7 Error: Orphaned: @dir /var/db/beats/metricbeat/kibana ===> Checking for items in pkg-plist which are not in STAGEDIR ===> Error: Plist issues found. *** Error code 1 Also the pkg-plist does not contain entries for packetbeat because this module is marked as broken and cannot be build. https://github.com/elastic/beats/issues/15256
Created attachment 212449 [details] shar to add beats7 - 7.6.1
Created attachment 212450 [details] poudriere testport build log / plist complaints The same issues are in current sysutils/beats port so I am not sure if it is intentional or not
Created attachment 212452 [details] new port as patch / upgrade from repo copied beats to beats7 plist options fix
Created attachment 212453 [details] shar to add beats7 - 7.6.1 plist options fix
I think that it makes a lot of sense to first move the current beats port to beats6 before we commit a beats7.
What is the proper workflow to rename the port? I did quite a bit of a work with beats6 so I'd be interested in renaming it, too.
Can somebody more experienced with packaging take a look at thees issue? https://lists.freebsd.org/pipermail/freebsd-elastic/2020-March/000563.html The problem is that install process is trying to create hardlink to file which can be on dofferent partition (hardlink from /usr/local to /var/db) and it failed if those are separate filesystems. But it passed pouderiere testport where everything is on one filesystem.
Renaming the port shouldn't be too difficult. If someone wants to copy it over, make the necessary changes and post a diff or shar I can handle committing the actual changes.
Created attachment 212894 [details] fix for packetbeat build I've added attachment patch for fixing the build issue for packetbeat. The Makefile will need to be updated to remove "PACKETBEAT_BROKEN= An underlying library is currently broken under FreeBSD" which disables it from being built. Tested and builds and runs without any issues. Currently using it in production enjoy!
Hello, we tested the shar file and compilation fails. Here are the details: # pwd /usr/ports/sysutils/beats7 # make ===> License APACHE20 accepted by the user ===> beats7-7.6.1 depends on file: /usr/local/sbin/pkg - found ===> Fetching all distfiles required by beats7-7.6.1 for building ===> Extracting for beats7-7.6.1 => SHA256 Checksum OK for elastic-beats-v7.6.1_GH0.tar.gz. ===> Patching for beats7-7.6.1 ===> Applying FreeBSD patches for beats7-7.6.1 ===> beats7-7.6.1 depends on executable: gmake - found ===> beats7-7.6.1 depends on file: /usr/local/bin/go - found ===> Configuring for beats7-7.6.1 ===> Building for beats7-7.6.1 gmake[2]: Entering directory '/usr/ports/usr/ports/sysutils/beats7/work/beats-7.6.1/filebeat' go build # github.com/elastic/beats/vendor/github.com/DataDog/zstd ../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:61:22: could not determine kind of name for C.ZSTD_compressBegin ../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:64:22: could not determine kind of name for C.ZSTD_compressBegin_usingDict ../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:94:13: could not determine kind of name for C.ZSTD_compressContinue ../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:120:13: could not determine kind of name for C.ZSTD_compressEnd gmake[2]: *** [../libbeat/scripts/Makefile:121: filebeat] Error 2 gmake[2]: Leaving directory '/usr/ports/usr/ports/sysutils/beats7/work/beats-7.6.1/filebeat' *** Error code 2 Stop. make[1]: stopped in /usr/ports/sysutils/beats7 *** Error code 1 Stop. make: stopped in /usr/ports/sysutils/beats7
(In reply to Ryan Lawrence from comment #9) File to patch seems not present anymore =) Regards, -- Regis A. Despres
Hi there, Letting know your work has been put together in addition w/few things (plist fix, auditbeat addition, fixed packetbeat patch ..) into a repo linked to cirrus-ci for automated poudriere runs and test artefact disposal here : https://github.com/kalw/beats-fbsd-port/releases Hope this help to get into mainstream soon :) Regards, -- Regis A. Despres
(In reply to Regis A. Despres from comment #12) Hello, thanks for your work. Still the same compilation problem: root@ports12x:/usr/ports/sysutils/beats7 # make ===> License APACHE20 accepted by the user ===> beats7-7.6.2 depends on file: /usr/local/sbin/pkg - found => elastic-beats-v7.6.2_GH0.tar.gz doesn't seem to exist in /usr/ports/distfiles/. => Attempting to fetch https://codeload.github.com/elastic/beats/tar.gz/v7.6.2?dummy=/elastic-beats-v7.6.2_GH0.tar.gz fetch: https://codeload.github.com/elastic/beats/tar.gz/v7.6.2?dummy=/elastic-beats-v7.6.2_GH0.tar.gz: size unknown fetch: https://codeload.github.com/elastic/beats/tar.gz/v7.6.2?dummy=/elastic-beats-v7.6.2_GH0.tar.gz: size of remote file is not known elastic-beats-v7.6.2_GH0.tar.gz 74 MB 6547 kBps 12s ===> Fetching all distfiles required by beats7-7.6.2 for building ===> Extracting for beats7-7.6.2 => SHA256 Checksum OK for elastic-beats-v7.6.2_GH0.tar.gz. ===> Patching for beats7-7.6.2 ===> Applying FreeBSD patches for beats7-7.6.2 ===> beats7-7.6.2 depends on executable: gmake - found ===> beats7-7.6.2 depends on file: /usr/local/bin/go - found ===> Configuring for beats7-7.6.2 ===> Building for beats7-7.6.2 gmake[2]: Entering directory '/usr/ports/usr/ports/sysutils/beats7/work/beats-7.6.2/filebeat' go build # github.com/elastic/beats/vendor/github.com/DataDog/zstd ../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:61:22: could not determine kind of name for C.ZSTD_compressBegin ../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:64:22: could not determine kind of name for C.ZSTD_compressBegin_usingDict ../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:94:13: could not determine kind of name for C.ZSTD_compressContinue ../../src/github.com/elastic/beats/vendor/github.com/DataDog/zstd/zstd_stream.go:120:13: could not determine kind of name for C.ZSTD_compressEnd gmake[2]: *** [../libbeat/scripts/Makefile:121: filebeat] Error 2 gmake[2]: Leaving directory '/usr/ports/usr/ports/sysutils/beats7/work/beats-7.6.2/filebeat' *** Error code 2 Stop. make[1]: stopped in /usr/ports/sysutils/beats7 *** Error code 1 Stop. make: stopped in /usr/ports/sysutils/beats7
(In reply to Patrice from comment #13) Hi Patrice, Did you get into the `portsnap fetch update` step ? While I don't have any solution yet for the usecase you report, you might leverage the automated build packages https://cirrus-ci.com/task/5367768021204992 if you are in 12x on amd64. Regards, -- Regis A. Despres
Thanks. Yes, portsnap update was run, still the same error. We don't use poudriere for building our ports but portmaster. So the current port doesn't work with portmaster (nor with standard make in /usr/ports directories). Best regards,
(In reply to Patrice from comment #15) Could share your `uname -a`, `go version` and `gmake -v` ouputs in order to reproduce your case ? CI matrix actually automatically build 12.1-RELEASE , 11.3-STABLE . 12.1-STABLE manually tested reports also ok , i.e. : vagrant init freebsd/FreeBSD-12.0-CURRENT vagrant up vagrant ssh uname -a FreeBSD freebsd 12.1-STABLE FreeBSD 12.1-STABLE r359553 GENERIC amd64 sudo portsnap fetch upgrade sudo pkg install -y git git clone https://github.com/kalw/beats-fbsd-port.git sudo mv beats-fbsd-port/sysutils/beats7 /usr/ports/sysutils/beats7 sudo make -DBATCH=yes -C /usr/ports/sysutils/beats7/ .. ====> Compressing man pages (compress-man) ===> Staging rc.d startup script(s) sudo make -DBATCH=yes -C /usr/ports/sysutils/beats7/ install package ===> Installing for beats7-7.6.2 ===> Checking if beats7 is already installed ===> Registering installation for beats7-7.6.2 Installing beats7-7.6.2... ===> Building package for beats7-7.6.2 % filebeat version filebeat version 7.6.2 (amd64), libbeat 7.6.2 [unknown built unknown] % metricbeat version metricbeat version 7.6.2 (amd64), libbeat 7.6.2 [unknown built unknown] -- Regis A. Despres
(In reply to Regis A. Despres from comment #16) For the record go version go version go1.14.2 freebsd/amd64 gmake -v GNU Make 4.2.1
(In reply to Regis A. Despres from comment #17) Thanks, will try your vagrant stuff. Could you share the content of your vagrantfile? Output for versions: root@ports12x:~ # uname -a FreeBSD ports12x 12.1-RELEASE-p2 FreeBSD 12.1-RELEASE-p2 GENERIC amd64 root@ports12x:~ # go version go version go1.14.2 freebsd/amd64 root@ports12x:~ # gmake -v GNU Make 4.2.1 Built for amd64-portbld-freebsd12.1 Copyright (C) 1988-2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
(In reply to Patrice from comment #18) `vagrant init something` create the vagrantfile you should not even opening it Anyway mine is slightly modified like that: # -*- mode: ruby -*- # vi: set ft=ruby : Vagrant.configure("2") do |config| config.vagrant.plugins = ["vagrant-disksize"] config.vm.define "bsd-12.1-STABLE" do |bsd| bsd.vm.box = "freebsd/FreeBSD-12.1-STABLE" bsd.disksize.size = '50GB' bsd.ssh.shell = "sh" bsd.vm.provider :virtualbox do |v| v.customize ["modifyvm", :id, "--memory", 2048] v.customize ["modifyvm", :id, "--name", "bsd-12.1-STABLE"] end end end
(In reply to Regis A. Despres from comment #19) The FreeBSD-12.0-CURRENT from Vagrant fails at git install: root@freebsd:/home/vagrant # pkg install -y git ld-elf.so.1: Shared object "libssl.so.111" not found, required by "pkg" root@freebsd:/home/vagrant # pkg upgrade ld-elf.so.1: Shared object "libssl.so.111" not found, required by "pkg" I am with trying your Vagrantfile now.
(In reply to Patrice from comment #20) Ok, I found the issue. Having zstd-1.4.4_1 port installed before performing the make triggers it. Deinstalling zstd-1.4.4_1 makes the compilation works ok. So, there is still someting weird somewhere. Regards,
(In reply to Patrice from comment #21) Cool I'll try to add the constraint -- Regis A. Despres
added in makefile , I guess once version confirmed it will need to be reflected on zstd side. Regards, -- Regis A. Despres
Hi there, How can we help to move this initiative forward ? Regards, -- Regis A. Despres
As of today, it appears the following is needed to progress: - A *single* patch (atached in this issue that has been QA'd (addressing any issues mentioned in comment 0 and comment 10 for example and pases portlint/poudriere at least) - Feedback (and approval) from beats maintainer (elastic@) regarding upgrade process from beats 6 -> 7 (comment 5) @Miroslav/Regis Could you integrate, if it hasn't been already, Ryans 'fix for packetbeat build' patch into your patch please (obsoleting the old one and Ryans during attachment) ^Triage: Since elastic@ maintainers the current beats port, assign this issue to them
Created attachment 214643 [details] overall/latest changes summarized Hi there, Thanks for the prompt reply ! Attached the shar that match the last successful build w/poudriere on 11.3-RELEASE and 12.3-RELEASE. Packetbeat patch included, version updated to 7.6.2, auditbeat added (in define option only), conflict with zstd added Overall build and test process can be reviewed here https://github.com/kalw/beats-fbsd-port/blob/master/.cirrus.yml Logs and artefacts can also be found here https://cirrus-ci.com/build/5651205462425600 Regards, -- Regis A. Despres
Using that last patch 214643 I get this error: [1/2] Deleting files for beats-6.8.7_1: 100% [2/2] Installing beats7-7.6.2... [2/2] Extracting beats7-7.6.2: 93% pkg: Fail to create hardlink: /var/db/beats/filebeat/kibana/7/dashboard/.pkgtemp.Filebeat-Kafka-overview.json.Mx6KDFDydX11:Cross-device link [2/2] Extracting beats7-7.6.2: 100%
I'd change the package comment to Send logs, network, heartbeat and system data to elasticsearch or logstash ---- Also, data like dashboard for Kibana really should go to /usr/local/share/beats/ rather than /var/db/. That's not the FreeBSD way. ---- In your filebeat.yml file you have #path.logs: ${path.home}/logs a more sane default might be #path.logs: /var/logs/filebeat
Hi there, Thanks for the review. Comment and configs are updated here https://github.com/kalw/beats-fbsd-port/commit/6f722215b2dda957a8aee719924573195f838331 and there https://github.com/kalw/beats-fbsd-port/commit/2fe15a9648c7594149b380ef0001e00a9d6eb186 Still rebranding file paths. I'll get back with whole bunch of changes and the associated shar in a few. Regards, -- Regis A. Despres
*** Bug 246444 has been marked as a duplicate of this bug. ***
While you were working on yours I did this: https://github.com/ari/beats7 The commits start from your previous draft, so it should be easy enough to merge.
Time really flies when you are struggling with Makefiles. tabs, spaces, oh my... Anyhow, I've done a lot more work on this now: https://github.com/ari/beats7 * passes portlint * builds, runs and pushes data to my elasticsearch on my system * improved configuration setup I have it running on a dozen FreeBSD machines right now. My port fixes https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244627 and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228785
(In reply to ari from comment #32) Hey Ari, Thanks for your inputs ! That's a significant rebrand of the original beat =) Btw, merged your changes referencing your contribution and added a few. (that would be a lot easier with a PR :) ) waiting for the automated build & poudriere run to attach the new shar. Regards, -- Regis A. Despres
Created attachment 214952 [details] ari contribution included
(In reply to ari from comment #28) (In reply to Kubilay Kocak from comment #25) Hi there, Since last shar include remarks, advices and contribution, how guys want to move forward ? Regards, -- Regis A. Despres
I'm not sure what else is needed, but the github repo I created contains the exact files I've been using in production since 24 May without any issues. Do you need to make further adjustments? If so, what needs to be fixed?
(In reply to ari from comment #36) A few, like expanding your work to metricbeat for instance. But don't get wrong, I don't want to take over anything I only wanted to leverage the CI in place there. Goal is to move this subject forward and happily enjoy beat7 mainstream asap .. [where|whom]ever is it from =) Regards, -- Regis A. Despres
(In reply to ari from comment #36) read a bit quick your reply, nothing on our side required, everything is living in the last shar =) -- Regis A. Despsres
I tried to unpack the latest shar, but it seems mangled towards the end of the patch-packetbeat_packetbeat.yml section and doesn't unpack subsequent files correctly.
A commit references this bug: Author: glewis Date: Sun Jun 7 21:21:35 UTC 2020 New revision: 538190 URL: https://svnweb.freebsd.org/changeset/ports/538190 Log: Add a port for version 7.x of the ELK beats platform PR: 244849 Submitted by: Miroslav Lachman <000.fbsd@quip.cz> Ryan Lawrence <bug@ryan-l.com> Regis A. Despres <regis.despres@gmail.com> Ari Maniatis <ari@ish.com.au> Changes: head/sysutils/Makefile head/sysutils/beats7/ head/sysutils/beats7/Makefile head/sysutils/beats7/distinfo head/sysutils/beats7/files/auditbeat.in head/sysutils/beats7/files/filebeat.in head/sysutils/beats7/files/heartbeat.in head/sysutils/beats7/files/metricbeat.in head/sysutils/beats7/files/packetbeat.in head/sysutils/beats7/files/patch-filebeat_filebeat.yml head/sysutils/beats7/files/patch-libbeat_scripts_Makefile head/sysutils/beats7/files/patch-metricbeat_metricbeat.yml head/sysutils/beats7/files/patch-packetbeat_packetbeat.yml head/sysutils/beats7/files/patch-vendor_github.com_docker_docker_pkg_system_mknod.go head/sysutils/beats7/files/patch-vendor_github.com_insomniacslk_dhcp_dhcpv4_bindtodevice_bsd.go head/sysutils/beats7/pkg-plist
Awesome Greg ! (Meanwhile I'll look into the auto-generated shar) -- Regis A. Despres
(In reply to Regis A. Despres from comment #41) I think the main thing is just to check that what I committed looks like what you have (save for a few changes in the Makefile).
(In reply to Greg Lewis from comment #42) Yep is is, except from the makefile diffs you've mentioned https://gist.github.com/kalw/edb1c9ec04c7927c9e19079088fd3874 Thanks again ! -- Regis A. Despres