Bug 217081

Summary: sysutils/beats port creates logs in /usr/local/sbin/
Product: Ports & Packages Reporter: Dmitry Sivachenko <demon>
Component: Individual Port(s)Assignee: Palle Girgensohn <girgen>
Status: Closed DUPLICATE    
Severity: Affects Only Me CC: w.schwarzenfeld
Priority: --- Flags: bugzilla: maintainer-feedback? (girgen)
Version: Latest   
Hardware: Any   
OS: Any   

Description Dmitry Sivachenko freebsd_committer 2017-02-13 18:52:37 UTC
Hello,

I installed filebeat port version 5.2.0.

After startup, it created /usr/local/sbin/logs/ directory (and writes log file there) and /usr/local/sbin/data directory (with registry file inside).

Probably startup script should provide sane defaults for logs and data directories to respect hier(7).

Also, there is no reason for this daemon to run as root, so I'd execute it under "nobody" user.
Comment 1 Palle Girgensohn freebsd_committer 2017-02-14 10:09:07 UTC
Hi!

Yeah, thanks for noticing this about the log files. I had some config that hid this problems.

About running as non-root, filebeat in its standard setup needs read access to log files that are not accessible as non-root, so running as root should be an optional, IMO.

Palle
Comment 2 Dmitry Sivachenko freebsd_committer 2017-02-14 10:13:33 UTC
Okay, we can make running as root optional (may be use daemon's -u option?)

Consider this:
command_args="-rP ${pidfile} -u nobody /usr/local/sbin/${name} -path.home /usr/local -path.config /usr/local/etc -path.data /var/db/filebeat -path.logs /var/log/filebeat"

(and also create relevant directories from pkg-plist).

(we can override this hardcoded "nobody").

Also, Debian's package provides shell wrapper to facilitate one-shot executions of filebeat with proper parameters:

# cat /usr/bin/filebeat.sh 
#!/bin/bash

# Script to run filebeat in foreground with the same path settings that
# the init script / systemd unit file would do.

/usr/share/filebeat/bin/filebeat \
  -path.home /usr/share/filebeat \
  -path.config /etc/filebeat \
  -path.data /var/lib/filebeat \
  -path.logs /var/log/filebeat \
  $@


It would be convenient to provide the same one :)
Comment 3 Palle Girgensohn freebsd_committer 2017-02-14 10:28:34 UTC
(In reply to Dmitry Sivachenko from comment #2)

Yes, that is very similar to what I've just configured in my test environment. I set it up with syslog instead of file logging, since that makes it a bit simpler. Dunno which is best, really.

An extra filebeat.sh file is not really customary in FreeBSD's hier(7). perhaps it would be better to patch the go binary to set up proper defaults? Or would that confuse users more than it helps, what do you think?
Comment 4 Dmitry Sivachenko freebsd_committer 2017-02-14 10:30:20 UTC
I vote for patching go binary in order to ship it with proper defaults suitable for FreeBSD.  And do not implement shell wrapper.
Comment 5 Dmitry Sivachenko freebsd_committer 2017-02-15 13:06:56 UTC
Also, something is wrong with terminal handling:

% ssh root@smetanka5b '/usr/local/etc/rc.d/filebeat onestart'
Starting filebeat.
<hung here>


It is solved by adding "> /dev/null" at the end of command_args=""
Comment 6 Palle Girgensohn freebsd_committer 2017-02-19 11:21:18 UTC
(In reply to Dmitry Sivachenko from comment #5)

This usually happens if only stderr is redirected and not stdout (or the opposite). Was stderr redirected?
Comment 7 Palle Girgensohn freebsd_committer 2017-02-19 11:30:31 UTC
There is a new beat binary as well, heartbeat. Its name clashes with another port sysutils/heartbeat, and while the elastic heartbeat is arguably better placed in net-mgmt/, it would still be confusing. I can either prefix it with "elastic-", but then all beats ports should have this prefix, or I'll just refactor then into one single port "beats". The single port could have options to opt out unwanted beats, but would by default build all of them.

There is also some common files, i.e. a migrate script and a binary "import_dasboards", that elastic themselves package verbatim copies for all for beats packages. With separate ports, this would require yet another package libbeat that all four other *beat packages would depend on. 

Does anyone following this thread have a preference between the proposed strategies? :)
Comment 8 Dmitry Sivachenko freebsd_committer 2017-02-19 12:12:41 UTC
With that said, a single port with OPTIONS for enabling the desired *beats sounds reasonable for me.  I would not rename beats to include elastic- prefix, i think CONFLICT would be enough, the probability for people to use both ports is rather low?
Comment 9 Palle Girgensohn freebsd_committer 2017-02-19 12:24:53 UTC
(In reply to Dmitry Sivachenko from comment #8)
With a single port, CONFLICT is not necessary. It does onöy conflict WRT the port name, there is no clash of installed files.
Comment 10 commit-hook freebsd_committer 2017-02-21 01:28:01 UTC
A commit references this bug:

Author: girgen
Date: Tue Feb 21 01:27:09 UTC 2017
New revision: 434496
URL: https://svnweb.freebsd.org/changeset/ports/434496

Log:
  Update to 5.2.1

  The three elasticsearch beats ports have merged into one single port
  systuils/beats that installs all of the above and some more. The reason is to
  make the port more maintainable and also that the new heartbeat beat would
  conflict with sysutils/heartbeat.

  Add heartbeat beat.

  Install utilities
    share/beats/import_dashboards
    share/beats/migrate_beat_config_1_x_to_5_0.py

  Fix problems with the default configurations [1]

  PR:	217081 [1]

Changes:
  head/MOVED
  head/UPDATING
  head/sysutils/Makefile
  head/sysutils/beats/
  head/sysutils/beats/Makefile
  head/sysutils/beats/distinfo
  head/sysutils/beats/files/
  head/sysutils/beats/files/filebeat.in
  head/sysutils/beats/files/heartbeat.in
  head/sysutils/beats/files/metricbeat.in
  head/sysutils/beats/files/packetbeat.in
  head/sysutils/beats/files/patch-filebeat_filebeat.yml
  head/sysutils/beats/files/patch-gopsutil
  head/sysutils/beats/files/patch-heartbeat_heartbeat.yml
  head/sysutils/beats/files/patch-libbeat_scripts_Makefile
  head/sysutils/beats/files/patch-metricbeat_metricbeat.yml
  head/sysutils/beats/files/patch-packetbeat_packetbeat.yml
  head/sysutils/beats/pkg-descr
  head/sysutils/beats/pkg-plist
  head/sysutils/filebeat/
  head/sysutils/metricbeat/
  head/sysutils/packetbeat/
Comment 11 Palle Girgensohn freebsd_committer 2017-02-21 01:33:08 UTC
(In reply to Dmitry Sivachenko from comment #5)
This should only happen if there is a bug in the configuration so the program has to produce output to the terminal. You can also get this problem if not all of stderr and stdout are equally redirected.

I made the port use syslog by default, so you should not se this once config is alright.
Comment 12 Palle Girgensohn freebsd_committer 2017-02-21 01:35:21 UTC
(In reply to Dmitry Sivachenko from comment #4)
In the end, I removed the patches to adhere to hier(7) since that would probly confuse more than it would help, since all elastic's docs expect it to be unpatched. The rc-scripts now use -path.xxx properly, this should suffice.
Comment 13 Dmitry Sivachenko freebsd_committer 2017-02-21 04:18:24 UTC
Thanks!

What about the ability to run as non-root user?
Comment 14 Palle Girgensohn freebsd_committer 2017-03-03 15:39:27 UTC
(In reply to Dmitry Sivachenko from comment #13)

With the default config, reading log files from /var/log requires filebeat to run as root. 

# find /var/log ! -perm +o+r ! -name '*.bz2' | wc -l 
      42
# 

That makes me a bit reluctant to add a knob for this. Please feel free to argue that I'm wrong though... :)
Comment 15 Dmitry Sivachenko freebsd_committer 2017-03-03 16:50:31 UTC
Well, nobody force you to collect system log files from /var/log.

I use filebeat to send nginx/haproxy logs to ElasticSearch cluster.
I need nothing from /var/log.

I think there are a lot of use cases when people would want to collect their application log files, not system logs from /var/log.
Comment 16 Dmitry Sivachenko freebsd_committer 2017-03-03 16:51:20 UTC
Also, you can make "root" a default user, just allow to easily custimise it via rc.conf variable.
Comment 17 Palle Girgensohn freebsd_committer 2017-03-03 17:21:38 UTC
(In reply to Dmitry Sivachenko from comment #15)

OK. cool. I was easily persuaded. I'll add the knob. :)
Comment 18 Mark Linimon freebsd_committer freebsd_triage 2017-07-19 11:35:31 UTC
Port was renamed.

Is this PR still valid?
Comment 19 Walter Schwarzenfeld freebsd_triage 2018-02-12 14:52:50 UTC
ping!
Comment 20 Palle Girgensohn freebsd_committer 2018-02-12 15:03:07 UTC
I seem to have forgotten about this. The logging is fixed, but not running as user != root.

So yes, still in progress, at least partly.
Comment 21 Palle Girgensohn freebsd_committer 2018-05-26 20:35:33 UTC
I'm closing this PR and open a separate one for the issue of *Beats daemons running as root.

*** This bug has been marked as a duplicate of bug 228519 ***