Bug 206022 - net-mgmt/icinga2: after upgrade starting gets stuck and blocks
Summary: net-mgmt/icinga2: after upgrade starting gets stuck and blocks
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Lars Engels
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-01-08 06:45 UTC by O. Hartmann
Modified: 2016-04-24 17:04 UTC (History)
2 users (show)

See Also:
bugzilla: maintainer-feedback? (lme)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description O. Hartmann 2016-01-08 06:45:45 UTC
After updating net-mgmt/icinga2 to 2.4.1 recently on CURRENT (FreeBSD 11.0-CURRENT #1 r293413: Fri Jan  8 05:54:24 CET 2016 amd64), autostart at boottime will end up in blocking host from becoming ready due to icinga2 blocks: 

[...]
Performing sanity check of icinga2 configuration: information/cli: Icinga application loader (version: r2.4.1-1)

In this state it remains forever until terminated.
Comment 1 Lars Engels freebsd_committer 2016-01-08 07:22:01 UTC
Thanks for the report. I don't see this on my machine (also running HEAD).
Could you try to find out what Icinga2 is doing then?
Comment 2 O. Hartmann 2016-01-21 07:16:34 UTC
After testing and probing and deleting everything locate in /var telated to icinga2, I was able to restart icinga2 again - until this morning! After an update of HEAD to most recent (FreeBSD 11.0-CURRENT #13 r294484: Thu Jan 21 05:59:25 CET 2016 amd64) I got stuck again (the system doesn't boot, gets stuck in starting icinga2 and so the whole server is blocked booting up).

I configure icinga2 down to have only two module enabled:

Enabled features: command ido-pgsql

but the result was the very same. Now I have these modules running:

Enabled features: checker command ido-pgsql livestatus mainlog notification opentsdb syslog


After a while being stuck, I receive this error:
[...]
[2016-01-21 08:03:22 +0100] information/ConfigItem: Activated all objects.
[2016-01-21 08:03:23 +0100] information/IdoPgsqlConnection: pgSQL IDO instance id: 1 (schema version: '1.14.0')
[2016-01-21 08:04:37 +0100] critical/TcpSocket: Invalid socket: Operation timed out

The database is up and running (postgresql94 and now postgresql95, both the same problems, the table-layout for the tables is up to date).

Disabling ALL icinga2 modules (which is a kind of rediculous) ends up in this:

[...]
[2016-01-21 08:14:22 +0100] information/ConfigItem: Instantiated 152 CheckCommands.
[2016-01-21 08:14:22 +0100] information/ConfigItem: Instantiated 2 NotificationCommands.
[2016-01-21 08:14:22 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2016-01-21 08:14:22 +0100] information/ConfigItem: Triggering Start signal for config items
[2016-01-21 08:14:22 +0100] information/ConfigItem: Activated all objects.

... stuck.
Comment 3 O. Hartmann 2016-01-21 14:11:57 UTC
A more funny observation is:

I started Icinga2 manually 9since it blocks automatically started vie /etc/rc.conflocal the boot process).

When starting, hitting Ctrl-C and arrow-up in shell to recall last command very quickly in a frequent matter makes the process one time starting - somehow. I can not reproduce this in any cases. In another case, starting the icinga2 process via "service icinga2 onestart" worked - out of the sudden for one time!

this is a weird and definitely not "normal" behaviour ...
Comment 4 Matthias Fechner freebsd_committer 2016-04-09 08:39:46 UTC
I have exactly the same problem.
The extrem delay is caused by the startup script in /usr/local/etc/rc.d/icinga2.


start_precmd() {
  # Create necessary directories / change ownership
  #
...
        chown ${icinga2_user}:${icinga2_group} "${d}"
      fi
  done

  install -d -o ${icinga2_user} -g www "${icinga2_rundir}/cmd"

  chown -R ${icinga2_user}:${icinga2_user} "${icinga2_libdir}"
  chown -R ${icinga2_user}:${icinga2_user} "${icinga2_spooldir}"

This 3 chown commands are blocking services like apache24 for more than 10 minutes to be started.

I have no idea why it is necessary to have here chown commands in, this should only be necessary at installation.
If a buggy installation requires this at startup, please make it configurable by a parameter that is by default disabled.

Thanks a lot.
Comment 5 Lars Engels freebsd_committer 2016-04-22 14:31:24 UTC
The parameters were added because some people have /var/icinga2 running on a tmpfs and need to create the directories on startup.
I'll take a look at how to make this optional.
Comment 6 commit-hook freebsd_committer 2016-04-24 16:59:37 UTC
A commit references this bug:

Author: lme
Date: Sun Apr 24 16:58:58 UTC 2016
New revision: 413950
URL: https://svnweb.freebsd.org/changeset/ports/413950

Log:
  - Update to 2.4.7
  - Introduce new rc.conf variable "icinga2_mkvar" defaulting to "NO". The
    variable controls the creation and chown'ing of missing files and
    directories in /var. [1]
  - Add an UPDATING entry for this

  PR:		206022 [1]
  Submitted by:	Oliver Hartmann <ohartman@zedat.fu-berlin.de>
  Sponsored by:	Essen Linuxhotel Hackathon 2016

Changes:
  head/UPDATING
  head/net-mgmt/icinga2/Makefile
  head/net-mgmt/icinga2/distinfo
  head/net-mgmt/icinga2/files/icinga2.in
Comment 7 Lars Engels freebsd_committer 2016-04-24 17:03:54 UTC
This is now fixed in the new version of the icinga2 port.

Thanks for reporting this.