Bug 220627 - net-im/ejabberd: Fails to start (inet_dist_listen_min 4200 : not found) after 17.04 -> 17.07 update
Summary: net-im/ejabberd: Fails to start (inet_dist_listen_min 4200 : not found) after...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Ashish SHUKLA
URL:
Keywords: needs-qa, regression
Depends on:
Blocks:
 
Reported: 2017-07-11 13:35 UTC by Marko Cupać
Modified: 2017-07-19 12:46 UTC (History)
1 user (show)

See Also:
bugzilla: maintainer-feedback? (ashish)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Marko Cupać 2017-07-11 13:35:48 UTC
Hi,

I've just upgraded ejabberd from 17.04 to 17.07, but it won't start.

Here's error message I get when trying to start it:

pacija@xmpp1:/var/log/ejabberd % sudo service ejabberd start
Starting ejabberd.
/usr/local/sbin/ejabberdctl: inet_dist_listen_min 4200 : not found

Checking status gives additional message:

pacija@xmpp1:/var/log/ejabberd % sudo service ejabberd status
/usr/local/sbin/ejabberdctl: inet_dist_listen_min 4200 : not found
Protocol 'inet_tcp': register/listen error: ejabberd is not running.

I guess it shouldn't make any difference (previous version worked) that this runs in a jail on a separate fib, but I'm mentioning it here anyway - perhaps it makes some difference after all.
Comment 1 Ashish SHUKLA freebsd_committer 2017-07-11 18:02:03 UTC
Hi,

Sorry about the issue.

Could you provide the output of: sudo -u ejabberd -H /bin/sh -x /usr/local/sbin/ejabberdctl start ?

Also, could you provide your `ejabberdctl.cfg' ? Since, `FIREWALL_WINDOW' (which i am guessing is maybe what is triggering this) is not enabled by default, the code path was not reached in default testing.

Thanks!
Comment 2 neil 2017-07-12 18:16:21 UTC
I had erroneously set the INET_DIST_INTERFACE variable in ejabberdctl.cfg to a non-default value (I am not running a cluster and there is no need to do this).

Reverting this change to what is in the ejabberdctl.cfg.example file allowed ejabberd to start correctly.
Comment 3 Marko Cupać 2017-07-13 09:32:29 UTC
(In reply to Ashish SHUKLA from comment #1)

Here's output of command you asked. ejabberdctl.cfg is below the output

pacija@xmpp1:~ % sudo -u ejabberd -H /bin/sh -x /usr/local/sbin/ejabberdctl start
+ POLL=true
+ SMP=auto
+ ERL_MAX_PORTS=32000
+ ERL_PROCESSES=250000
+ ERL_MAX_ETS_TABLES=1400
+ FIREWALL_WINDOW=''
+ ERLANG_NODE=ejabberd@localhost
+ ERL=/usr/local/bin/erl
+ IEX=/usr/local/bin/iex
+ EPMD=/usr/local/bin/epmd
+ INSTALLUSER=''
+ EXEC_CMD=false
+ [ -n '' ]
+ EXEC_CMD=as_current_user
+ [ as_current_user '=' false ]
+ break
+ : /usr/local/etc/ejabberd
+ : /var/log/ejabberd
+ : /var/spool/ejabberd
+ : /usr/local/etc/ejabberd/ejabberd.yml
+ : /usr/local/etc/ejabberd/ejabberdctl.cfg
+ [ -f /usr/local/etc/ejabberd/ejabberdctl.cfg ]
+ . /usr/local/etc/ejabberd/ejabberdctl.cfg
+ POLL=true
+ SMP=auto
+ ERL_MAX_PORTS=32000
+ FIREWALL_WINDOW=4200-4210
+ INET_DIST_INTERFACE=193.53.106.129
+ ERL_EPMD_ADDRESS=193.53.106.129
+ ERL_PROCESSES=250000
+ ERL_MAX_ETS_TABLES=1400
+ ERL_OPTIONS=''
+ ERLANG_NODE=ejabberd@xmpp1.mimar.rs
+ EJABBERD_PID_PATH=/var/run/ejabberdctl/ejabberd.pid
+ [ '' '!=' '' ]
+ [ ejabberd@xmpp1.mimar.rs '=' ejabberd@xmpp1.mimar ]
+ : /usr/local/share/doc/ejabberd
+ : /var/log/ejabberd/ejabberd.log
+ ERLANG_OPTS='+K true -smp auto +P 250000 '
+ [ 4200-4210 '!=' '' ]
+ ERLANG_OPTS='+K true -smp auto +P 250000  -kernel ' 'inet_dist_listen_min 4200 ' 'inet_dist_listen_max 4210'
/usr/local/sbin/ejabberdctl: inet_dist_listen_min 4200 : not found
+ [ 193.53.106.129 '!=' '' ]
+ /usr/local/bin/erl -noshell -eval 'case inet:parse_address("193.53.106.129") of {ok,IP} -> io:format("~p",[IP]); _ -> ok end.' -s erlang halt
+ INET_DIST_INTERFACE2={193,53,106,129}
+ [ {193,53,106,129} '!=' '' ]
+ ERLANG_OPTS='+K true -smp auto +P 250000  -kernel inet_dist_use_interface "{193,53,106,129}"'
+ ERL_LIBS=/usr/local/lib/erlang/lib/ejabberd-17.07
+ ERL_LIBS=/usr/local/lib/erlang/lib/ejabberd-17.07:/usr/local/lib/erlang/lib/ejabberd-17.07/lib
+ date +%Y%m%d-%H%M%S
+ ERL_CRASH_DUMP=/var/log/ejabberd/erl_crash_20170713-112457.dump
+ ERL_INETRC=/usr/local/etc/ejabberd/inetrc
+ sed '/^log_rate_limit/!d;s/:[ \t]*\([0-9]*\).*/ \1/;s/^/ /' /usr/local/etc/ejabberd/ejabberd.yml
+ sed '/^log_rotate_size/!d;s/:[ \t]*\([0-9]*\).*/ \1/;s/^/ /' /usr/local/etc/ejabberd/ejabberd.yml
+ sed '/^log_rotate_count/!d;s/:[ \t]*\([0-9]*\).*/ \1/;s/^/ /' /usr/local/etc/ejabberd/ejabberd.yml
+ sed '/^log_rotate_date/!d;s/:[ \t]*\(.[^ ]*\).*/ \1/;s/^/ /' /usr/local/etc/ejabberd/ejabberd.yml
+ EJABBERD_OPTS=' log_rate_limit 100 log_rotate_size 0 log_rotate_count 7 log_rotate_date "$D0"'
+ [ -n ' log_rate_limit 100 log_rotate_size 0 log_rotate_count 7 log_rotate_date "$D0"' ]
+ EJABBERD_OPTS='-ejabberd  log_rate_limit 100 log_rotate_size 0 log_rotate_count 7 log_rotate_date "$D0"'
+ EJABBERD_OPTS='-mnesia dir "/var/spool/ejabberd"  -ejabberd  log_rate_limit 100 log_rotate_size 0 log_rotate_count 7 log_rotate_date "$D0" -s ejabberd'
+ export EJABBERD_CONFIG_PATH
+ export EJABBERD_LOG_PATH
+ export EJABBERD_DOC_PATH
+ export EJABBERD_PID_PATH
+ export ERL_CRASH_DUMP
+ export ERL_EPMD_ADDRESS
+ export ERL_INETRC
+ export ERL_MAX_PORTS
+ export ERL_MAX_ETS_TABLES
+ export CONTRIB_MODULES_PATH
+ export CONTRIB_MODULES_CONF_DIR
+ export ERL_LIBS
+ check_start
+ /usr/local/bin/epmd -names
+ grep -q ' ejabberd '
+ [ -d /var/spool/ejabberd ]
+ cd /var/spool/ejabberd
+ exec_erl ejabberd@xmpp1.mimar.rs -mnesia dir '"/var/spool/ejabberd"' -ejabberd log_rate_limit 100 log_rotate_size 0 log_rotate_count 7 log_rotate_date '"$D0"' -s ejabberd -noinput -detached
+ NODE=ejabberd@xmpp1.mimar.rs
+ shift
+ exec_cmd /usr/local/bin/erl -name ejabberd@xmpp1.mimar.rs +K true -smp auto +P 250000 -kernel inet_dist_use_interface '"{193,53,106,129}"' -mnesia dir '"/var/spool/ejabberd"' -ejabberd log_rate_limit 100 log_rotate_size 0 log_rotate_count 7 log_rotate_date '"$D0"' -s ejabberd -noinput -detached
+ /usr/local/bin/erl -name ejabberd@xmpp1.mimar.rs +K true -smp auto +P 250000 -kernel inet_dist_use_interface '"{193,53,106,129}"' -mnesia dir '"/var/spool/ejabberd"' -ejabberd log_rate_limit 100 log_rotate_size 0 log_rotate_count 7 log_rotate_date '"$D0"' -s ejabberd -noinput -detached


ejabberdctl.cfg:
POLL=true
SMP=auto
ERL_MAX_PORTS=32000
FIREWALL_WINDOW=4200-4210
INET_DIST_INTERFACE=193.53.106.129
ERL_EPMD_ADDRESS=193.53.106.129
ERL_PROCESSES=250000
ERL_MAX_ETS_TABLES=1400
ERL_OPTIONS=""
ERLANG_NODE=ejabberd@xmpp1.mimar.rs
EJABBERD_PID_PATH=/var/run/ejabberdctl/ejabberd.pid

Thanks for your help.
Comment 4 Marko Cupać 2017-07-14 09:22:33 UTC
According to Neil's instruction, I commented out the following from ejabberdctl.cfg, after which ejabberd starts:

#FIREWALL_WINDOW=4200-4210
#INET_DIST_INTERFACE=193.53.106.129

However, this appears to open server on port 49449 that binds to all addresses:

pacija@xmpp1:~ % sudo netstat -an | grep LISTEN
tcp4       0      0 193.53.106.129.7777    *.*                    LISTEN
tcp4       0      0 193.53.106.129.5349    *.*                    LISTEN
tcp4       0      0 193.53.106.129.5280    *.*                    LISTEN
tcp4       0      0 193.53.106.129.5269    *.*                    LISTEN
tcp4       0      0 193.53.106.129.5222    *.*                    LISTEN
tcp4       0      0 *.49449                *.*                    LISTEN
...

I should be able to make it bind to specific address.
Comment 5 neil 2017-07-14 12:01:48 UTC
I am a little confused about the INET_DIST_INTERFACE variable.

The description suggests that it is an IP while the naming suggests an interface name. Probably a bad choice of variable name.

The default for INET_DIST_INTERFACE is 0.0.0.0 which means listen on all interfaces. This would be no different to what ejabberd does with its services. I see entries for e.g. *.5222 so I don't see a problem there.

I am wondering if there is a problem with INET_DIST_INTERFACE when it is set to something that is not the default?
Comment 6 commit-hook freebsd_committer 2017-07-15 12:16:49 UTC
A commit references this bug:

Author: ashish
Date: Sat Jul 15 12:16:25 UTC 2017
New revision: 445871
URL: https://svnweb.freebsd.org/changeset/ports/445871

Log:
  - Import diff from ejabberd repository, commitid: d4cdc3a
  - Add fix to make ejabberdctl work, when INET_DIST_INTERFACE is
    defined. Submitted a pull-request upstream.

  PR:		220627

Changes:
  head/net-im/ejabberd/Makefile
  head/net-im/ejabberd/files/patch-ejabberdctl.template
Comment 7 Ashish SHUKLA freebsd_committer 2017-07-15 12:18:40 UTC
Hi neil@ and Marko,

I have committed a fix. Could you folks please try and see if it resolves issue for you ?

I'm not able to reproduce your issue anymore on my end.

Thanks!
Comment 8 neil 2017-07-16 20:22:30 UTC
I have 3 test cases producing different results:

1) INET_DIST_INTERFACE commented.

ejabberdctl starts beam.smp and epmd correctly at ejabberd start
ejabberdctl stops beam.smp and leaves empd running at ejabberd stop

2) INET_DIST_INTERFACE = 127.0.0.1

ejabberdctl starts beam.smp and epmd correctly at ejabberd start
ejabberdctl fails to stop beam.smp (reporting ejabberd not running) and leaves epmd running at ejabberd stop

3) INET_DIST_INTERFACE = 192.168.1.2 (my VPS net interface)

ejabberdctl sends error timestamps to the console and beam.smp is not started but epmd is started at ejabberd start. A crash dump is produced.

So it appears to only work correctly when INET_DIST_INTERFACE is commented.

epmd being left running is a long-standing artifact which we have discussed before.
Comment 9 Ashish SHUKLA freebsd_committer 2017-07-16 20:47:26 UTC
(In reply to neil from comment #8)

As address specified in `INET_DIST_INTERFACE' is used by `ejabberdctl', so to make sure that works, we need to have the hostname of ejabberd node, i.e. $HOST in "--node ejabberd@$HOST" resolve to that IP address.

I tried reproducing your changes in my setup, my $HOST resolves to 127.0.0.1, so with INET_DIST_INTERFACE=127.0.0.1, it works. When I switch to another IP address, say 192.168.1.1, then for it work I need to add a DNS/hosts entry for that $HOST to point to 192.168.1.1, otherwise ejabberd gets started, but enabled to get stopped, or being polled for status (getting `nodedown').

Could you confirm that this is indeed the case with you ? Also, could you provide the relevant log messages, and ejabberdctl.cfg ? And do you run ejabberd in a FreeBSD jail ? If yes, could you post its network configuration too.

> epmd being left running is a long-standing artifact which we have discussed before.

I don't have a solution for it yet.
Comment 10 Ashish SHUKLA freebsd_committer 2017-07-16 20:48:28 UTC
(In reply to Ashish SHUKLA from comment #9)

> but enabled to get stopped

s/enable/unable/
Comment 11 Marko Cupać 2017-07-18 14:54:43 UTC
Everything's fine here after update to ejabberd-17.07_1, including restarting ejabberd with `service ejabberd restart'.

As I said, I am running this in (ez)jail on separate fib, where loopback interface is lo2 with 127.0.2.129.

I have the following in my hosts file:

pacija@xmpp1:~ % sudo cat /etc/hosts
127.0.2.129	localhost
193.53.106.129	xmpp1 xmpp1.mimar.rs

Thank you for fixing it. As far as I'm concerned this can be closed, but I'll leave it open in case Neal has something to add.
Comment 12 neil 2017-07-19 12:44:39 UTC
I am happy for this bug to be closed.

The commit fixes the original reporter's problem and any issues with my test cases are likely to do with node name mismatches between settings and the database.