Bug 274248 - net-mgmt/librenms: service librenms start failing
Summary: net-mgmt/librenms: service librenms start failing
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Dan Langille
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-10-03 19:13 UTC by Dries Michiels
Modified: 2023-10-07 14:53 UTC (History)
0 users

See Also:
bugzilla: maintainer-feedback? (dvl)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dries Michiels freebsd_committer freebsd_triage 2023-10-03 19:13:14 UTC
After some upgrade, one of the recent ones; not sure if librenms related or python upgrade related or php related but the error I get when starting librenms with "service librenms star" is :

Oct  3 21:06:10 librenms librenms[13235]: MainThread(CRITICAL):Error in config fetching process: 127
Oct  3 21:06:10 librenms librenms[13235]: MainThread(CRITICAL):ERROR: Could not execute command [['/usr/bin/env', 'ph
p', '/usr/local/www/librenms/config_to_json.php']]: Expecting value: line 1 column 1 (char 0)
Oct  3 21:06:10 librenms librenms[13235]: 'NoneType' object has no attribute 'get'

When I execute the command manually from the CLI which the rc script produces it works fine. I'm on FreeBSD 14-STABLE. Will take a look later, if you could also take a quick look Dan that would great!
Comment 1 Dan Langille freebsd_committer freebsd_triage 2023-10-03 19:49:54 UTC
It's not this file:

[19:29 besser dan ~] % head /usr/local/www/librenms/config_to_json.php 
#!/usr/local/bin/php
<?php

/*
 * Configuration to JSON converter
 * Written by Job Snijders <job@instituut.net>
 *
 */

use LibreNMS\Config;


It's LibreNMS/__init__.py

[19:30 besser dan /usr/local/www/librenms] % grep -r config_to_json.php  * | head -2
LibreNMS/__init__.py:    config_cmd = ["/usr/bin/env", "php", "%s/config_to_json.php" % base_dir]
doc/General/Changelogs/2019.md:* Don't require db for config_to_json.php ([#10100](https://github.com/librenms/librenms/pull/10100)) - [murrant](https://github.com/murrant)

The reason this works on Linux (I think), php is in /usr/bin. On most sane systems, it's in /usr/local/bin - and Apache's $PATH does not include that directory.

The solution is much like shebang fixes.

It reminds me of https://cgit.freebsd.org/ports/tree/net-mgmt/librenms/files/patch-LibreNMS_wrapper.py 

I'll work on a patch.
Comment 2 Dan Langille freebsd_committer freebsd_triage 2023-10-03 20:05:41 UTC
I have this untested patch:

config_cmd = "/usr/local/bin/php %s/config_to_json.php" % base_dir

If you can  try it out, great.
Comment 3 Dan Langille freebsd_committer freebsd_triage 2023-10-03 21:06:08 UTC
(In reply to Dries Michiels from comment #0)
Are you using Nginx?

What version of LibreNMS are you running now?

Do those messages appear in /var/log/messages?  /var/log/librenms/librenms-service.log

I ask because I'm using:

apache24-2.4.57_1
librenms-23.9.1,1

I cannot reproduce the issue:



[21:02 besser dan /usr/local/www/librenms] % sudo service librenms start
Starting librenms.
[21:02 besser dan /usr/local/www/librenms] %

thanks
Comment 4 Dan Langille freebsd_committer freebsd_triage 2023-10-03 21:47:39 UTC
Corrected patch to be applied via edit.

config_cmd = ["/usr/local/bin/php", "%s/config_to_json.php" % base_dir]

that should replace line 170 of /usr/local/www/librenms/LibreNMS/__init__ :

config_cmd = ["/usr/bin/env", "php", "%s/config_to_json.php" % base_dir]
Comment 5 Dries Michiels freebsd_committer freebsd_triage 2023-10-04 07:54:24 UTC
Yeah patch works great!
Comment 6 Dries Michiels freebsd_committer freebsd_triage 2023-10-04 07:55:01 UTC
I'm on FreeBSD 14-STABLE with 23.9.1,1 LibreNMS package version. I indeed use nginx.
Comment 7 Dries Michiels freebsd_committer freebsd_triage 2023-10-04 07:56:57 UTC
I found the error messages in librenms-service.log, and of course noticed the problem when my graphs didn't show no data anymore (poller was not started).
Comment 8 Dan Langille freebsd_committer freebsd_triage 2023-10-04 12:38:45 UTC
(In reply to Dries Michiels from comment #5)
I will have a port update soon. I will also patch other similar situations and add more to the SHEBANG fixes.

The nginx guess was based on my use of Apache and not seeing the error. I am not using Apache over Nginx for any particular reason.
Comment 9 commit-hook freebsd_committer freebsd_triage 2023-10-04 15:32:37 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=8ff190a6fb75c40acd4ebf3a579187ebd8a26d57

commit 8ff190a6fb75c40acd4ebf3a579187ebd8a26d57
Author:     Dan Langille <dvl@FreeBSD.org>
AuthorDate: 2023-10-04 14:10:57 +0000
Commit:     Dan Langille <dvl@FreeBSD.org>
CommitDate: 2023-10-04 15:30:33 +0000

    net-mgmt/librenms: remove /usr/bin/env usage

    This problem occurred on Nginx, but not Apache. Instead of
    "/usr/bin/env php", use "/usr/local/bin/php".

    While here:

    * add more to SHEBANG_FILES
    * portfmnt

    PR:             274248

 net-mgmt/librenms/Makefile                         | 26 +++++++++++++---------
 .../files/patch-LibreNMS_____init____.py (new)     | 11 +++++++++
 net-mgmt/librenms/files/patch-snmp-scan.py (new)   | 21 +++++++++++++++++
 3 files changed, 48 insertions(+), 10 deletions(-)
Comment 10 Dan Langille freebsd_committer freebsd_triage 2023-10-04 15:37:27 UTC
Done, thank you for reporting this.
Comment 11 Dries Michiels freebsd_committer freebsd_triage 2023-10-06 17:06:25 UTC
There seems something else at play here too. The daemon now starts correctly after the patch, but the actual polling does not happen. Again, when I start the service on the CLI, it does. Probably somewhere another $path issue not including /usr/local/[sbin/bin]?

Command I use to start from the CLI;

"daemon -p /var/run/librenms/librenms.pid -u www -T librenms /usr/local/bin/python3.9 /usr/local/www/librenms/librenms-service.py"
Comment 12 Dries Michiels freebsd_committer freebsd_triage 2023-10-06 17:11:22 UTC
# find . -type file -exec grep "/usr/bin/env" {}  \;
        base = ("/usr/bin/env", "php")

First hit seems suspicious, also in __init__.py.

I'm wondering if the change of the root shell (in STABLE 14, csh to sh) has anything to do with the problem as you have no observed anything so far?
Comment 13 Dan Langille freebsd_committer freebsd_triage 2023-10-06 17:12:30 UTC
Reopening.
Comment 14 Dan Langille freebsd_committer freebsd_triage 2023-10-06 17:36:18 UTC
I am not sure, I am guessing.  Try changing that suspect line to:

base = ("/usr/local/bin/php")


I have tried it here, and it works, but then, mine was also working.
Comment 15 Dries Michiels freebsd_committer freebsd_triage 2023-10-07 11:03:29 UTC
For me, 

base = ("/usr/local/bin/php") 

does not work (/var/log/messages) output

Oct  7 12:58:00 librenms librenms[98158]: Traceback (most recent call last):
Oct  7 12:58:00 librenms syslogd: last message repeated 1 times
Oct  7 12:58:00 librenms librenms[98158]:   File "/usr/local/www/librenms/LibreNMS/queuemanager.py", line 85, in _ser
vice_worker
Oct  7 12:58:00 librenms librenms[98158]:     self.do_work(device_id, queue_id)
Oct  7 12:58:00 librenms librenms[98158]:   File "/usr/local/www/librenms/LibreNMS/queuemanager.py", line 85, in _ser
vice_worker
Oct  7 12:58:00 librenms librenms[98158]:     self.do_work(device_id, queue_id)
Oct  7 12:58:00 librenms librenms[98158]:   File "/usr/local/www/librenms/LibreNMS/queuemanager.py", line 370, in do_
work
Oct  7 12:58:00 librenms librenms[98158]:     exit_code, output = LibreNMS.call_script("billing-calculate.php", args)
Oct  7 12:58:00 librenms librenms[98158]:   File "/usr/local/www/librenms/LibreNMS/queuemanager.py", line 510, in do_
work
Oct  7 12:58:00 librenms librenms[98158]:     exit_code, output = LibreNMS.call_script("alerts.php", args)
Oct  7 12:58:00 librenms librenms[98158]:   File "/usr/local/www/librenms/LibreNMS/__init__.py", line 203, in call_sc
ript
Oct  7 12:58:00 librenms librenms[98158]:     cmd = base + ("{}/{}".format(base_dir, script),) + tuple(map(str, args)
)
Oct  7 12:58:00 librenms librenms[98158]:   File "/usr/local/www/librenms/LibreNMS/__init__.py", line 203, in call_sc
ript
Oct  7 12:58:00 librenms librenms[98158]:     cmd = base + ("{}/{}".format(base_dir, script),) + tuple(map(str, args)
)
Oct  7 12:58:00 librenms librenms[98158]: TypeError: can only concatenate str (not "tuple") to str

with

base = ("/usr/local/bin/php",)

it works
Comment 16 Dan Langille freebsd_committer freebsd_triage 2023-10-07 11:52:55 UTC
I could investigate the patch issue and look at adding /usr/local/bin ... but I'm not.

I suspect that may be the root cause. I'm happy for someone else to look into that possibility.  Until then, I'll use your solution.
Comment 17 commit-hook freebsd_committer freebsd_triage 2023-10-07 11:53:15 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=73922d6b363a6902a05721d9102386ce657b03d4

commit 73922d6b363a6902a05721d9102386ce657b03d4
Author:     Dan Langille <dvl@FreeBSD.org>
AuthorDate: 2023-10-07 11:49:41 +0000
Commit:     Dan Langille <dvl@FreeBSD.org>
CommitDate: 2023-10-07 11:49:41 +0000

    net-mgmt/librenms: Fix start on FreeBSD 14/nginx

    This may be a path issue or a shell issue, but nginx on FreeBSD 14
    is not finding /usr/local/bin/php. The fix: use /usr/local/bin/php
    as we have done with other patches.

    PR:             274248

 net-mgmt/librenms/Makefile                             |  2 +-
 net-mgmt/librenms/files/patch-LibreNMS_____init____.py | 11 ++++++++++-
 2 files changed, 11 insertions(+), 2 deletions(-)
Comment 18 Dries Michiels freebsd_committer freebsd_triage 2023-10-07 14:53:53 UTC
Works and polling also works now. Great!