Created attachment 238237 [details] Patch to set grafana's working directory at startup Grafana9 doesn't start at system startup (tested in a jail). Discussion on this bug about grafana7 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255676 tracks down the cause to grafana trying to read something from the working directory at startup. Following the suggestion of Nico Wiseman on that bug, the attached patch ensures that grafana starts up in a directory it has appropriate permissions for.
(In reply to Jony Hudson from comment #0) How do you execute the `daemon` command outside of `/root`? I cannot reproduce it. I've asked Nico Wiseman at bug #263760...
Patchfile I use for changing the provided rc.d script --- /usr/local/etc/rc.d/grafana 2022-09-22 01:28:56.000000000 +0000 +++ /usr/local/etc/rc.d/grafana_new 2022-10-04 19:09:10.168212000 +0000 @@ -67,6 +67,7 @@ start_precmd="grafana_start_precmd" +grafana_chdir="${grafana_homepath}" grafana_start_precmd() { if [ ! -d "/var/run/${name}" ]; then
Ahh, I hadn't seen https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263760 . I'm also running it in a jail, so it does look like it could be something jail specific. This reproduces for me, without using any jail management tools other than in the base system: uname -a # FreeBSD pop-bsd 13.1-RELEASE-p3 FreeBSD 13.1-RELEASE-p3 GENERIC amd64 sudo zfs create zroot/jails/grafana-test # Empty root password, nothing but base installed, no services, no users: sudo bsdinstall jail /usr/jails/grafana-test sudo sysrc -f /usr/jails/grafana-test/etc/rc.conf sendmail_enable="NONE" sudo cp /etc/resolv.conf /usr/jails/grafana-test/etc/resolv.conf sudo sh -c 'cat << EOF >> /etc/jail.conf grafana-test { host.hostname = grafana-test.j-star.org; ip4.addr = hn0|10.0.0.199/24; path = /usr/jails/grafana-test; mount.devfs; exec.clean; exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; } EOF' sudo service jail start grafana-test sudo pkg -j grafana-test install grafana9 sudo jexec grafana-test service grafana enable sudo jexec grafana-test service grafana start # Grafana is running as expected at this point sudo service jail restart grafana-test # Grafana has not restarted
(In reply to Jony Hudson from comment #3) I've tried to reproduce your case, but I've gotten restarted Grafana after `sudo service jail restart grafana-test`. I need more time to investigate the problem. But I think, my environment is differ from your environment (I've setup root password and custom world, ex.)
Thanks, that's interesting. Did you find it works building the jail using the steps I posted (i.e. it's a "host" system difference), or did you build the jail differently? Let me know if there's anything I can test on my side. I'll also try and investigate to understand what's going on.
I'm also having the same issue as Jony Hudson reported. My system is on FreeBSD 13.1-RELEASE. I created my jail using Bastille. Applying Nico Wiseman's patch worked for me.
I've been doing a bit of sleuthing here. I suspect it might be an environment problem. What I find is that if I remove the `exec.clean;` line from my jail.conf, then grafana starts up as expected. It's perhaps plausible that this is why some people are finding it works in a jail and others not. If I add a line to /etc/rc in the jail to dump the env (specifically, adding `echo `/usr/bin/env` > /root/env.txt` to /etc/rc just after `export HOME PATH`) then I find that with exec.clean enabled the env is: LANG=C.UTF-8 MAIL=/var/mail/root PATH=/sbin:/bin:/usr/sbin:/usr/bin PWD=/root USER=root HOME=/ SHELL=/bin/csh MM_CHARSET=UTF-8 BLOCKSIZE=K whereas without exec.clean, it is: LANG=C.UTF-8 PATH=/sbin:/bin:/usr/sbin:/usr/bin PWD=/ HOME=/ MM_CHARSET=UTF-8 BLOCKSIZE=K RC_PID=92907 The differing PWDs in particular are suspicious. One can confirm that grafana doesn't start if /root is the working directory by running it directly. Running grafana with / as the working directory, from a shell in the jail created above (with the grafana service disabled temporarily): cd / su -m grafana -c '/usr/local/bin/grafana-server -config=/usr/local/etc/grafana/grafana.ini -homepath=/usr/local/share/grafana' gives: INFO [11-25|17:57:45] Starting Grafana logger=settings version=9.2.4 commit=64017e8 branch=main compiled=2022-11-25T17:57:45Z <snip> INFO [11-25|17:57:45] HTTP Server Listen logger=http.server address=10.0.0.199:3000 protocol=http subUrl= socket= Grafana is up and running as expected. Running it from /root: cd /root su -m grafana -c '/usr/local/bin/grafana-server -config=/usr/local/etc/grafana/grafana.ini -homepath=/usr/local/share/grafana' gives: panic: error getting work directory: stat .: permission denied goroutine 1 [running]: github.com/grafana/grafana/pkg/api/static.init.0() github.com/grafana/grafana/pkg/api/static/static.go:38 +0x94 The offending code on Github (https://github.com/grafana/grafana/blob/main/pkg/api/static/static.go#L38) shows that grafana is indeed trying to access the working directory as it starts up. I don't really understand why the environment looks the way it does when the jail is created with exec.clean. On the face of it, it doesn't line up with what the jail man page says about exec.clean, but I can imagine that several things happen that alter the environment before /etc/rc is reached. I'm not familiar with them, though! So I suspect this might be the problem, but I don't actually have a feeling for what the proper fix is.
(In reply to Jony Hudson from comment #7) I've a jail.conf with `exec.clean;` and env.txt with: LANG=C.UTF-8 MAIL=/var/mail/root PATH=/sbin:/bin:/usr/sbin:/usr/bin PWD=/root USER=root HOME=/ SHELL=/bin/sh MM_CHARSET=UTF-8 RC_PID=16733 BLOCKSIZE=K Also, I've tried to run Grafana from /root, and I've got: # cd /root # su -m grafana -c '/usr/local/bin/grafana-server -config=/usr/local/etc/grafana/grafana.ini -homepath=/usr/local/share/grafana' INFO [11-25|19:30:09] Starting Grafana logger=settings version=9.2.5 commit=042e4d2 branch=main compiled=2022-11-25T19:30:09Z <snip> INFO [11-25|19:30:09] HTTP Server Listen logger=http.server address=0.0.0.0:3000 protocol=http subUrl= socket=
Checking the obvious stuff in my jail, I think it is expected that grafana wouldn't be able to stat the /root directory. Do you see the same as the following? root@grafana-test:/ # ls -al / <snip> drwxr-x--- 2 root wheel 11 Nov 25 20:29 root <snip> root@grafana-test:/ # groups grafana grafana And your grafana is configured to run under the default user and group grafana:grafana?
It seems to me that regardless of what is happening outside of the actual service the RC script provided with grafana just seems to me to be missing the directive to run in the grafana directory. Currently, the RC script gives grafana a home directory to run in but it is not instructed to run inside this directory. I feel like the proper fix would just be to always tell grafana to run inside its home directory as the grafana user.
(In reply to Jony Hudson from comment #9) Yep, you are right. My /root dir have 0755 perms. Thx. (In reply to Nico Wiseman from comment #10) And so, do you propose to add `grafana_chdir` string to RC only? I think it should be opened a bug report about the problem (attempt to address to the current dir instead homedir) on Grafana, shouldn't it?
(In reply to Boris Korzun from comment #11) Ahh, that explains it! IMO, this is arguably a bug in Grafana. Looking at their code, the only reason they stat the current working directory is to resolve the configuration file path if specified relatively. It should, again IMO, be fine to start Grafana in a directory it can't read if all paths are absolute. I'll file an upstream issue with them. Since grafana has a well defined working directory it seems like chdir'ing to that is a reasonable workaround to the issue, but I do agree that it's not really the fix. And I find something odd about the way the env vars are being set in the exec.clean jail, but I think that's a separate thing to look into, and I'll start a discussion on that/file a bug as appropriate if I figure it out.
For reference: https://github.com/grafana/grafana/issues/59441
Do you think it's worth merging this patch until upstream resolve the underlying issue? They currently have 2,626 open issues, so I can imagine it might not get fixed any time soon :-)
Hello, I also had the same issue when starting jail with Grafana on 13.1-STABLE. After applying the attached patch I was able restart jail with Grafana running. Could you please let us know if this mitigation patch could be merged to a branch: stable or current? Thanks Max
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=a553cd5c93ed65e82f153c516e84eb1b8f4ee0be commit a553cd5c93ed65e82f153c516e84eb1b8f4ee0be Author: Boris Korzun <drtr0jan@yandex.ru> AuthorDate: 2023-03-09 08:05:06 +0000 Commit: Fernando Apesteguía <fernape@FreeBSD.org> CommitDate: 2023-03-10 11:11:33 +0000 www/grafana9: Update to 9.4.3 ChangeLog: https://grafana.com/blog/2023/02/28/grafana-9.4-release/ PR: 270009 267914 Reported by: drtr0jan@yandex.ru (maintainer) www/grafana9/Makefile | 5 +- www/grafana9/distinfo | 14 +- www/grafana9/files/grafana.in | 1 + www/grafana9/pkg-plist | 2058 ++++++++++++++++++++++------------------- 4 files changed, 1139 insertions(+), 939 deletions(-)
This should be fixed with the update in bug #270009. Can any of you confirm?