Bug 208793 - www/gitlab: gitlab did not start on boot
Summary: www/gitlab: gitlab did not start on boot
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Torsten Zuehlsdorff
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-04-14 07:37 UTC by Torsten Zühlsdorff
Modified: 2017-12-17 07:12 UTC (History)
9 users (show)

See Also:


Attachments
Fix boot problem (6.17 KB, patch)
2016-04-17 10:09 UTC, Johannes Jost Meixner
no flags Details | Diff
Next try to fix boot problem (5.65 KB, patch)
2016-06-23 11:50 UTC, Torsten Zuehlsdorff
no flags Details | Diff
unified patch for /usr/local/etc/rc.d/gitlab (7.04 KB, patch)
2016-08-06 05:13 UTC, Horst Kapfenberger
no flags Details | Diff
rc.d script for gitlab adapted to FreeBSD (12.28 KB, text/plain)
2016-08-09 21:38 UTC, Horst Kapfenberger
no flags Details
Updated rc.d script (7.99 KB, patch)
2016-12-23 15:04 UTC, Torsten Zuehlsdorff
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Torsten Zühlsdorff 2016-04-14 07:37:43 UTC
Hello,

i move the problem described here into an own PR:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=208511

Currently gitlab did not start on boot or on jail-start, but it starts when you do a "service gitlab start".

This is through problems within the rc script. xmj@ already provided a patch, but i think the patch needs some more thoughts. But the patch fixes the problem and is a good start :)

Greetings,
Torsten
Comment 1 Johannes Jost Meixner freebsd_committer 2016-04-17 10:09:34 UTC
Created attachment 169394 [details]
Fix boot problem

Patch attached fixes the boot problems and brings the rc file into rc(8)-compliance.
Comment 2 Johannes Jost Meixner freebsd_committer 2016-04-17 10:15:32 UTC
Torsten, I do understand your desire to stay as close to upstream as possible.
With respect to this however it is worth noting that the init systems on Linux and FreeBSD are different and you cannot expect the startup scripts to be as close.

We do have some abstractions that allow us to remove certain things you'd need to specify manually on Linux (and, additionally, across many Linux rc scripts).

I used them in the patch attached, and tried to keep the difference to the upstream functionality as limited as possible. I changed function names, variables we use for rc.conf, and added a dependency on NGINX and Redis as it otherwise does not start.

I confirmed with restarting a jail a few times that this patch actually fixes the bug attached.
Comment 3 miks.mikelsons 2016-04-18 06:54:11 UTC
There is serious flaw in this patched init script. All services started from this script is owned by root instead of git user.
There are security risks and also, new repositories is created with "root" user as owner, therefore I can't push later to them when connection from ssh.
Comment 4 Johannes Jost Meixner freebsd_committer 2016-04-18 09:07:39 UTC
(In reply to miks.mikelsons from comment #3)
Clarifying: Is this with the default script?
Comment 5 miks.mikelsons 2016-04-18 09:09:56 UTC
What you mean with "default"? In gitlab init.d there us "su" call which do its job to run all gitlab processes under "git" user.
Comment 6 Johannes Jost Meixner freebsd_committer 2016-04-18 09:53:49 UTC
(In reply to miks.mikelsons from comment #5)
Where are you seeing the issue. 

https://svnweb.freebsd.org/ports/head/www/gitlab/files/gitlab.in

or my patch attached?
Comment 7 miks.mikelsons 2016-04-18 10:58:10 UTC
In your patch (but your patch at least is starting gitlan on boot).
Comment 8 Rodomar705 2016-05-15 08:20:02 UTC
Indeed, all processes is owned by root and not by git user. The problem is that reversing that specific hunk of the patch itself retrigger the problem, and the services doesn't start at boot again. Any suggestions?
Comment 9 Torsten Zuehlsdorff freebsd_committer 2016-06-23 11:50:15 UTC
Created attachment 171708 [details]
Next try to fix boot problem

Hello,

since the patch no longer applies i recreated it. 

Also i put back the line with the user-switch. Since there was no consens if this work or not, lets try again. Does the new patch:
- let gitlab start after the boot?
- execute gitlab with correct user-rights?

Greetings,
Torsten
Comment 10 Rodomar705 2016-06-23 13:16:08 UTC
(In reply to Torsten Zuehlsdorff from comment #9)

Tested. No changes, unfortunately. The service doesn't start at boot, all that is shown by nginx is a 502 page, and sending service gitlab status reports gitlab not running. After starting manually everything works. Tried gitlab:check, was thinking about permissions problems (gitlab folder was owned by root:wheel, when the normal install requires it under git:git). Fixed it, all checks are green besides sidekick and init.d scripts (normal, since the check scripts are wrote with linux in mind), but that's not changed a thing. Can't really where the problem resides, if this helps you out, thanks a lot.
Comment 11 Jun Kuriyama freebsd_committer 2016-07-28 08:17:51 UTC
FWIW, I'm using GitLab which installed manually (not from ports).

I installed https://svnweb.freebsd.org/ports/head/www/gitlab/files/gitlab.in file and it didn't work on startup.

The problem is, on startup $0 is not 'gitlab' script itself.  It is "/etc/rc" (or "/etc/rc.shutdown" on shutdown).

I hacked like below to work around, but not yet submitted as patch because this is not right way (I think).

> # Switch to the app_user if it is not he/she who is running the script.
> if [ "$USER" != "$app_user" ]; then
>   script=$0
>   [ "$script" = "/etc/rc" -o "$script" = "/etc/rc.shutdown" ] && script="service gitlab"
>   eval su - "$app_user" -c $(echo \")$script "$@"$(echo \"); exit;
> fi
Comment 12 Horst Kapfenberger 2016-08-06 05:11:54 UTC
possible solution:

# create a login class for service user like this (add to /etc/login.conf)
git:\
        :path=/sbin /bin /usr/sbin /usr/bin /usr/local/sbin /usr/local/bin:\
        :setenv=MAIL=/var/mail/$,BLOCKSIZE=K,RAILS_ENV=production:\
        :tc=default:

# reload
cap_mkdb /etc/login.conf
# and set it on user
pw moduser gitlab -L gitlab


# patch /usr/local/etc/rc.d/gitlab
#   unix sockets used in this version, though not mandatory
# 


--- gitlab.orig	2016-08-06 05:00:56.782535000 +0200
+++ gitlab	2016-08-06 06:46:54.637809000 +0200
@@ -2,17 +2,6 @@
 
 # $FreeBSD: head/www/gitlab/files/gitlab.in 417194 2016-06-21 08:36:56Z tz $
 
-### BEGIN INIT INFO
-# Provides:          gitlab
-# Required-Start:    $local_fs $remote_fs $network $syslog redis-server
-# Required-Stop:     $local_fs $remote_fs $network $syslog
-# Default-Start:     2 3 4 5
-# Default-Stop:      0 1 6
-# Short-Description: GitLab git repository management
-# Description:       GitLab git repository management
-# chkconfig: - 85 14
-### END INIT INFO
-
 # Maintainer: Torsten Zuehlsdorff <ports@toco-domains.de>
 # Based on work of: @charlienewey, rovanion.luckey@gmail.com, @randx
 
@@ -51,30 +40,36 @@
 pid_path="$app_root/tmp/pids"
 socket_path="$app_root/tmp/sockets"
 rails_socket="$socket_path/gitlab.socket"
+workhorse_socket=$socket_path/gitlab-workhorse.socket
 web_server_pid_path="$pid_path/unicorn.pid"
 sidekiq_pid_path="$pid_path/sidekiq.pid"
 mail_room_enabled=false
 mail_room_pid_path="$pid_path/mail_room.pid"
 gitlab_workhorse_dir=$(cd $app_root/../gitlab-workhorse && pwd)
 gitlab_workhorse_pid_path="$pid_path/gitlab-workhorse.pid"
-gitlab_workhorse_options="-listenUmask 0 -listenNetwork unix -listenAddr $socket_path/gitlab-workhorse.socket -authBackend http://127.0.0.1:8080 -authSocket $rails_socket -documentRoot $app_root/public"
+gitlab_workhorse_options="-listenNetwork unix -listenAddr $workhorse_socket -authSocket $rails_socket -documentRoot $app_root/public"
 gitlab_workhorse_log="$app_root/log/gitlab-workhorse.log"
 shell_path="/bin/bash"
 
 # Read configuration variable file if it is present
-test -f /etc/default/gitlab && . /etc/default/gitlab
+# test -f /etc/default/gitlab && . /etc/default/gitlab
 
 # Switch to the app_user if it is not he/she who is running the script.
-if [ "$USER" != "$app_user" ]; then
-  eval su - "$app_user" -c $(echo \")$0 "$@"$(echo \"); exit;
-fi
+# if [ "$USER" != "$app_user" ]; then
+#   eval su - "$app_user" -c $(echo \")$0 "$@"$(echo \"); exit;
+# fi
 
 # Switch to the gitlab path, exit on failure.
-if ! cd "$app_root" ; then
- echo "Failed to cd into $app_root, exiting!";  exit 1
+# if ! cd "$app_root" ; then
+#  echo "Failed to cd into $app_root, exiting!";  exit 1
+# fi
+
+if ! su - $app_user -c "echo \$RAILS_ENV" >/dev/null
+then
+  echo "ERROR: RAILS_ENV not set for user $app_user, login.conf(5)"
+  exit 80
 fi
 
-
 ### Init Script functions
 
 ## Gets the pids from the files
@@ -218,34 +213,26 @@
 start_gitlab() {
   check_stale_pids
 
-  if [ "$web_status" != "0" ]; then
-    echo "Starting GitLab Unicorn"
-  fi
-  if [ "$sidekiq_status" != "0" ]; then
-    echo "Starting GitLab Sidekiq"
-  fi
-  if [ "$gitlab_workhorse_status" != "0" ]; then
-    echo "Starting gitlab-workhorse"
-  fi
-  if [ "$mail_room_enabled" = true ] && [ "$mail_room_status" != "0" ]; then
-    echo "Starting GitLab MailRoom"
-  fi
-
   # Then check if the service is running. If it is: don't start again.
   if [ "$web_status" = "0" ]; then
     echo "The Unicorn web server already running with pid $wpid, not restarting."
   else
+    echo "Starting GitLab Unicorn"
     # Remove old socket if it exists
     rm -f "$socket_path"/gitlab.socket 2>/dev/null
-    # Start the web server
-    RAILS_ENV=$RAILS_ENV bin/web start
+    # unicorn may break the current process on freebsd during 
+    # deamonizing (defunct exists until new master has prefilled 
+    # caches) - therefore start itself is decoupled
+    /usr/sbin/daemon -u ${app_user} ${app_root}/bin/web start
+    # 
   fi
 
   # If sidekiq is already running, don't start it again.
   if [ "$sidekiq_status" = "0" ]; then
     echo "The Sidekiq job dispatcher is already running with pid $spid, not restarting"
   else
-    RAILS_ENV=$RAILS_ENV bin/background_jobs start &
+    echo "Starting GitLab Sidekiq"
+    su - $app_user -c "${app_root}/bin/background_jobs start"
   fi
 
   if [ "$gitlab_workhorse_status" = "0" ]; then
@@ -254,10 +241,11 @@
     # No need to remove a socket, gitlab-workhorse does this itself.
     # Because gitlab-workhorse has multiple executables we need to fix
     # the PATH.
-    $app_root/bin/daemon_with_pidfile $gitlab_workhorse_pid_path  \
-      /usr/bin/env PATH=$gitlab_workhorse_dir:$PATH \
-        gitlab-workhorse $gitlab_workhorse_options \
-      >> $gitlab_workhorse_log 2>&1 &
+    echo "Starting gitlab-workhorse"
+    cd ${gitlab_workhorse_dir} &&
+        /usr/sbin/daemon -p ${gitlab_workhorse_pid_path} -u ${app_user} \
+            $gitlab_workhorse_dir/gitlab-workhorse $gitlab_workhorse_options \
+            >> $gitlab_workhorse_log 2>&1
   fi
 
   if [ "$mail_room_enabled" = true ]; then
@@ -265,7 +253,8 @@
     if [ "$mail_room_status" = "0" ]; then
       echo "The MailRoom email processor is already running with pid $mpid, not restarting"
     else
-      RAILS_ENV=$RAILS_ENV bin/mail_room start &
+        echo "Starting GitLab MailRoom"
+        /usr/sbin/daemon -u ${app_user} ${app_root}/bin/mail_room start
     fi
   fi
 
@@ -281,19 +270,21 @@
 
   if [ "$web_status" = "0" ]; then
     echo "Shutting down GitLab Unicorn"
-    RAILS_ENV=$RAILS_ENV bin/web stop
+    su - $app_user -c "${app_root}/bin/web stop"
+    [ -S ${rails_socket} ] && rm -f ${rails_socket}
   fi
   if [ "$sidekiq_status" = "0" ]; then
     echo "Shutting down GitLab Sidekiq"
-    RAILS_ENV=$RAILS_ENV bin/background_jobs stop
+    su - $app_user -c "${app_root}/bin/background_jobs stop"
   fi
   if [ "$gitlab_workhorse_status" = "0" ]; then
     echo "Shutting down gitlab-workhorse"
     kill -- $(cat $gitlab_workhorse_pid_path)
+    [ -S ${workhorse_socket} ] && rm -f ${workhorse_socket}
   fi
   if [ "$mail_room_enabled" = true ] && [ "$mail_room_status" = "0" ]; then
     echo "Shutting down GitLab MailRoom"
-    RAILS_ENV=$RAILS_ENV bin/mail_room stop
+    su - $app_user -c "${rails_root}/bin/mail_room stop"
   fi
 
   # If something needs to be stopped, lets wait for it to stop. Never use SIGKILL in a script.
@@ -361,15 +352,15 @@
     exit 1
   fi
   printf "Reloading GitLab Unicorn configuration... "
-  RAILS_ENV=$RAILS_ENV bin/web reload
+  $sudo env RAILS_ENV=$RAILS_ENV bin/web reload
   echo "Done."
 
   echo "Restarting GitLab Sidekiq since it isn't capable of reloading its config..."
-  RAILS_ENV=$RAILS_ENV bin/background_jobs restart
+  $sudo env RAILS_ENV=$RAILS_ENV bin/background_jobs restart
 
   if [ "$mail_room_enabled" != true ]; then
     echo "Restarting GitLab MailRoom since it isn't capable of reloading its config..."
-    RAILS_ENV=$RAILS_ENV bin/mail_room restart
+    $sudo env RAILS_ENV=$RAILS_ENV bin/mail_room restart
   fi
 
   wait_for_pids
@@ -386,29 +377,5 @@
 }
 
 
-### Finally the input handling.
-
-case "$1" in
-  start)
-        start_gitlab
-        ;;
-  stop)
-        stop_gitlab
-        ;;
-  restart)
-        restart_gitlab
-        ;;
-  reload|force-reload)
-	reload_gitlab
-        ;;
-  status)
-        print_status
-        exit $gitlab_status
-        ;;
-  *)
-        echo "Usage: service gitlab {start|stop|restart|reload|status}"
-        exit 1
-        ;;
-esac
 
-exit
+run_rc_command "$1"
Comment 13 Horst Kapfenberger 2016-08-06 05:13:31 UTC
Created attachment 173348 [details]
unified patch for /usr/local/etc/rc.d/gitlab

again as attachment
Comment 14 Horst Kapfenberger 2016-08-09 21:38:50 UTC
Created attachment 173478 [details]
rc.d script for gitlab adapted to FreeBSD

full version of gitlab rc.d script
- no login class necessary
- start, stop, status, restart, reload
- bugs from last diff file fixed
Comment 15 Torsten Zuehlsdorff freebsd_committer 2016-12-23 15:04:30 UTC
Created attachment 178224 [details]
Updated rc.d script

I rebased the script provided by Horst, but it is not executable. It ends with this error:
/usr/local/etc/rc.d/gitlab: 264: Syntax error: Unterminated quoted string

Did anyone has used the script provided by Horst? Can someone point me to the problem. It is more than time to close this issue!
Comment 16 commit-hook freebsd_committer 2017-03-31 11:28:17 UTC
A commit references this bug:

Author: tz
Date: Fri Mar 31 11:27:27 UTC 2017
New revision: 437349
URL: https://svnweb.freebsd.org/changeset/ports/437349

Log:
  www/gitlab: enable GitLab to start on boot

  PR:           208793
  Submitted by: L?szl? K?rolyi <laszlo@karolyi.hu>, Horst Kapfenberger <horst.kapfenberger@agoracon.at>

Changes:
  head/www/gitlab/Makefile
  head/www/gitlab/files/gitlab.in
Comment 17 Torsten Zuehlsdorff freebsd_committer 2017-03-31 11:30:13 UTC
I added a quick fix to ensure it will got into the next quarterly branch.

For the bigger patch of Horst i need more time to review and i'm just having no time currently. :/
Comment 18 Torsten Zuehlsdorff freebsd_committer 2017-11-21 11:09:16 UTC
Since it now starts on boot, i close this ticket.
Comment 19 vali gholami 2017-12-17 07:12:23 UTC
MARKED AS SPAM