Bug 247564 - www/gitlab-ce Switch application server from unicorn to puma
Summary: www/gitlab-ce Switch application server from unicorn to puma
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Matthias Fechner
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-26 14:51 UTC by hakan
Modified: 2020-08-11 14:15 UTC (History)
0 users

See Also:
bugzilla: maintainer-feedback? (mfechner)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description hakan 2020-06-26 14:51:15 UTC
Greetings,

upstream GitLab switched their default application server in the 13.0 release [0] and it would be great if this port would also switch to it.
While Unicorn still works fine, switching to Puma might reduce GitLab's memory consumption by a lot, and it might even allow users to run GitLab on smaller VPS instances.

[0] https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/#reduced-memory-consumption-of-gitlab-with-puma
Comment 1 Matthias Fechner freebsd_committer 2020-06-27 08:25:51 UTC
You can try it be changing the option or environment variable USE_WEB_SERVER in file /usr/local/etc/rc.d/gitlab.

This should use puma instead of unicorn.
But as gitlab does not use the standard puma package:
https://gitlab.com/gitlab-org/gitlab/-/blob/master/Gemfile#L174

And the rails60 package fetches the standard puma package:
https://svnweb.freebsd.org/ports/head/www/rubygem-rails60/Makefile?view=markup&pathrev=539880#l36

we have here a conflict for executables and that is the reason why the executable for the rubygem-gitlab-puma is changed from puma to gpuma.

Maybe it is enough to modify the startup script for gitlab.

If you like you can test it and report back if it works or not.
Comment 2 hakan 2020-06-27 09:01:31 UTC
I changed the USE_WEB_SERVER value in /usr/local/etc/rc.d/gitlab to 'puma' and got this error:

```
sudo service gitlab start
Removing stale Sidekiq job dispatcher pid. This is most likely caused by Sidekiq crashing the last time it ran.
Starting GitLab web server (puma)
Starting GitLab Sidekiq
Starting GitLab Workhorse
Starting Gitaly
bundler: failed to load command: puma (/usr/local/bin/puma)
LoadError: cannot load such file -- /home/git/gitlab/lib/gitlab/puma_logging/json_formatter
  /usr/local/www/gitlab-ce/config/puma.rb:73:in `require_relative'
  /usr/local/www/gitlab-ce/config/puma.rb:73:in `_load_from'
  /usr/local/lib/ruby/gems/2.6/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/dsl.rb:46:in `instance_eval'
  /usr/local/lib/ruby/gems/2.6/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/dsl.rb:46:in `_load_from'
  /usr/local/lib/ruby/gems/2.6/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/configuration.rb:195:in `block in load'
  /usr/local/lib/ruby/gems/2.6/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/configuration.rb:195:in `each'
  /usr/local/lib/ruby/gems/2.6/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/configuration.rb:195:in `load'
  /usr/local/lib/ruby/gems/2.6/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/launcher.rb:58:in `initialize'
  /usr/local/lib/ruby/gems/2.6/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cli.rb:71:in `new'
  /usr/local/lib/ruby/gems/2.6/gems/gitlab-puma-4.3.3.gitlab.2/lib/puma/cli.rb:71:in `initialize'
  /usr/local/lib/ruby/gems/2.6/gems/gitlab-puma-4.3.3.gitlab.2/bin/puma:8:in `new'
  /usr/local/lib/ruby/gems/2.6/gems/gitlab-puma-4.3.3.gitlab.2/bin/puma:8:in `<top (required)>'
  /usr/local/bin/puma:23:in `load'
  /usr/local/bin/puma:23:in `<top (required)>'
Sidekiq output will be written to /usr/local/www/gitlab-ce/log/sidekiq.log
..............................Waited 30s for the processes to write their pids, something probably went wrong.
```


Then I ran the stop script and changed USE_WEB_SERVER to 'gpuma' while also adjusting the case condition that comes right after it.
Then I got: 

```
sudo service gitlab start
Removing stale Sidekiq job dispatcher pid. This is most likely caused by Sidekiq crashing the last time it ran.
Starting GitLab web server (gpuma)
Starting GitLab Sidekiq
Starting GitLab Workhorse
Starting Gitaly
Unkown web server used by USE_WEB_SERVER: gpuma.
Sidekiq output will be written to /usr/local/www/gitlab-ce/log/sidekiq.log
..............................Waited 30s for the processes to write their pids, something probably went wrong.
```

This "Unkown web server used by USE_WEB_SERVER:" error seems to originate from outside the rc.d script.
Comment 3 Matthias Fechner freebsd_committer 2020-08-11 10:05:58 UTC
Do you want to test this patch:
https://gitlab.fechner.net/mfechner/Gitlab/-/commit/1ef39c17382bb6c2e9b8e8a943c3ab494c6f4379

It runs fine here now with puma.
If it works fine for you as well, I can commit this patch in around two weeks.

Thanks.
Comment 4 hakan 2020-08-11 10:39:43 UTC
Thanks for spending your time on this patch. I am not sure if my procedure for applying the patch was right, but I downloaded the patch file by appending .patch to the commit's URL and applied it on my local ports tree like this:

sudo patch -d /usr/ports -i /usr/home/hakan/1ef39c17382bb6c2e9b8e8a943c3ab494c6f4379.patch
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 1ef39c17382bb6c2e9b8e8a943c3ab494c6f4379 Mon Sep 17 00:00:00 2001
|From: Matthias Fechner <idefix@fechner.net>
|Date: Tue, 11 Aug 2020 12:01:39 +0200
|Subject: [PATCH] Switch standard webserver to puma.
|
|---
| www/gitlab-ce/Makefile                            |  2 +-
| www/gitlab-ce/files/gitlab.in                     |  2 +-
| www/gitlab-ce/files/patch-bin_actioncable         | 11 +++++++++++
| www/gitlab-ce/files/patch-bin_web__puma           | 11 +++++++++++
| www/gitlab-ce/files/patch-config_puma.rb.example  | 15 ++++++++++++++-
| www/rubygem-gitlab-puma/Makefile                  |  7 ++++---
| .../files/patch-gitlab-puma.gemspec               | 14 ++++++++++++++
| 7 files changed, 56 insertions(+), 6 deletions(-)
| create mode 100644 www/gitlab-ce/files/patch-bin_actioncable
| create mode 100644 www/gitlab-ce/files/patch-bin_web__puma
| create mode 100644 www/rubygem-gitlab-puma/files/patch-gitlab-puma.gemspec
|
|diff --git a/www/gitlab-ce/Makefile b/www/gitlab-ce/Makefile
|index dec4520a83..aa072a2c0b 100644
|--- a/www/gitlab-ce/Makefile
|+++ b/www/gitlab-ce/Makefile
--------------------------
Patching file www/gitlab-ce/Makefile using Plan A...
Hunk #1 succeeded at 3.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/www/gitlab-ce/files/gitlab.in b/www/gitlab-ce/files/gitlab.in
|index a59f45ea62..f6a482f083 100644
|--- a/www/gitlab-ce/files/gitlab.in
|+++ b/www/gitlab-ce/files/gitlab.in
--------------------------
Patching file www/gitlab-ce/files/gitlab.in using Plan A...
Hunk #1 succeeded at 49.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/www/gitlab-ce/files/patch-bin_actioncable b/www/gitlab-ce/files/patch-bin_actioncable
|new file mode 100644
|index 0000000000..f712a282eb
|--- /dev/null
|+++ b/www/gitlab-ce/files/patch-bin_actioncable
--------------------------
(Creating file www/gitlab-ce/files/patch-bin_actioncable...)
Patching file www/gitlab-ce/files/patch-bin_actioncable using Plan A...
Empty context always matches.
Hunk #1 succeeded at 1.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/www/gitlab-ce/files/patch-bin_web__puma b/www/gitlab-ce/files/patch-bin_web__puma
|new file mode 100644
|index 0000000000..528bfe4ce1
|--- /dev/null
|+++ b/www/gitlab-ce/files/patch-bin_web__puma
--------------------------
(Creating file www/gitlab-ce/files/patch-bin_web__puma...)
Patching file www/gitlab-ce/files/patch-bin_web__puma using Plan A...
Empty context always matches.
Hunk #1 succeeded at 1.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/www/gitlab-ce/files/patch-config_puma.rb.example b/www/gitlab-ce/files/patch-config_puma.rb.example
|index 1e8a8a51bd..24928ff974 100644
|--- a/www/gitlab-ce/files/patch-config_puma.rb.example
|+++ b/www/gitlab-ce/files/patch-config_puma.rb.example
--------------------------
Patching file www/gitlab-ce/files/patch-config_puma.rb.example using Plan A...
Hunk #1 succeeded at 1.
Hunk #2 succeeded at 32.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/www/rubygem-gitlab-puma/Makefile b/www/rubygem-gitlab-puma/Makefile
|index 7790f07524..f4f85f7d2d 100644
|--- a/www/rubygem-gitlab-puma/Makefile
|+++ b/www/rubygem-gitlab-puma/Makefile
--------------------------
Patching file www/rubygem-gitlab-puma/Makefile using Plan A...
Hunk #1 succeeded at 3.
Hunk #2 succeeded at 23.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/www/rubygem-gitlab-puma/files/patch-gitlab-puma.gemspec b/www/rubygem-gitlab-puma/files/patch-gitlab-puma.gemspec
|new file mode 100644
|index 0000000000..5fef51f7d4
|--- /dev/null
|+++ b/www/rubygem-gitlab-puma/files/patch-gitlab-puma.gemspec
--------------------------
(Creating file patch-gitlab-puma.gemspec...)
Patching file patch-gitlab-puma.gemspec using Plan A...
Empty context always matches.
Hunk #1 succeeded at 1.
Hmm...  Ignoring the trailing garbage.
done

sudo portmaster -d www/gitlab-ce

===>>> Currently installed version: gitlab-ce-13.2.3
===>>> Port directory: /usr/ports/www/gitlab-ce

===>>> Gathering distinfo list for installed ports

===>>> Launching 'make checksum' for www/gitlab-ce in background
===>>> Gathering dependency list for www/gitlab-ce from ports
===>>> Launching child to update rubygem-gitlab-puma-4.3.3.g.2 to rubygem-gitlab-puma-4.3.3.g.2_1

===>>> gitlab-ce-13.2.3 >> rubygem-gitlab-puma-4.3.3.g.2 (1/1)

===>>> Currently installed version: rubygem-gitlab-puma-4.3.3.g.2
===>>> Port directory: /usr/ports/www/rubygem-gitlab-puma

===>>> Launching 'make checksum' for www/rubygem-gitlab-puma in background
===>>> Gathering dependency list for www/rubygem-gitlab-puma from ports
===>>> Initial dependency check complete for www/rubygem-gitlab-puma

===>>> Continuing initial dependency check for www/gitlab-ce
===>>> Initial dependency check complete for www/gitlab-ce


===>>> gitlab-ce-13.2.3 >> (1)

===>>> The following actions will be taken if you choose to proceed:
        Upgrade gitlab-ce-13.2.3 to gitlab-ce-13.2.3_1
        Upgrade rubygem-gitlab-puma-4.3.3.g.2 to rubygem-gitlab-puma-4.3.3.g.2_1

===>>> Proceed? y/n [y]


===>>> Starting build for www/gitlab-ce <<<===

===>>> Starting check for build dependencies
===>>> Gathering dependency list for www/gitlab-ce from ports
===>>> Launching child to update rubygem-gitlab-puma-4.3.3.g.2 to rubygem-gitlab-puma-4.3.3.g.2_1

===>>> gitlab-ce-13.2.3 >> rubygem-gitlab-puma-4.3.3.g.2 (1/1)

===>>> Currently installed version: rubygem-gitlab-puma-4.3.3.g.2
===>>> Port directory: /usr/ports/www/rubygem-gitlab-puma

===>>> Starting check for build dependencies
===>>> Gathering dependency list for www/rubygem-gitlab-puma from ports
===>>> Dependency check complete for www/rubygem-gitlab-puma

===>>> gitlab-ce-13.2.3 >> rubygem-gitlab-puma-4.3.3.g.2 (1/1)

===>  Cleaning for rubygem-gitlab-puma-4.3.3.g.2_1
===>  License BSD3CLAUSE accepted by the user
===>   rubygem-gitlab-puma-4.3.3.g.2_1 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by rubygem-gitlab-puma-4.3.3.g.2_1 for building
===>  Extracting for rubygem-gitlab-puma-4.3.3.g.2_1
=> SHA256 Checksum OK for rubygem/gitlab-puma-4.3.3.gitlab.2.gem.
===>   rubygem-gitlab-puma-4.3.3.g.2_1 depends on file: /usr/local/bin/ruby26 - found
===>   rubygem-gitlab-puma-4.3.3.g.2_1 depends on file: /usr/local/bin/gem26 - found
Unpacked gem: '/usr/ports/www/rubygem-gitlab-puma/work/gitlab-puma-4.3.3.gitlab.2'
===>  Patching for rubygem-gitlab-puma-4.3.3.g.2_1
===>   rubygem-gitlab-puma-4.3.3.g.2_1 depends on file: /usr/local/bin/ruby26 - found
/bin/mv /usr/ports/www/rubygem-gitlab-puma/work/gitlab-puma-4.3.3.gitlab.2/bin/puma /usr/ports/www/rubygem-gitlab-puma/work/gitlab-puma-4.3.3.gitlab.2/bin/gpuma
/bin/mv /usr/ports/www/rubygem-gitlab-puma/work/gitlab-puma-4.3.3.gitlab.2/bin/pumactl /usr/ports/www/rubygem-gitlab-puma/work/gitlab-puma-4.3.3.gitlab.2/bin/gpumactl
===>   rubygem-gitlab-puma-4.3.3.g.2_1 depends on file: /usr/local/bin/ruby26 - found
===>   rubygem-gitlab-puma-4.3.3.g.2_1 depends on file: /usr/local/bin/gem26 - found
===>   rubygem-gitlab-puma-4.3.3.g.2_1 depends on file: /usr/local/bin/ccache - found
===>  Configuring for rubygem-gitlab-puma-4.3.3.g.2_1
===>  Building for rubygem-gitlab-puma-4.3.3.g.2_1
ERROR:  While executing gem ... (Errno::ENOENT)
    No such file or directory @ rb_file_s_lstat - bin/puma
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

Stop.
make: stopped in /usr/ports/www/rubygem-gitlab-puma

===>>> make build failed for www/rubygem-gitlab-puma
===>>> Aborting update

===>>> Update for rubygem-gitlab-puma-4.3.3.g.2 failed
===>>> Aborting update


===>>> You can restart from the point of failure with this command line:
       portmaster <flags> www/gitlab-ce www/rubygem-gitlab-puma

This command has been saved to ~/portmasterfail.txt
Comment 5 Matthias Fechner freebsd_committer 2020-08-11 11:37:27 UTC
Make sure this file exists:
https://gitlab.fechner.net/mfechner/Gitlab/-/tree/13.2/www/rubygem-gitlab-puma/files

It is maybe better to apply the patch with:
curl https://gitlab.fechner.net/mfechner/Gitlab/-/commit/1ef39c17382bb6c2e9b8e8a943c3ab494c6f4379.patch |git apply
Comment 6 hakan 2020-08-11 13:09:21 UTC
I had to fiddle around a bit, but after adding the `--unsafe-paths` option to `git apply`, everything else went fine. GitLab is now running on Puma. :)

I also changed the number of my worker processes according to https://docs.gitlab.com/ee/install/requirements.html#puma-settings in my `puma.rb` config file.
Comment 7 commit-hook freebsd_committer 2020-08-11 14:15:24 UTC
A commit references this bug:

Author: mfechner
Date: Tue Aug 11 14:15:02 UTC 2020
New revision: 544686
URL: https://svnweb.freebsd.org/changeset/ports/544686

Log:
  Switch gitlab internal webserver to puma.
  Gitlab switched with version 13.0 to puma as default.

  PR:		247564

Changes:
  head/www/gitlab-ce/Makefile
  head/www/gitlab-ce/files/gitlab.in
  head/www/gitlab-ce/files/patch-bin_actioncable
  head/www/gitlab-ce/files/patch-bin_web__puma
  head/www/gitlab-ce/files/patch-config_puma.rb.example
  head/www/rubygem-gitlab-puma/Makefile
  head/www/rubygem-gitlab-puma/files/
  head/www/rubygem-gitlab-puma/files/patch-gitlab-puma.gemspec
Comment 8 Matthias Fechner freebsd_committer 2020-08-11 14:15:46 UTC
Thanks, I committed it now.