Bug 248302 - www/gitlab-ce: upgrade to gitlab 13.2.1 breaks CI job display
Summary: www/gitlab-ce: upgrade to gitlab 13.2.1 breaks CI job display
Status: Closed Works As Intended
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Matthias Fechner
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-07-27 17:43 UTC by gwright
Modified: 2020-07-30 18:35 UTC (History)
2 users (show)

See Also:
linimon: maintainer-feedback? (mfechner)


Attachments
Gitlab 13.2.1 CI job display screenshot (691.40 KB, image/png)
2020-07-27 18:44 UTC, gwright
no flags Details
Gitlab 13.2.1 CI job display html page source (38.43 KB, text/plain)
2020-07-27 18:46 UTC, gwright
no flags Details
Gitlab 13.2.1 CI job downloaded 'raw' output (3.26 KB, text/plain)
2020-07-27 18:49 UTC, gwright
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description gwright 2020-07-27 17:43:57 UTC

    
Comment 1 gwright 2020-07-27 18:34:29 UTC
An upstream bug fix in gitlab-ce 13.2.1 requires users to revalidate their email addresses.  What is supposed to happen is that gitlab automatically generates an email to each user, and by responding, the user's account is reenabled.  If a user attempts to log in before reenabling his/her account, a 422 error is generated.

The issue is described here:
https://docs.gitlab.com/ee/user/upgrade_email_bypass.html

When I upgraded from 13.1.4 to 13.2.1, my user account was locked out (generating a 422 error) but it was still possible to log into the administrator account.  No email message was sent asking me to reconfirm my account, even though I used the administrator's console to run all the delayed tasks.

In my case, the workaround that succeeded was to log into the rails console and force reconfirmation of all users.  This is what I did:
{{{
root@gitlab:/usr/local/www/gitlab-ce # su -l git -c "cd /usr/local/www/gitlab-ce && rails console -e production"
--------------------------------------------------------------------------------
 GitLab:       13.2.1 (Unknown) FOSS
 GitLab Shell: 13.3.0
 PostgreSQL:   11.8
--------------------------------------------------------------------------------

Loading production environment (Rails 6.0.3.2)
irb(main):001:0> User.all.each  {|u| u.confirmed_at = Time.now; u.save}
=> [#<User id:1 @root>, #<User id:6 @alert-bot>, #<User id:5 @ghost>, #<User id:4 @gwright>]
irb(main):002:0> exit
}}}
The command that I entered at the prompt was `User.all.each {|u| u.confirmed_at = Time.now; u.save}`.  Once that ran and the prompt returned, I was able to type `exit`.  I could then log on to my non-adminstrator account.

The documentation for upgrading should probably include the above link to the gitlab documentation, which gives instructions for what to do if you are locked ou
Comment 2 gwright 2020-07-27 18:44:57 UTC
Created attachment 216813 [details]
Gitlab 13.2.1 CI job display screenshot
Comment 3 gwright 2020-07-27 18:46:57 UTC
Created attachment 216814 [details]
Gitlab 13.2.1 CI job display html page source
Comment 4 gwright 2020-07-27 18:49:38 UTC
Created attachment 216816 [details]
Gitlab 13.2.1 CI job downloaded 'raw' output
Comment 5 gwright 2020-07-27 18:57:02 UTC
Please ignore my first comment; it was inadvertently pasted from another bug report.

After upgrading to gitlab 13.2.1, the continuous integration (CI) job display page broke.  When a CI job runs, instead of the output being shown in black display window, there is only the occasional flashing of "undefined" which lasts until the job finishes running.  I have attached a screenshot which shows the web page.

The job does run successfully: I can download the raw output using the "show complete raw" at the top of the black display window and it is as expected.  I have attached it as well, with tokens changed to strings of Xs.

I have also attached the page source for the page in the screenshot.

Has the javascript for the web page display been mangled?
Comment 6 gwright 2020-07-27 18:57:39 UTC
(In reply to gwright from comment #1)
Ignore this comment; it was inadvertently pasted from another Gitlab bug report.
Comment 7 gwright 2020-07-28 13:56:32 UTC
To check whether some piece of javascript is missing or mangled, I ran:

root@gitlab:/usr/local/www/gitlab-ce # yarn check --production
yarn check v1.22.4
warning Resolution field "ts-jest@24.0.0" is incompatible with requested version "ts-jest@^23.10.5"
info fsevents@2.1.3: The platform "freebsd" is incompatible with this module.
info "fsevents@2.1.3" is an optional dependency and failed compatibility check. Excluding it from installation.
info fsevents@1.2.13: The platform "freebsd" is incompatible with this module.
info "fsevents@1.2.13" is an optional dependency and failed compatibility check. Excluding it from installation.
warning Resolution field "monaco-editor@0.20.0" is incompatible with requested version "monaco-yaml#monaco-editor@^0.19.2"
warning Resolution field "chokidar@3.4.0" is incompatible with requested version "watchpack#watchpack-chokidar2#chokidar@^2.1.8"
info Found 2 warnings.
success Folder in sync.
Done in 21.51s.
root@gitlab:/usr/local/www/gitlab-ce #

The warnings about 'ts-jest' and 'fsevents' have been present for many versions, and as far as I can tell, have not caused trouble.  The 'monaco-editor' and 'watchpack' warnings may be new.

I should also mention that I also cleaned and rebuilt all gitlab assets before running this check, which is recommended upstream.  The incorrect behavior is still there.
Comment 8 gwright 2020-07-28 14:29:14 UTC
More tests:  the incorrect behavior persists when I view the same CI display page on Firefox, Safari, Brave or Chrome.
Comment 9 gwright 2020-07-28 14:39:34 UTC
Same behavior on Microsoft Edge.
Comment 10 Matthias Fechner freebsd_committer 2020-07-28 15:51:08 UTC
Can you report this upstream, please?
And insert a link to it here?

Thanks a lot.
Comment 11 gwright 2020-07-28 15:58:01 UTC
I will report it upstream and link back here.
Comment 12 gwright 2020-07-28 22:46:46 UTC
This issue is reported upstream:

https://gitlab.com/gitlab-org/gitlab/-/issues/232747
Comment 13 gwright 2020-07-29 21:07:23 UTC
I uninstalled all ports, reinstalled everything as packages from the "latest" repo, then started postgresql and redis.  I checked to ensure that all
configuration files were carried over from the previous installation.

Then I ran:

su -l git -c "cd /usr/local/www/gitlab-ce && yarn install --production --pure-lockfile"
su -l git -c "cd /usr/local/www/gitlab-ce && RAILS_ENV=production NODE_ENV=production USE_DB=false SKIP_STORAGE_VALIDATION=true NODE_OPTIONS='--max_old_space_size=3584' bundle exec rake gitlab:assets:compile"

after which I started gitlab and nginx.  Gitlab started correctly and the
problem I had before with the CI job display window was gone.

I do not know why this worked, but I did observe that the asset compilation step
ran far faster (five or ten minutes instead of more than an hour) when I
did the reinstallation compared to the upgrade from 13.1.4 -> 13.2.1.  There
was also much more output than when I upgraded.  Perhaps some corruption of the
assets directory?  If this were the case, it is odd that compiling the assets
earlier did not throw an error.

This bug can be closed.
Comment 14 Matthias Fechner freebsd_committer 2020-07-30 04:07:02 UTC
Thanks.
You have followed the upgrade manual?
This command is sometimes crucial:
pkg autoremove
Comment 15 gwright 2020-07-30 13:55:03 UTC
(In reply to Matthias Fechner from comment #14)

My installation that showed the bug was from ports.  I had not run 'pkg autoremove' after upgrading via 'portmaster'.  The upgrade documentation
seems specific to installation from packages (which is one reason why I
reinstalled from packages, rather than ports).
Comment 16 Matthias Fechner freebsd_committer 2020-07-30 18:29:47 UTC
(In reply to gwright from comment #15)
This is not true. Even if you install your package from ports, pkg is used to manage the packages.
You just replace the pkg upgrade with your way to upgrade the packages, then you follow exactly the procedure (even if I not recommend this approach, it will cause many build deps on the system and other problems due to polluted build environment).
If you need packages build by yourself think about creating your own poudriere build and use this.
Comment 17 gwright 2020-07-30 18:35:01 UTC
So if I build from ports, the correct upgrade procedure is

    $ portmaster -aG --no-confirm
    $ pkg autoremove

?

I run gitlab-ce in a jail, so to upgrade I simply upgraded all ports (after upgrading the the basejail and using ezjail to propagate those changes).

As I mentioned, to fix my problem I switched to using packages, so I can now
run the documented procedure exactly.