Created attachment 242113 [details] FrozenError log I've just installed latest gitlab-ce package, 15.10.6, but it seems to be at least in since 15.10.3, since I jumped from 15.5.x to 15.9, then 15.10 since this week-end. The bug I'm reporting has somewhat to do with FrozenError, and frozen_string_literal in ruby. I've attached the log to this report. This error happens when: - you have a shared runner (gitlab runner) registered in the admin - you try to access: http://gitlab.example.com/groupname/projectname/-/settings/ci_cd An error 500 is showing on the ci/cd project ci/cd settings page, for every project in your gitlab instance. I've tried to look for this bug, search gitlab for that, I've also tried some of the "recommended" resolution: - some sql command removing runners_token and runners_token_encrypted values - some sql command to "finish" an unfinished upgrade path - removing, adding the runner again - reinstalling gitlab-ce package after backup'ing important files, and removing the complete /usr/local/www/gitlab-ce folder - trying to set frozen_string_literal to false in the runners_helper.rb I'm still at some point where I don't get why there is that frozen error, it was working well for gitlab-ce 15.9.x I think, since my company dev teams were adding things in the ci/cd settings before I upgraded to 15.10.x this last week-end. I'd really like some help here please, obviously because I can't go back to 15.9.x now that I've upgraded/autoremoved packages as recommended. Our company dev team is now blocked since they can't add ci/cd settings to their projects that we are trying to test in the hope we can make it to production. I've got 2 installations: - a test instance where I usually test gitlab-ce upgrades, since it's not always working great, and need a stable production instance. It's a jail with all inside: gitlab-ce and postgresql - a production instance, which consists of one server installed with gitlab-ce, and a jail in which postgresql is installed. Both, with 15.10.x different versions: production is 15.10.3_3, and test instance is 15.10.6. Also both have this same frozen error.
Thanks a lot for your very detailed report. I can reproduce the problem and I will try now to build a tree with commit c9b398d5a82c306121229a3fb9a3e84e5ea09f8b. But as this will consume a lot of resources, this will take some days to find where it stopped to work exactly. I post my progress here, so you are up-to-date.
ok, just for reference: git bisect start git bisect bad c9b398d5a82c306121229a3fb9a3e84e5ea09f8b git bisect good 40e3c194a360705948f35705a188b547a01971fd
Not sure if this will bring the correct commit that causes the problem, currently I have this: git bisect start # status: waiting for both good and bad commits # bad: [c9b398d5a82c306121229a3fb9a3e84e5ea09f8b] www/gitlab-ce: security update to 15.10.6 git bisect bad c9b398d5a82c306121229a3fb9a3e84e5ea09f8b # status: waiting for good commit(s), bad commit known # good: [40e3c194a360705948f35705a188b547a01971fd] www/gitlab-ce: bugfix update to 15.10.3 git bisect good 40e3c194a360705948f35705a188b547a01971fd # bad: [f001ef66d9190ce02c6a11fb39a5ee4bb94117ff] misc/trurl: Add new port git bisect bad f001ef66d9190ce02c6a11fb39a5ee4bb94117ff # good: [a4724ab76b0ed17b34adc209c04faaf5005cf93b] www/xcaddy: update to version 0.3.3 git bisect good a4724ab76b0ed17b34adc209c04faaf5005cf93b # skip: [07555752a580b9c1c01cfccb049135cb4bfd3321] textproc/p5-CAM-PDF: Add NO_ARCH git bisect skip 07555752a580b9c1c01cfccb049135cb4bfd3321 # skip: [14952ad6fc0c88a3a987905b3db21c2778efb12c] devel/p5-MooseX-Object-Pluggable: Add NO_ARCH git bisect skip 14952ad6fc0c88a3a987905b3db21c2778efb12c # skip: [19e8304345f46039fbcf33afc85b386939fe58d7] www/rubygem-ruby-openai: Add rubygem-ruby-openai 3.7.0 git bisect skip 19e8304345f46039fbcf33afc85b386939fe58d7 # skip: [e4e81c046c0ee9267ed28fd65e6970483d425017] devel/maven-wrapper: Add NO_ARCH git bisect skip e4e81c046c0ee9267ed28fd65e6970483d425017 # good: [c403c2461db3c2d08580bd99a7a5ef61b61a6f6f] devel/rubygem-aws-sdk-directoryservice: Update to 1.54.0 git bisect good c403c2461db3c2d08580bd99a7a5ef61b61a6f6f # bad: [1d338b269562de2e807a6fa25bcfd6a9d352b3b4] lang/nickle: Update to 2.91 git bisect bad 1d338b269562de2e807a6fa25bcfd6a9d352b3b4
Ok here an update of my findings: git bisect start # status: waiting for both good and bad commits # bad: [c9b398d5a82c306121229a3fb9a3e84e5ea09f8b] www/gitlab-ce: security update to 15.10.6 git bisect bad c9b398d5a82c306121229a3fb9a3e84e5ea09f8b # status: waiting for good commit(s), bad commit known # good: [40e3c194a360705948f35705a188b547a01971fd] www/gitlab-ce: bugfix update to 15.10.3 git bisect good 40e3c194a360705948f35705a188b547a01971fd # bad: [f001ef66d9190ce02c6a11fb39a5ee4bb94117ff] misc/trurl: Add new port git bisect bad f001ef66d9190ce02c6a11fb39a5ee4bb94117ff # good: [a4724ab76b0ed17b34adc209c04faaf5005cf93b] www/xcaddy: update to version 0.3.3 git bisect good a4724ab76b0ed17b34adc209c04faaf5005cf93b # skip: [07555752a580b9c1c01cfccb049135cb4bfd3321] textproc/p5-CAM-PDF: Add NO_ARCH git bisect skip 07555752a580b9c1c01cfccb049135cb4bfd3321 # skip: [14952ad6fc0c88a3a987905b3db21c2778efb12c] devel/p5-MooseX-Object-Pluggable: Add NO_ARCH git bisect skip 14952ad6fc0c88a3a987905b3db21c2778efb12c # skip: [19e8304345f46039fbcf33afc85b386939fe58d7] www/rubygem-ruby-openai: Add rubygem-ruby-openai 3.7.0 git bisect skip 19e8304345f46039fbcf33afc85b386939fe58d7 # skip: [e4e81c046c0ee9267ed28fd65e6970483d425017] devel/maven-wrapper: Add NO_ARCH git bisect skip e4e81c046c0ee9267ed28fd65e6970483d425017 # good: [c403c2461db3c2d08580bd99a7a5ef61b61a6f6f] devel/rubygem-aws-sdk-directoryservice: Update to 1.54.0 git bisect good c403c2461db3c2d08580bd99a7a5ef61b61a6f6f # bad: [1d338b269562de2e807a6fa25bcfd6a9d352b3b4] lang/nickle: Update to 2.91 git bisect bad 1d338b269562de2e807a6fa25bcfd6a9d352b3b4 # skip: [d326632501e7405ee0076675a6f70f34d24efc76] www/gitlab-ce: Change RUN_DEPENDS from rubygem-pg to rubygem-pg14 git bisect skip d326632501e7405ee0076675a6f70f34d24efc76 # skip: [ab2b77962f277fb78deb3535912c808099d20bb7] science/cdo: Fix PLIST git bisect skip ab2b77962f277fb78deb3535912c808099d20bb7 # bad: [99bf4287a9247fca2810b271bb35086888669c5b] devel/rubygem-rbs: Update to 3.1.0 git bisect bad 99bf4287a9247fca2810b271bb35086888669c5b # good: [fea40d5ca16b02bd2e2b25ea8ba55ed59640c396] devel/rubygem-google-apis-cloudresourcemanager_v3: Update to 0.43.0 git bisect good fea40d5ca16b02bd2e2b25ea8ba55ed59640c396 # good: [bfe48e6c54c0ede41ad0f3108f2f8940821d9d3e] devel/rubygem-googleapis-common-protos-types: Update to 1.6.0 git bisect good bfe48e6c54c0ede41ad0f3108f2f8940821d9d3e # bad: [5040e90318995f335402e8aec3789405701df93c] devel/rubygem-parser: Update to 3.2.2.1 git bisect bad 5040e90318995f335402e8aec3789405701df93c # bad: [af67981180d49f24b1754433a281e8d79cceaf1c] devel/rubygem-i18n: Update to 1.13.0 git bisect bad af67981180d49f24b1754433a281e8d79cceaf1c # first bad commit: [af67981180d49f24b1754433a281e8d79cceaf1c] devel/rubygem-i18n: Update to 1.13.0 af67981180d49f24b1754433a281e8d79cceaf1c is the first bad commit commit af67981180d49f24b1754433a281e8d79cceaf1c Author: Po-Chuan Hsieh <sunpoet@FreeBSD.org> Date: Fri Apr 28 07:59:24 2023 +0800 devel/rubygem-i18n: Update to 1.13.0 Changes: https://github.com/ruby-i18n/i18n/releases devel/rubygem-i18n/Makefile | 2 +- devel/rubygem-i18n/distinfo | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) So it seems that the commit that causes this problems is: https://cgit.freebsd.org/ports/commit/?id=af67981180d49f24b1754433a281e8d79cceaf1c I looked a little bit around in the issue list: https://github.com/ruby-i18n/i18n/issues And found this issue: https://github.com/ruby-i18n/i18n/issues/648 It could be related. It was fixed by this merge: https://github.com/ruby-i18n/i18n/pull/587 So I would try first to maybe apply the patch there manually and regenerate all assets again, like described here: https://gitlab.fechner.net/mfechner/Gitlab-docu/-/blob/master/update/freebsd_patch_versions.md I will now work on a security update for gitlab and will have later a look into it.
Can you please restest with gitlab 15.11.3, I currently are not able to reproduce the problem there.
(In reply to Matthias Fechner from comment #5) It is reproduceable, it just took a big amount of time and clicking around to see the error again.
I just did now a downgrade of rubygem-i18n to version 1.12.0 and it seems to run fine. I did this downgrade using my poudriere build system, but it should work also if you do something like this (untested, but should work). Go into /usr/ports/devel/rubygem-i18n and edit the Makefile. Put as PORTVERSION 1.12.0. make makesum pkg delete -f rubygem-i18n make install Lock the package: pkg lock rubygem-i18n Then follow the steps in the patch manual: https://gitlab.fechner.net/mfechner/Gitlab-docu/-/blob/master/update/freebsd_patch_versions.md Let me know if that also fixes your problem.
(In reply to Matthias Fechner from comment #7) Hello, sadly it does not fix the problem on my side. I've created the package for rubygem-i18n version 1.12.0, installed it in my test env for gitlab, used your documentation page. test env gitlab-ce version is still 15.10.6. The error in the log/production.log file looks the same.
(In reply to Guillaume Bibaut from comment #8) it seems like I've messed up something this morning... I've tried your steps again on a new gitlab instance (at home), and it's working alright. Reproduced the exact same steps on the test instance at work, it's working too. I've created the package rubygem-i18n for 13.1-RELEASE, and I'm waiting to test these steps when noone is working on the production server. I'll keep you updated asap.
(In reply to Guillaume Bibaut from comment #9) rubygem-i18n 1.12.0 fixed it for me. Thank you.
(In reply to Guillaume Bibaut from comment #10) nice to hear this. I keep this report open till sunpoet found a fix for rubygem-i18n port or a downgrade was done I hope that helps other users to work-around the problem too.
I can confirm that the workaround given by Matthias in comment #7 fixed this problem for me (gitlab-ce 15.11.3_2 installed from packages on 24 May 2023).
I can confirm that the downgrade to 1.12.0 of rubygen-i18n fixes the issue for GitLab 16.0.1 :)
Created attachment 242550 [details] Fix frozen string I tested it with this patch and could not reproduce the error anymore.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=d9c7bd925458d6d51ddbdafe03cf1e5a1a422c16 commit d9c7bd925458d6d51ddbdafe03cf1e5a1a422c16 Author: Matthias Fechner <mfechner@FreeBSD.org> AuthorDate: 2023-06-02 06:10:14 +0000 Commit: Matthias Fechner <mfechner@FreeBSD.org> CommitDate: 2023-06-06 03:32:56 +0000 devel/rubygem-i18n: fix frozen string With version v1.13.0 code was changed that returns a frozen string. This breaks e.g. gitlab-ce on some pages. Geoffrey Mainland (mainland@apeiron.net) pointed to the issue logged upstream: https://github.com/ruby-i18n/i18n/issues/658 PR: 271420 PR: 271356 Approved by: sunpoet (maintainer) devel/rubygem-i18n/Makefile | 1 + devel/rubygem-i18n/files/patch-bug-271420 (new) | 11 +++++++++++ 2 files changed, 12 insertions(+)
The included patch should fix the problem.
*** Bug 271858 has been marked as a duplicate of this bug. ***