Aloha Matthias, after the upgrade from 11.8.x to 11.9.4 (now 11.9.7) the Pipelines for CI / CD do not work any more. I get a 500 error. In production.log there is this entry: Completed 500 Internal Server Error in 17392ms (ActiveRecord: 17063.0ms) NoMethodError (undefined method `delete_suffix' for "web":String): lib/gitlab/ci/build/policy/refs.rb:47:in `sanitized_source_name' lib/gitlab/ci/build/policy/refs.rb:32:in `matches_pattern?' lib/gitlab/ci/build/policy/refs.rb:17:in `block in satisfied_by?' lib/gitlab/ci/build/policy/refs.rb:13:in `any?' lib/gitlab/ci/build/policy/refs.rb:13:in `satisfied_by?' lib/gitlab/ci/pipeline/seed/build.rb:24:in `block (2 levels) in included?' lib/gitlab/ci/pipeline/seed/build.rb:24:in `each' lib/gitlab/ci/pipeline/seed/build.rb:24:in `all?' lib/gitlab/ci/pipeline/seed/build.rb:24:in `block in included?' lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize' lib/gitlab/ci/pipeline/seed/build.rb:23:in `included?' lib/gitlab/ci/pipeline/seed/stage.rb:31:in `select' lib/gitlab/ci/pipeline/seed/stage.rb:31:in `block in seeds' lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize' lib/gitlab/ci/pipeline/seed/stage.rb:30:in `seeds' lib/gitlab/ci/pipeline/seed/stage.rb:36:in `included?' app/models/ci/pipeline.rb:489:in `select' app/models/ci/pipeline.rb:489:in `block in stage_seeds' lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize' app/models/ci/pipeline.rb:484:in `stage_seeds' lib/gitlab/ci/pipeline/chain/populate.rb:28:in `perform!' lib/gitlab/ci/pipeline/chain/sequence.rb:19:in `block in build!' lib/gitlab/ci/pipeline/chain/sequence.rb:16:in `each' lib/gitlab/ci/pipeline/chain/sequence.rb:16:in `build!' app/services/ci/create_pipeline_service.rb:47:in `execute' app/controllers/projects/pipelines_controller.rb:56:in `create' lib/gitlab/i18n.rb:55:in `with_locale' lib/gitlab/i18n.rb:61:in `with_user_locale' app/controllers/application_controller.rb:434:in `set_locale' lib/gitlab/middleware/multipart.rb:103:in `call' lib/gitlab/request_profiler/middleware.rb:16:in `call' lib/gitlab/middleware/go.rb:20:in `call' lib/gitlab/etag_caching/middleware.rb:13:in `call' lib/gitlab/middleware/correlation_id.rb:16:in `block in call' lib/gitlab/correlation_id.rb:15:in `use_id' lib/gitlab/middleware/correlation_id.rb:15:in `call' lib/gitlab/middleware/read_only/controller.rb:40:in `call' lib/gitlab/middleware/read_only.rb:18:in `call' lib/gitlab/middleware/basic_health_check.rb:25:in `call' lib/gitlab/request_context.rb:26:in `call' lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call' lib/gitlab/middleware/release_env.rb:13:in `call' Do you have any idea or fix for this? Greetings, Torsten
They are working fine here. Have you did all the upgrades in the config files? Have you also execute pkg autoremove?
Yes, i did all of this. Just to be sure i re-did everything. Now i get an error when compiling the assets: # su -l git -c "cd /usr/local/www/gitlab-ce && rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production" yarn install v1.15.2 [1/4] Resolving packages... warning Resolution field "ts-jest@24.0.0" is incompatible with requested version "ts-jest@^23.10.5" success Already up-to-date. $ node ./scripts/frontend/postinstall.js success Dependency postinstall check passed. Done in 0.63s. W, [2019-04-12T15:29:08.675434 #14554] WARN -- : Please set prometheus_multiproc_dir to persistent directory path. Temporarily writing mmap files to /tmp/prometheus-mmap20190412-14554-1cav7qj warning Resolution field "ts-jest@24.0.0" is incompatible with requested version "ts-jest@^23.10.5" `yarn:check` finished in 1.754446892 seconds Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/es Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/cy_GB Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/el_GR Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/sr_SP Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/hi_IN Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/pl_PL Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/nb_NO Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/pa_IN Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/da_DK Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/mn_MN Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/pt_BR Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/pt_PT Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/hr_HR Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/fil_PH Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/bg Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/hu_HU Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/en Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/fr Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/et_EE Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/sq_AL Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/he_IL Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/sr_CS Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/sk_SK Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/ro_RO Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/ko Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/ru Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/cs_CZ Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/zh_TW Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/zh_HK Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/de Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/ja Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/gl_ES Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/uk Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/zh_CN Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/ca_ES Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/tr_TR Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/sv_SE Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/ar_SA Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/id_ID Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/sw_KE Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/it Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/nl_NL Created app.js in /usr/local/www/gitlab-ce/app/assets/javascripts/locale/eo All files created, make sure they are being added to your assets. If they are not, you can add them with this line (configurable): All files created, make sure they are being added to your assets. If they are not, you can add them with this line (configurable): //= require_tree ./locale //= require gettext/all `gettext:po_to_json` finished in 3.955330219 seconds `rake:assets:precompile` finished in 2.688036122 seconds /usr/local/www/gitlab-ce/node_modules/.bin/webpack --config /usr/local/www/gitlab-ce/config/webpack.config.js --bail <--- Last few GCs ---> io[14664:0x804c7d880] 114600 ms: Mark-sweep 1382.5 (1440.2) -> 1382.5 (1440.2) MB, 633.2 / 0.0 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 634 ms) (average mu = 0.069, current mu = 0.002) allocatio[14664:0x804c7d880] 115254 ms: Mark-sweep 1382.5 (1440.2) -> 1382.5 (1440.2) MB, 652.6 / 0.0 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 654 ms) (average mu = 0.036, current mu = 0.002) last reso <--- JS stacktrace ---> ==== JS stack trace ========================================= 0: ExitFrame [pc: 0x2313ac54fc7d] Security context: 0x161e2091d9d1 <JSObject> 1: /* anonymous */ [0x1dda299fccb1] [/usr/local/www/gitlab-ce/node_modules/source-map/lib/source-node.js:~86] [pc=0x2313ad867c5b](this=0x1890b1060ff1 <JSFunction SourceNode (sfi = 0x2aad87572e81)>,0x3018d6f136c1 <Object map = 0x2c85b5d82521>) 2: arguments adaptor frame: 3->1 3: forEach [0x161e20906d59](this=0x2a6efa502241 <JSArray[819520]>,0x1dda... FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0x926600 node::Abort(void) [/usr/local/bin/node] 2: 0x926ca0 node::OnFatalError(char const*, char const) [/usr/local/bin/node] 3: 0xa82d33 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node] 4: 0xa82cdc v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node] 5: 0xe355d2 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node] 6: 0xe34490 v8::internal::Heap::OldGenerationSizeOfObjects(void) [/usr/local/bin/node] 7: 0xe32245 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node] 8: 0xe326c5 v8::internal::Heap::CollectAllAvailableGarbage(v8::internal::GarbageCollectionReason) [/usr/local/bin/node] 9: 0xe3d9b1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node] 10: 0xe246bc v8::internal::Factory::NewLoadHandler(int) [/usr/local/bin/node] 11: 0xea067a v8::internal::MaybeObjectHandle v8::internal::Isolate*<v8::internal::JSReceiver> v8::internal::LoadHandler::LoadFromPrototype(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>(v8::internal::Isolate*<v8::internal::Smi>(v8::internal::Isolate*<v8::internal::Smi>) [/usr/local/bin/node] 12: 0xea5ad6 v8::internal::LoadIC::ComputeHandler(v8::internal::LookupIterator*) [/usr/local/bin/node] 13: 0xea4a4d v8::internal::LoadIC::UpdateCaches(v8::internal::LookupIterator*) [/usr/local/bin/node] 14: 0xea46ff v8::internal::LoadIC::Load<v8::internal::Name> v8::internal::LoadIC::Load(v8::internal::Handle<v8::internal::Object>() [/usr/local/bin/node] 15: 0xeab0bc v8::internal::Runtime_LoadIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node] rake aborted! Command failed with status (): [/usr/local/www/gitlab-ce/node_modules/.bin...] /usr/local/www/gitlab-ce/lib/gitlab/task_helpers.rb:18:in `invoke_and_time_task' /usr/local/www/gitlab-ce/lib/tasks/gitlab/assets.rake:13:in `each' /usr/local/www/gitlab-ce/lib/tasks/gitlab/assets.rake:13:in `block (3 levels) in <top (required)>' Tasks: TOP => webpack:compile (See full trace by running task with --trace)
Execute the command noted in the manual: # in case you see a out of heap memory error you can try: Make sure your gitaly config is correct, there was a change necessary to get the pipelines working again. If you modified the file, the upgrade maybe does not touch it anymore.
> Execute the command noted in the manual: > # in case you see a out of heap memory error you can try: This worked. :) > Make sure your gitaly config is correct, there was a change necessary to get > the pipelines working again. > If you modified the file, the upgrade maybe does not touch it anymore. I didn't touch it before. I also executed the git diff from your manual, but no changes were shown. The error still persists.
Created attachment 203617 [details] Pipeline We are talking about these pipelines or?
Maybe related to this: https://about.gitlab.com/2019/04/01/security-release-gitlab-11-dot-9-dot-4-released/
(In reply to Matthias Fechner from comment #5) > We are talking about these pipelines or? Yes. I'm trying to use the button "Run Pipelines". The next site will be shown, but when creating the action i get a 500 error page. Also no pipelines are created when pushing.
Just to be very sure: i completely installed GitLab source from the scratch (only keeping /home/git and the config files database.yml, secrets.yml and gitlab.yml). The error still persists :/
ok, then we should look and build options. Are you sure that option RAILS50 is selected for port devel/rubygem-peek-rails50?
And do you use postgresql 9.6 which is the minimum version required for gitlab?
(In reply to Matthias Fechner from comment #9) > ok, then we should look and build options. > Are you sure that option RAILS50 is selected for port > devel/rubygem-peek-rails50? Yes. This is my current option: # This file is auto-generated by 'make config'. # Options for rubygem-peek-rails50-1.0.1_2 _OPTIONS_READ=rubygem-peek-rails50-1.0.1_2 _FILE_COMPLETE_OPTIONS_LIST= RAILS4 RAILS5 RAILS50 OPTIONS_FILE_UNSET+=RAILS4 OPTIONS_FILE_UNSET+=RAILS5 OPTIONS_FILE_SET+=RAILS50 (In reply to Matthias Fechner from comment #10) > And do you use postgresql 9.6 which is the minimum version required for gitlab? Yes, i'm using PostgreSQL 10. My make.conf file contains this line: DEFAULT_VERSIONS= pgsql=10
Is it possible to retest it with postgresql 9.6? I had put the version explicitly into the manual: https://gitlab.fechner.net/mfechner/Gitlab-docu/blob/master/install/11.9-freebsd.md#2-database I remember that there were problems with an older version and gitlab only supports version 9.6. I'm not sure if a newer version will work, maybe postgresql had here a breaking change with version 10 compared to 9.6.
While a test would be possible i really doubt, that this will be the cause. I'm PostgreSQL 10 for more than a year for this GitLab installation. Also it just broke with the update to 10.8. Also the error message "NoMethodError (undefined method `delete_suffix' for "web":String):" does not really sound like an database error :D
Maybe you can address this upstream? I do not have the problem in 3 gitlab installations, so it is not possible for me to reproduce it. Pipelines are working fine in all installation. I do trigger them by commits. But also the Pipeline button you mentioned works fine here.
Seems upstream is the way to go. :/ Can you provide me your current pkg info list? I want to double check this. If this doesn't work i could try to downgrade it and see if this helps.
Here is the last buildlog for gitlab-ce (version 11.9.7) which is installed on all my systems: https://pkg.fechner.net/data/120amd64-default/2019-04-11_19h16m36s/logs/gitlab-ce-11.9.7.log Besides this: postgresql96-client-9.6.12_1 PostgreSQL database (client) postgresql96-contrib-9.6.12_1 The contrib utilities from the PostgreSQL distribution postgresql96-server-9.6.12_1 PostgreSQL is the most advanced open-source database available anywhere nginx-1.14.2_13,2 Robust and small WWW server
Please can i have pkg info? :D The build-log is very good, but i can't use a simple diff to spot any difference.
Sent by email.
I actually managed to fix the bug. And the bug is in the ports-tree. Background for the bug is here: https://docs.gitlab.com/ee/install/installation.html#2-ruby Currently Ruby 2.5 is required by GitLab, support for default of 2.4 was dropped with GitLab 11.6. The method "delete_suffix" was introduced with Ruby 2.5. So i got this message, because Ruby 2.4 does not have this method. After upgrading to Ruby 2.5 everything is fine again :)
Thanks Thorsten, this is really very valuable information! I have ruby 2.4.6 installed and it works for an unknown reason for me on more than one installation. If I execute: su -l git -c "cd /usr/local/www/gitlab-ce && rake gitlab:check RAILS_ENV=production" it tells me: Ruby version >= 2.3.5 ? ... yes (2.4.6) I reported it here: https://gitlab.com/gitlab-org/gitlab-ce/issues/60704
Default version of ruby is now 2.5 (r499391).