Bug 263867

Summary: www/gitlab-ce v14.10.0 failures when pushing or pulling
Product: Ports & Packages Reporter: gwright
Component: Individual Port(s)Assignee: Matthias Fechner <mfechner>
Status: Closed FIXED    
Severity: Affects Some People CC: tz
Priority: --- Flags: bugzilla: maintainer-feedback? (mfechner)
Version: Latest   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
Include gitaly patches for git binary none

Description gwright 2022-05-08 17:42:18 UTC
Hi Matthais,

The recent gitlab-ce (v14.10.0) introduced an annoying bug.  It shows up as
failure to push/pull from a server running gitlab, and apparent corruption of
repositories on the Gitlab server.

Here are the observed symptoms.  When pushing, I get:

(base)  ~/src/vev-renormalization> git push
warning: core.fsyncObjectFiles is deprecated; use core.fsync instead
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 389 bytes | 389.00 KiB/s, done.
Total 3 (delta 2), reused 1 (delta 0), pack-reused 0
remote: warning: core.fsyncObjectFiles is deprecated; use core.fsync instead
remote: warning: core.fsyncObjectFiles is deprecated; use core.fsync instead
remote: fatal: commit-graph requires overflow generation data but has none
warning: core.fsyncObjectFiles is deprecated; use core.fsync instead
fatal: commit-graph requires overflow generation data but has none
remote: warning: core.fsyncObjectFiles is deprecated; use core.fsync instead
To ssh://<HOST REDACTED>/software/vev-renormalization.git
 ! [remote rejected] master -> master (missing necessary objects)
error: failed to push some refs to 'ssh://<HOST REDACTED>/software/vev-renormalization.git'


Going to the corresponding repository on the server, I see:

[git@gitlab ~/repositories/@hashed/92/2c/922c7954216ccfe7a61def609305ce1dc7c67e225f873f256d30d7a8ee4f404c.git]$ git fsck --full
Checking object directories: 100% (256/256), done.
fatal: commit-graph requires overflow generation data but has none
[git@gitlab ~/repositories/@hashed/92/2c/922c7954216ccfe7a61def609305ce1dc7c67e225f873f256d30d7a8ee4f404c.git]$ echo $?
128

After some searching, I found this recommendation: https://docs.gitlab.com/ee/install/installation.html#git

Now GitLab seems to depend on a version of git patched to work with a particular
version of gitaly.  In the jail in which I run gitlab, I did:

> cd /tmp
> git clone https://gitlab.com/gitlab-org/gitaly.git -b 14-10-stable /tmp/gitaly
> cd /tmp/gitaly
> gmake git GIT_PREFIX=/usr/local

This installed the patched version of git, overwriting the binaries installed
in /usr/local/by the git package.  After I did this, the repository checked
clean:

05ce1dc7c67e225f873f256d30d7a8ee4f404c.git # git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (107/107), done.
Verifying commits in commit graph: 100% (24/24), done.

Also after switching to the patched version of git, pushing from a remote
worked without error.

It is rather annoying that gitlab (or perhaps more specifically gitaly) can not
use the released git.  Let me know if I can run more experiments.  Reinstalling
git from packages seems to reliably recreate the failed state, at least for one
of my repositories.

BR,
Greg
Comment 1 Matthias Fechner freebsd_committer freebsd_triage 2022-05-27 18:43:43 UTC
Could you please retry with version 15.0.0?
The changelog is mentioning some bugfixes:
https://gitlab.com/gitlab-org/gitaly/-/blob/master/CHANGELOG.md

that could be related to the problem you describe.
Comment 2 Matthias Fechner freebsd_committer freebsd_triage 2022-06-06 06:15:31 UTC
Created attachment 234480 [details]
Include gitaly patches for git binary

You can try to apply this patch, it maybe fixes the problem that the repository got broken.

But I think you must restore the broken repository from a backup.

I have the same problem now on some repositories and just installing the new git version with the patch included does not fix it for the broken repositories.
But it maybe prevents it from happening again.
Comment 3 Torsten Zuehlsdorff freebsd_committer freebsd_triage 2022-06-06 17:53:09 UTC
This issue also happens in GitLab 15.0 (i got them after the update) and persists whenever using ti 2.36.1. 

There are upstream issues like this:
https://groups.google.com/g/linux.debian.bugs.dist/c/zbojhXyfFfY?pli=1
https://lore.kernel.org/git/f50e74f0-9ffa-f4f2-4663-269801495ed3@github.com/

If the issue happens to you, you can't:
* checkout a repo anymore
* the "Merge" Button in GitLab MR page checks for ever if a merge is possible
* The CI / CD page list a big error message

You can fix the issue by deleting the commit-graphs dir. Like this way:
rm -rfv objects/info/commit-graphs

I fixed all my 100+ repos with:
find /usr/local/git/repositories -name commit-graphs -type d | xargs rm -rfv

The issue will happen again, until the git is patched.
Comment 4 commit-hook freebsd_committer freebsd_triage 2022-06-07 05:15:32 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=57d9640f3c3669d318865222a743652a6952ba2d

commit 57d9640f3c3669d318865222a743652a6952ba2d
Author:     Matthias Fechner <mfechner@FreeBSD.org>
AuthorDate: 2022-06-06 06:02:05 +0000
Commit:     Matthias Fechner <mfechner@FreeBSD.org>
CommitDate: 2022-06-07 05:14:23 +0000

    devel/git: fix bug  commit-graph requires overflow generation data

    Using www/gitlab-ce I see problems that git repositories got broken.
    Gitlab includes some patches for the git binary they build:
    https://gitlab.com/gitlab-org/gitaly/-/tree/master/_support/git-patches

    These patches should have no impact to normal git usage but fixes some
    ref handling problems, like:
    https://gitlab.com/gitlab-org/gitaly/-/blob/master/_support/git-patches/v2.36.0.gl1/0006-refs-skip-hooks-when-deleting-uncovered-packed-refs.patch

    The problem got visible that at any point it can happen that the repository
    got broken (by merges happen) and if you try to fetch it you get:
    "C:\Program Files\Git\bin\git.exe" pull --progress "origin"
    fatal: commit-graph requires overflow generation data but has none
    fatal: the remote end hung up unexpectedly
    Done

    PR:             263867
    Approved by:    garga (maintainer)

 devel/git/Makefile | 12 ++++++++++++
 devel/git/distinfo | 14 +++++++++++++-
 2 files changed, 25 insertions(+), 1 deletion(-)
Comment 5 Matthias Fechner freebsd_committer freebsd_triage 2022-06-07 05:34:18 UTC
I got now permission to commit the patch for git, I hope that fixes it.
Comment 6 Dmitry Marakasov freebsd_committer freebsd_triage 2022-11-15 19:52:54 UTC
Please revert this ASAP.

First of all, the way these patches were added is just not acceptable: git is used for many kinds of sensitive data handling and unconditionally adding patches from untrusted third party is a security violation. These could've been under option, disabled by default (though I don't like the very idea of plugging bugs of unrelated software in git). Next, urls for these patches refer to git master branch on gitlab and are prone to breakage. In fact, they are all broken now. Upstream has removed these patches [claiming](https://gitlab.com/gitlab-org/gitaly/-/commit/248e71d0921d44a6fc3dfbdbb7402b02962bebf7) these are no longer needed, so let's clear our git port of these as well.
Comment 7 commit-hook freebsd_committer freebsd_triage 2022-11-16 12:23:04 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=52fafafdc0a2d5540d3228ea59261e79188de5e0

commit 52fafafdc0a2d5540d3228ea59261e79188de5e0
Author:     Dmitry Marakasov <amdmi3@FreeBSD.org>
AuthorDate: 2022-06-07 14:18:47 +0000
Commit:     Dmitry Marakasov <amdmi3@FreeBSD.org>
CommitDate: 2022-11-16 12:21:22 +0000

    devel/git: remove no longer needed third party patches

    - This reverts 57d9640f3c3669d318865222a743652a6952ba2d

    PR:             267792, 263867
    Approved by:    garga (maintainer)

 devel/git/Makefile | 13 +------------
 devel/git/distinfo | 12 ------------
 2 files changed, 1 insertion(+), 24 deletions(-)