Hi Matthais, The recent gitlab-ce (v14.10.0) introduced an annoying bug. It shows up as failure to push/pull from a server running gitlab, and apparent corruption of repositories on the Gitlab server. Here are the observed symptoms. When pushing, I get: (base) ~/src/vev-renormalization> git push warning: core.fsyncObjectFiles is deprecated; use core.fsync instead Enumerating objects: 5, done. Counting objects: 100% (5/5), done. Delta compression using up to 8 threads Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 389 bytes | 389.00 KiB/s, done. Total 3 (delta 2), reused 1 (delta 0), pack-reused 0 remote: warning: core.fsyncObjectFiles is deprecated; use core.fsync instead remote: warning: core.fsyncObjectFiles is deprecated; use core.fsync instead remote: fatal: commit-graph requires overflow generation data but has none warning: core.fsyncObjectFiles is deprecated; use core.fsync instead fatal: commit-graph requires overflow generation data but has none remote: warning: core.fsyncObjectFiles is deprecated; use core.fsync instead To ssh://<HOST REDACTED>/software/vev-renormalization.git ! [remote rejected] master -> master (missing necessary objects) error: failed to push some refs to 'ssh://<HOST REDACTED>/software/vev-renormalization.git' Going to the corresponding repository on the server, I see: [git@gitlab ~/repositories/@hashed/92/2c/922c7954216ccfe7a61def609305ce1dc7c67e225f873f256d30d7a8ee4f404c.git]$ git fsck --full Checking object directories: 100% (256/256), done. fatal: commit-graph requires overflow generation data but has none [git@gitlab ~/repositories/@hashed/92/2c/922c7954216ccfe7a61def609305ce1dc7c67e225f873f256d30d7a8ee4f404c.git]$ echo $? 128 After some searching, I found this recommendation: https://docs.gitlab.com/ee/install/installation.html#git Now GitLab seems to depend on a version of git patched to work with a particular version of gitaly. In the jail in which I run gitlab, I did: > cd /tmp > git clone https://gitlab.com/gitlab-org/gitaly.git -b 14-10-stable /tmp/gitaly > cd /tmp/gitaly > gmake git GIT_PREFIX=/usr/local This installed the patched version of git, overwriting the binaries installed in /usr/local/by the git package. After I did this, the repository checked clean: 05ce1dc7c67e225f873f256d30d7a8ee4f404c.git # git fsck --full Checking object directories: 100% (256/256), done. Checking objects: 100% (107/107), done. Verifying commits in commit graph: 100% (24/24), done. Also after switching to the patched version of git, pushing from a remote worked without error. It is rather annoying that gitlab (or perhaps more specifically gitaly) can not use the released git. Let me know if I can run more experiments. Reinstalling git from packages seems to reliably recreate the failed state, at least for one of my repositories. BR, Greg
Could you please retry with version 15.0.0? The changelog is mentioning some bugfixes: https://gitlab.com/gitlab-org/gitaly/-/blob/master/CHANGELOG.md that could be related to the problem you describe.
Created attachment 234480 [details] Include gitaly patches for git binary You can try to apply this patch, it maybe fixes the problem that the repository got broken. But I think you must restore the broken repository from a backup. I have the same problem now on some repositories and just installing the new git version with the patch included does not fix it for the broken repositories. But it maybe prevents it from happening again.
This issue also happens in GitLab 15.0 (i got them after the update) and persists whenever using ti 2.36.1. There are upstream issues like this: https://groups.google.com/g/linux.debian.bugs.dist/c/zbojhXyfFfY?pli=1 https://lore.kernel.org/git/f50e74f0-9ffa-f4f2-4663-269801495ed3@github.com/ If the issue happens to you, you can't: * checkout a repo anymore * the "Merge" Button in GitLab MR page checks for ever if a merge is possible * The CI / CD page list a big error message You can fix the issue by deleting the commit-graphs dir. Like this way: rm -rfv objects/info/commit-graphs I fixed all my 100+ repos with: find /usr/local/git/repositories -name commit-graphs -type d | xargs rm -rfv The issue will happen again, until the git is patched.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=57d9640f3c3669d318865222a743652a6952ba2d commit 57d9640f3c3669d318865222a743652a6952ba2d Author: Matthias Fechner <mfechner@FreeBSD.org> AuthorDate: 2022-06-06 06:02:05 +0000 Commit: Matthias Fechner <mfechner@FreeBSD.org> CommitDate: 2022-06-07 05:14:23 +0000 devel/git: fix bug commit-graph requires overflow generation data Using www/gitlab-ce I see problems that git repositories got broken. Gitlab includes some patches for the git binary they build: https://gitlab.com/gitlab-org/gitaly/-/tree/master/_support/git-patches These patches should have no impact to normal git usage but fixes some ref handling problems, like: https://gitlab.com/gitlab-org/gitaly/-/blob/master/_support/git-patches/v2.36.0.gl1/0006-refs-skip-hooks-when-deleting-uncovered-packed-refs.patch The problem got visible that at any point it can happen that the repository got broken (by merges happen) and if you try to fetch it you get: "C:\Program Files\Git\bin\git.exe" pull --progress "origin" fatal: commit-graph requires overflow generation data but has none fatal: the remote end hung up unexpectedly Done PR: 263867 Approved by: garga (maintainer) devel/git/Makefile | 12 ++++++++++++ devel/git/distinfo | 14 +++++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-)
I got now permission to commit the patch for git, I hope that fixes it.
Please revert this ASAP. First of all, the way these patches were added is just not acceptable: git is used for many kinds of sensitive data handling and unconditionally adding patches from untrusted third party is a security violation. These could've been under option, disabled by default (though I don't like the very idea of plugging bugs of unrelated software in git). Next, urls for these patches refer to git master branch on gitlab and are prone to breakage. In fact, they are all broken now. Upstream has removed these patches [claiming](https://gitlab.com/gitlab-org/gitaly/-/commit/248e71d0921d44a6fc3dfbdbb7402b02962bebf7) these are no longer needed, so let's clear our git port of these as well.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=52fafafdc0a2d5540d3228ea59261e79188de5e0 commit 52fafafdc0a2d5540d3228ea59261e79188de5e0 Author: Dmitry Marakasov <amdmi3@FreeBSD.org> AuthorDate: 2022-06-07 14:18:47 +0000 Commit: Dmitry Marakasov <amdmi3@FreeBSD.org> CommitDate: 2022-11-16 12:21:22 +0000 devel/git: remove no longer needed third party patches - This reverts 57d9640f3c3669d318865222a743652a6952ba2d PR: 267792, 263867 Approved by: garga (maintainer) devel/git/Makefile | 13 +------------ devel/git/distinfo | 12 ------------ 2 files changed, 1 insertion(+), 24 deletions(-)