Bug 256122

Summary: Git operations in section 11.1 of the Porters Manual do not work.
Product: Documentation Reporter: Neal Nelson <ports>
Component: Books & ArticlesAssignee: freebsd-doc (Nobody) <doc>
Status: Closed Not A Bug    
Severity: Affects Some People CC: bcr, blackend, imp, ygy
Priority: ---    
Version: Latest   
Hardware: Any   
OS: Any   
URL: https://reviews.freebsd.org/D30422

Description Neal Nelson 2021-05-24 15:13:50 UTC
There appear to be two problems with the git instructions as detailed in section 11.1 of the porters manual:

1. The "git pull --rebase" command does not work after adding files to the repository:

"error: cannot pull with rebase: You have unstaged changes.
error: additionally, your index contains uncommitted changes.
error: please commit or stash them."

This error message is of course with unstaged and uncommited changes, but either will stop a pull from happening.

2. The "git diff . > ../`make -VPKGNAME`.diff" command does not work if the changes have been staged. Adding the "--staged" option changes this.

If we have to commit the changes before we can pull any new updates, how do we then get a diff of the changes?

Is it necessary to commit the changes at all as this will never be pushed anywhere.

The old subversion workflow was straightforward and worked, but it's not obvious yet what the required workflow is in git. I am in the middle of a large update to a port, with five new ports that it will be dependent upon, that is now stalled because I can no longer get an up to date ports repository.
Comment 1 Guangyuan Yang freebsd_committer freebsd_triage 2021-05-24 17:14:47 UTC
(In reply to Neal Nelson from comment #0)

Thanks, these are indeed valid concerns, I will attempt to improve the paragraph soon.
Comment 2 Guangyuan Yang freebsd_committer freebsd_triage 2021-05-24 18:09:36 UTC
(In reply to Neal Nelson from comment #0)

And for your immediate issue, please try the following commands instead of the git pull:

% git pull origin main
% git rebase origin/main

... assuming you were following the paragraph you referred to, in which "origin" is https://git.FreeBSD.org/ports.git.
Comment 3 Guangyuan Yang freebsd_committer freebsd_triage 2021-05-24 18:10:15 UTC
(In reply to Neal Nelson from comment #0)

Scratch the last code block, should be:

% git fetch origin main
% git rebase origin/main
Comment 4 commit-hook freebsd_committer freebsd_triage 2021-05-26 04:12:18 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/doc/commit/?id=cb632051dc2b42ea8c8b92792867eba19a894905

commit cb632051dc2b42ea8c8b92792867eba19a894905
Author:     Guangyuan Yang <ygy@FreeBSD.org>
AuthorDate: 2021-05-26 04:08:47 +0000
Commit:     Guangyuan Yang <ygy@FreeBSD.org>
CommitDate: 2021-05-26 04:11:49 +0000

    porters-handbook: Fix the Git process in Chapter 11.1

    In Chapter 11.1 Using Git to Make Patches, there are multiple issues
    with the process:

    - `git pull --rebase` will refuse to work if there are any staged or
      unstaged changes, so the users are unable to update the repo to the
      latest.
    - `git diff` does not work for staged changes, need to add `--staged`.

    This commit attempts to address the above.

    PR:             256122
    Reported by:    Neal Nelson <ports@nicandneal.net>
    Reviewed by:    bcr, PauAmma <pauamma@gundo.com>
    Differential Revision:  https://reviews.freebsd.org/D30422

 .../en/books/porters-handbook/upgrading/_index.adoc  | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)
Comment 5 Neal Nelson 2021-05-26 08:46:01 UTC
Sorry for the late reply after this has been closed, but I have tried your suggested operations and the rebase still fails:

"error: cannot rebase: You have unstaged changes.
error: additionally, your index contains uncommitted changes.
error: Please commit or stash them."

Does this mean that I need to save my changes outside of the repository in order to get the most up to date changes? Surely there is a way to updage the repository with my changes in place, liek there used to be with subversion?
Comment 6 Guangyuan Yang freebsd_committer freebsd_triage 2021-05-26 08:56:03 UTC
(In reply to Neal Nelson from comment #5)

Sorry for the confusion - I *think* that this fix deals with staged changes, though in your situation, I personally choose one of two routes:

- For a quick change + update, first `git stash` the change then do a normal update.
- For a feature that I am continuously working on, add all my changes and make one commit to make things clean. If I need to chase the upstream, simply do what the (updated) instructions say. If I continue to make changes to the commit, use `git add` and `git commit --amend`.

And either way, develop on a feature branch is *strongly* recommended personally.

Please let me know if it helps, and don't hesitate to reach out to me, or reopen this again if you are still encountering errors.
Comment 7 Neal Nelson 2021-05-26 09:18:48 UTC
(In reply to Guangyuan Yang from comment #6)

Since the error message I got already said that it won't accept staged changes, I don't see that the proposed solution will work unfortunately.

I suppose I will have to go back to reading the git manual and see if I can work out how to do the things you suggest. I have managed to avoid using git until now, as I find the name is quite apt. It is vastly more complex and less usable than any version control system that I've used before.

In the meantime I have had to resort to cloning another copy of the ports repository for a quick update that I need to do to another port, but for a more long term and troublesome porting project I shall have to find another way of working.

Just as an aside: if we're having to commit changes to our repositories, will we be able to do some sort of pull request at some point, instead of the usual pr with diffs? This might alleviate some of the current pain with the mismatch between how things have been done for years and how they need to be done now. After all, once changes are committed it's harder to keep track of what's been done locally in order to make the diff files.
Comment 8 Guangyuan Yang freebsd_committer freebsd_triage 2021-05-26 09:28:47 UTC
(In reply to Neal Nelson from comment #7)

I totally understand your pain, and I am not on the git team so cannot advise or comment further - one thing I can say is that, based on my experience, the project is currently using git in a very limited and somewhat weird way - this is because we only migrated to git not long ago, and will gradually work on features next. (This answers the Pull Request question - not yet, but surely something we are considering.)

Also, from my personal experience, git is surely uncomfortable to learn, but rewarding once you mastered it.

From your description, I think the first thing for you to look at is git branching - separate each set of changes in feature branches, and keep the local `main` branch tracking remote upstream. Do not work on local main branch. git rebase onto feature branches to merge upstream changes to them.

I would direct your further questions and comments to the mailing list https://lists.freebsd.org/subscription/freebsd-git, just so that you can get timely and more correct responses. Also, your (contributor's) feedback is important and is needed to be heard there.
Comment 9 Warner Losh freebsd_committer freebsd_triage 2021-05-26 14:48:39 UTC
> 1. The "git pull --rebase" command does not work after adding files to the  repository:
>
> "error: cannot pull with rebase: You have unstaged changes.
> error: additionally, your index contains uncommitted changes.
> error: please commit or stash them."
>
> This error message is of course with unstaged and uncommited changes, but either > will stop a pull from happening.

Almost any pull will cause this. Easiest Solution: commit the changes. Then git pull --rebase will rebase them on the top if main. Alternatively, make the changes on a branch in your local repo and rebase that to keep main unchanged.

> 2. The "git diff . > ../`make -VPKGNAME`.diff" command does not work if the changes have been staged. Adding the "--staged" option changes this.

Yea, I'd not recommend that. You are almost always better off committing the changes than to monkey around with things like this.

> If we have to commit the changes before we can pull any new updates, how do we then get a diff of the changes?

% git diff HEAD^ HEAD
or
% git show HEAD

if there's one change.
% git diff freebsd/main..main
if there's more.

> Is it necessary to commit the changes at all as this will never be pushed anywhere.

Is there some reason not to commit the changes?

> The old subversion workflow was straightforward and worked, but it's not obvious yet what the required workflow is in git. I am in the middle of a large update to a port, with five new ports that it will be dependent upon, that is now stalled because I can no longer get an up to date ports repository.

This is a poster-child use-case for doing things on a branch:

% git checkout -b port-update
% git add / .... git commit

then to update

% git checkout main
% git pull --ff-only                (or --rebase, they will be the same)
% git rebase -i main port-update

then, when you are happy with the commit:

% git rebase -i main port-update
% git checkout main
% git merge --ff-only port-update
% git pul --rebase
% git push

After I typed this I noticed this was closed. I should go review the ports docs that were committed.
Comment 10 Warner Losh freebsd_committer freebsd_triage 2021-05-26 14:50:56 UTC
After reading the advice in the change, I think it's wrong. I think we should strongly encourage commit early, commit often, curate with git rebase -i. The suggested workarounds are tedious and error prone. I used to use them and they have @^!#$& me in the past. By committing and rebasing, you have complete control over the end result. If you 'oops' in the merging, then there's no 'checkpoint' to go back to, especially if there are merge conflicts with the method suggested.
Comment 11 Neal Nelson 2021-05-26 15:42:55 UTC
The only reason that I'm reluctant to commit anything is that I can not push the changes anywhere. For a port update I have to make a diff and submit it as a pr. 

I suppose I could create a branch, do my stuff, commit, rebase, then get a diff with main and then trash the branch as it's no longer needed once the changes have been committed (assuming I can do that). I've found that other changes may happen to my ports without my knowledge, so it's never safe to assume that they are exactly as I submitted them when coming to make the next change.

This workflow is very different to how it used to be, so I'm really surprised that no one else has had problems, unless I'm the only one who's managed to avoid using git for this long.
Comment 12 Guangyuan Yang freebsd_committer freebsd_triage 2021-05-26 18:45:06 UTC
Reopen this PR to track further improvements to the committed fix.
Comment 13 Marc Fonvieille freebsd_committer freebsd_triage 2021-05-26 19:13:49 UTC
(In reply to Neal Nelson from comment #11)
Hi,

I often have the same problem.
Could you try:

% git stash
% git pull --ff-only --rebase
% git stash pop

It always worked in my case.
Comment 14 Neal Nelson 2021-05-28 07:17:35 UTC
Given all that I've read in this pr so far, I think that I've found a viable, and reasonably simple, workflow for ports management. I updated one of my ports yesterday and it all seemed to go well.

I did the following:

% git pull --rebase
% git switch -c update

Do the changes.
Add the changes.
Commit the changes.

% git switch main
% git pull --rebase
% git rebase main changes
% git diff main..update > ../`make -VPKGNAME`.diff

Once the pr has been committed then I will:

% git branch -d update

or in case the changes committed aren't exactly the same as submitted in the pr:

% git branch -D update
Comment 15 Warner Losh freebsd_committer freebsd_triage 2021-05-28 16:07:35 UTC
(In reply to Neal Nelson from comment #14)
> % git diff main..update > ../`make -VPKGNAME`.diff

I'd be tempted to having the 'submitter' curate things in the 'update' branch, and then submit 'git format-patch' style diffs. Those diffs have the commit message, the proper author and the changes all in a series of files that can easily be imported into a git tree with 'git am' saving time and hassle.

So maybe:

% cd ~/ports
% mkdir ~/patch-20210522
% git format-patch -o ~/patch-20210522 main..update

would be better, and then submit the files that are in ~/patch-20210522 with the PR instead? They could then be snarfed in with git am and all the details have a better chance of being right in the resulting commit message.
Comment 16 Benedict Reuschling freebsd_committer freebsd_triage 2024-05-30 20:37:12 UTC
I'm closing this. The issues have been resolved in the audit trail of this PR (thanks to those who provided comments, help and insights) and there was no further activity on it. No required changes to docs resulted from the discussion, so I think it is safe to close the PR for good.