Bug 258858 - ports-mgmt/pkg: 1.17.2 fails to upgrade en-freebsd-doc and de-freebsd-doc 20210125,1 -> 20210918,1
Summary: ports-mgmt/pkg: 1.17.2 fails to upgrade en-freebsd-doc and de-freebsd-doc 202...
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-pkg (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-10-02 08:56 UTC by Matthias Andree
Modified: 2021-10-06 17:48 UTC (History)
3 users (show)

See Also:
bugzilla: maintainer-feedback? (pkg)


Attachments
xz-compressed logs of pkg -d upgrade en-freebsd-doc, three subsequent runs starting from a prior installed en-freebsd-doc-20210125,1 (26.19 KB, application/x-xz-compressed)
2021-10-02 08:56 UTC, Matthias Andree
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Matthias Andree freebsd_committer 2021-10-02 08:56:48 UTC
Created attachment 228348 [details]
xz-compressed logs of pkg -d upgrade en-freebsd-doc, three subsequent runs starting from a prior installed en-freebsd-doc-20210125,1

Greetings,

several of my machines require two "pkg upgrade" runs to upgrade the en-freebsd-doc and de-freebsd-doc packages from 20210125,1 to 20210918,1.  There is no note of this in ports/CHANGES.  The relevant change is apparently 7ee8c0803bbf5763c4c84bf23ccef422db9a692b.

The workaround appears to be pkg delete-ing the old *-freebsd-doc packages and reinstalling them by name afterwards. Please add a note of this to ports/CHANGES.

Data Points as Debugging Aids:

* I have rebuilt the old package from ports 84ebfb568a682d804cdcf77514670c5085a4f368 for FreeBSD:13:* - get it from https://people.freebsd.org/~mandree/en-freebsd-doc-20210125,1.pkg (35 MibiBytes)

* 1st run: pkg: Fail to rename /usr/local/share/doc/freebsd/en/articles/bsdl-gpl/.pkgtemp.article.html.bYSz3D04nZod -> /usr/local/share/doc/freebsd/en/articles/bsdl-gpl/article.html:No such file or directory

* 2nd run: pkg: unlinkat(usr/local/share/doc/freebsd/en): Operation not permitted


Relevant contents (sorry, de_DE locale for dates but you get the file types):

Old package en-freebsd-doc: 20210125,1
$ ls -ldb /usr/local/share/doc/freebsd/en
lrwxr-xr-x  1 root  wheel  15  1 Okt. 09:27 /usr/local/share/doc/freebsd/en -> en_US.ISO8859-1

New package en-freebsd-doc: 20210918,1
$ ls -ldb /usr/local/share/doc/freebsd/en
drwxr-xr-x  7 root  wheel  7  1 Okt. 09:41 en


NOTE: Failed upgrades appear to leave the old en_US.ISO8859-1 and other directories behind, and also .pkg* temp files.


Longer logs:
see attachment for pkg -d upgrade logs.



Reference screenshots of plain pkg upgrade:


First pkg run aborts:

...
Checking for upgrades (7 candidates): 100%
Processing candidates (7 candidates): 100%
The following 2 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
	automake: 1.16.3 -> 1.16.4 [FreeBSD-latest]
	en-freebsd-doc: 20210125,1 -> 20210918,1 [FreeBSD-latest]

Number of packages to be upgraded: 2

The operation will free 13 MiB.
20 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/2] Fetching en-freebsd-doc-20210918,1.pkg: 100%   20 MiB   1.9MB/s    00:11    
[2/2] Fetching automake-1.16.4.pkg: 100%  550 KiB 562.9kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/2] Upgrading en-freebsd-doc from 20210125,1 to 20210918,1...
[1/2] Extracting en-freebsd-doc-20210918,1: 100%
pkg: Fail to rename /usr/local/share/doc/freebsd/en/articles/bsdl-gpl/.pkgtemp.article.html.bYSz3D04nZod -> /usr/local/share/doc/freebsd/en/articles/bsdl-gpl/article.html:No such file or directory


Second run emits a truckload of errors (see log) but succeeds:

[root@rsv1 ~]# pkg upgrade -y
Updating FreeBSD-latest repository catalogue...
FreeBSD-latest repository is up to date.
Updating local repository catalogue...
local repository is up to date.
All repositories are up to date.
Checking for upgrades (7 candidates): 100%
Processing candidates (7 candidates): 100%
Checking integrity... done (0 conflicting)
The following 2 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        automake: 1.16.3 -> 1.16.4 [FreeBSD-latest]
        en-freebsd-doc: 20210125,1 -> 20210918,1 [FreeBSD-latest]

Number of packages to be upgraded: 2

The operation will free 13 MiB.
[1/2] Upgrading en-freebsd-doc from 20210125,1 to 20210918,1...
[1/2] Extracting en-freebsd-doc-20210918,1: 100%
pkg: unlinkat(usr/local/share/doc/freebsd/en): Operation not permitted
en-freebsd-doc-20210125,1: missing file /usr/local/share/doc/freebsd/en_US.ISO8859-1/articles/bsdl-gpl/addenda.html
en-freebsd-doc-20210125,1: missing file /usr/local/share/doc/freebsd/en_US.ISO8859-1/articles/bsdl-gpl/article.html
...
en-freebsd-doc-20210125,1: missing file /usr/local/share/doc/freebsd/en_US.ISO8859-1/books/porters-handbook/versions.html
en-freebsd-doc-20210125,1: missing file /usr/local/share/doc/freebsd/en_US.ISO8859-1/books/porters-handbook/why-port.html
en-freebsd-doc-20210125,1: missing file /usr/local/share/doc/freebsd/faq
en-freebsd-doc-20210125,1: missing file /usr/local/share/doc/freebsd/handbook
en-freebsd-doc-20210125,1: missing file /usr/local/share/licenses/en-freebsd-doc-20210125,1/BSD2CLAUSE
en-freebsd-doc-20210125,1: missing file /usr/local/share/licenses/en-freebsd-doc-20210125,1/LICENSE
en-freebsd-doc-20210125,1: missing file /usr/local/share/licenses/en-freebsd-doc-20210125,1/catalog.mk
[2/2] Upgrading automake from 1.16.3 to 1.16.4...
[2/2] Extracting automake-1.16.4: 100%
Comment 1 Matthias Andree freebsd_committer 2021-10-02 08:57:35 UTC
Note that the first, failing pkg upgrade run actually crashes after logging the error given above.
Comment 2 Marc Fonvieille freebsd_committer 2021-10-02 10:21:29 UTC
Hi,

Thanks for this feedback.
About CHANGES, I had the feeling that UPDATING should be more correct (I may
be wrong)?

Is something like the following correct?:

20210927:
AFFECTS: user of misc/freebsd-doc-*
AUTHOR: blackend@FreeBSD.org

The misc/freebsd-doc-* ports have been updated to the new doc toolchain.
Some problems may occur during an upgrade.  Users are encouraged to delete
the misc/freebsd-doc-* ports, then reinstall the new ones without using the
pkg upgrade facility.
Comment 3 Matthias Andree freebsd_committer 2021-10-02 12:16:45 UTC
Oh of course it's UPDATING not CHANGES which I had meant to propose (but did not write).

But this is merely a workaround, and we should fix pkg instead to handle such upgrades properly.
Comment 4 Matthias Andree freebsd_committer 2021-10-02 12:17:17 UTC
(In reply to Marc Fonvieille from comment #2)
The text itself seems fine for me.
Comment 5 Matthias Andree freebsd_committer 2021-10-02 12:22:00 UTC
On one of my machines where pkg aborted, after deinstallation of en-freebsd-doc, I have a truckload of .pkgtemp* garbage files and leftover directories lying around, find prints 458 lines:

# find /usr/local/share/doc/freebsd/
/usr/local/share/doc/freebsd/
/usr/local/share/doc/freebsd/en_US.ISO8859-1
/usr/local/share/doc/freebsd/en_US.ISO8859-1/articles
/usr/local/share/doc/freebsd/en_US.ISO8859-1/articles/bsdl-gpl
/usr/local/share/doc/freebsd/en_US.ISO8859-1/articles/bsdl-gpl/.pkgtemp.article.html.bYSz3D04nZod
/usr/local/share/doc/freebsd/en_US.ISO8859-1/articles/bsdl-gpl/.pkgtemp.article.pdf.V0SSyVb0QZzJ
/usr/local/share/doc/freebsd/en_US.ISO8859-1/articles/building-products
/usr/local/share/doc/freebsd/en_US.ISO8859-1/articles/building-products/.pkgtemp.article.html.tFnRvYuSyvAa
/usr/local/share/doc/freebsd/en_US.ISO8859-1/articles/building-products/.pkgtemp.article.pdf.Gp96ab4pxlSz
...
Comment 6 Matthias Andree freebsd_committer 2021-10-02 12:23:25 UTC
Another proposal:


20210927:
AFFECTS: user of misc/freebsd-doc-*
AUTHOR: blackend@FreeBSD.org

The misc/freebsd-doc-* ports have been updated to the new doc toolchain,
and use a somewhat different arrangement of directories and symlinks,
which pkg 1.17 does not handle well.

Users are encouraged to delete the misc/freebsd-doc-* ports, then
reinstall the new ones without using the pkg upgrade facility.
Comment 7 Marc Fonvieille freebsd_committer 2021-10-03 15:24:44 UTC
I just found misc/freebsd-doc-all wasn't in sync with others misc/freebsd-doc-*
(see https://cgit.freebsd.org/ports/commit/?id=a10f21cbae209a6fbafcdf4d60c2c1e125c137ec).
Could you check if it solves the issue for you?

I think the problem is elsewhere: between both versions of the ports, some plists and files have been removed.  Some install sub-directories have been changed. These changes could explain why pkg has issue with upgrade.

I did not commit the UPDATING entry yet, I'm waiting for your feedback for the change on misc/freebsd-doc-all.
Comment 8 Baptiste Daroussin freebsd_committer 2021-10-06 08:16:46 UTC
For sure this is a pkg bug, the issue being I think:
doc/en becomes a directory and pkg fails at handling it correctly during upgrade.

When such things happen pkg should detect it, drop its "as atomic as possible way to upgrade" and find a new upgrade strategy.

This will take time to fix correctly but at least it is reproducible!!
Comment 9 commit-hook freebsd_committer 2021-10-06 17:46:37 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=c1e0478c8f5a12efda1abcae69612849853d7cdd

commit c1e0478c8f5a12efda1abcae69612849853d7cdd
Author:     Marc Fonvieille <blackend@FreeBSD.org>
AuthorDate: 2021-10-06 17:41:28 +0000
Commit:     Marc Fonvieille <blackend@FreeBSD.org>
CommitDate: 2021-10-06 17:41:28 +0000

    UPDATING: Document misc/freebsd-doc-* upgrade

    misc/freebsd-doc-* need to be deleted by end.  pkg fails at
    handling correctly new directories.
    PR:             258858
    Reported by:    Matthias Andree (mandree)

 UPDATING | 11 +++++++++++
 1 file changed, 11 insertions(+)
Comment 10 Marc Fonvieille freebsd_committer 2021-10-06 17:48:23 UTC
(In reply to Baptiste Daroussin from comment #8)
Ok.  I updated UPDATING.
Do I need to let this PR open as a reminder for pkg bug?