Bug 273950

Summary: freebsd-update rollback (from 14.0-BETA2 back to 13.2) fails
Product: Base System Reporter: Ed Maste <emaste>
Component: miscAssignee: Ed Maste <emaste>
Status: Open ---    
Severity: Affects Only Me CC: andriys, grahamperrin, pi
Priority: ---    
Version: CURRENT   
Hardware: Any   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256252
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273661
Bug Depends on:    
Bug Blocks: 271607    

Description Ed Maste freebsd_committer freebsd_triage 2023-09-19 21:47:54 UTC 
This is freebsd-update with https://reviews.freebsd.org/D41893 applied, after updating from 13.2 to 14.0-BETA2:

root@vultr:~ # sh freebsd-update.sh rollback
Uninstalling updates...install: ///lib/casper/libcap_dns.so.2: No such file or directory
install: ///lib/casper/libcap_fileargs.so.1: No such file or directory
install: ///lib/casper/libcap_grp.so.1: No such file or directory
install: ///lib/casper/libcap_net.so.1: No such file or directory
install: ///lib/casper/libcap_pwd.so.1: No such file or directory
install: ///lib/casper/libcap_sysctl.so.2: No such file or directory
install: ///lib/casper/libcap_syslog.so.1: No such file or directory
rm: ///usr/include/c++/v1/__string/extern_template_lists.h: Not a directory
rm: ///usr/include/c++/v1/__string/char_traits.h: Not a directory
 done.
root@vultr:~ # strings /boot/kernel/kernel | grep 'FreeBSD'
ld-elf.so.1: Shared object "libcap_fileargs.so.1" not found, required by "strings"

The __string errors are related to D41893 (and shouldn't cause a functional issue); the /lib/casper errors appear to be preexisting.
Comment 1 Ed Maste freebsd_committer freebsd_triage 2023-09-25 17:54:53 UTC 
In 13 these are in /lib/casper, e.g. /lib/casper/libcap_net.so.1
In 14 they are in lib, e.g. /lib/libcap_net.so.1

After running the first 'sh freebsd-update.sh install', an install.Fuk6PT directory is created:
lrwxr-xr-x  1 root  wheel       14 Sep 25 17:20 f465c3739385890c221dff1a05e578c6cae0d0430e46996d319db7439f884336-install -> install.Fuk6PT

grep '/lib/casper/' INDEX* in there:

/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/lib/casper/libcap_dns.so.2|f|0|0|0444|0|1a1d07bf89222c1ea8c7d051ab89844dde4444f93dad798f2d08069330b5b98e|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/lib/casper/libcap_fileargs.so.1|f|0|0|0444|0|e93b3eab9f04913eb7aefa8576a3fd7d593325b0901f9a462f12682045a8452a|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/lib/casper/libcap_grp.so.1|f|0|0|0444|0|848714e6a5fb2633e5b6cff16fd936aecefc77488dfb7c8333ce4e5c26dea047|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/lib/casper/libcap_net.so.1|f|0|0|0444|0|1d001dd45a61ea50e3b4cc9d42fe0e2c0b51ca671fe6db939dea43a859b9c695|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/lib/casper/libcap_pwd.so.1|f|0|0|0444|0|80a37d372633979b8cd50ea7f550c5f5a054a9c1c14f359bd971e1ea1fb5dec9|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/lib/casper/libcap_sysctl.so.2|f|0|0|0444|0|3de8a10f163de8ea9d7541677c523147a9b0206e436cc342a31258e2e85af123|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/lib/casper/libcap_syslog.so.1|f|0|0|0444|0|078fe519e3526e955ec72ce178bb4b66bb31273155e093d397bd296b4fb04a48|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/lib/casper|d|0|0|0755|0||
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/usr/lib/libcap_dns.so|L|0|0|0755|0|../../lib/casper/libcap_dns.so.2|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/usr/lib/libcap_fileargs.so|L|0|0|0755|0|../../lib/casper/libcap_fileargs.so.1|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/usr/lib/libcap_grp.so|L|0|0|0755|0|../../lib/casper/libcap_grp.so.1|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/usr/lib/libcap_net.so|L|0|0|0755|0|../../lib/casper/libcap_net.so.1|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/usr/lib/libcap_pwd.so|L|0|0|0755|0|../../lib/casper/libcap_pwd.so.1|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/usr/lib/libcap_sysctl.so|L|0|0|0755|0|../../lib/casper/libcap_sysctl.so.2|
/var/db/freebsd-update/install.Fuk6PT/INDEX-OLD:/usr/lib/libcap_syslog.so|L|0|0|0755|0|../../lib/casper/libcap_syslog.so.1|

freebsd-update.sh:

# Install old files, delete new files, and update linker.hints
rollback_files () {
        # Install old shared library files which don't have the same path as
        # a new shared library file.
        grep -vE '^/boot/' $1/INDEX-NEW |
            grep -E '/lib/.*\.so\.[0-9]+\|' |
            cut -f 1 -d '|' |
            sort > INDEX-NEW.libs.flist
        grep -vE '^/boot/' $1/INDEX-OLD |
            grep -E '/lib/.*\.so\.[0-9]+\|' |
            sort -k 1,1 -t '|' - |
            join -t '|' -v 1 - INDEX-NEW.libs.flist > INDEX-OLD
        install_from_index INDEX-OLD || return 1

So there are two issues:
1) We attempt to install /lib/casper/* via the first block in rollback_files() before /lib/casper has been recreated.
2) The /lib/casper directory entry appears after all of the files that it contains, in INDEX-OLD.
Comment 2 Ed Maste freebsd_committer freebsd_triage 2023-09-27 20:19:02 UTC 
For the __string directory/file issue: https://reviews.freebsd.org/D41945
Comment 3 commit-hook freebsd_committer freebsd_triage 2023-10-18 14:54:28 UTC 
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=6b27e1f2eaab9360a796c943da9d206515e1742b

commit 6b27e1f2eaab9360a796c943da9d206515e1742b
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2023-09-29 15:28:35 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2023-10-18 14:52:59 +0000

    freebsd-update: for rollback, first create directories

    rollback_files() tried to install files before creating the directories
    for those files.  In some cases this is due to special handling to
    install certain classes of files earlier than others.

    Just create all directories up front when performing rollback.

    PR:             273950
    Reviewed by:    dim
    Sponsored by:   The FreeBSD Foundation
    Differential Revision: https://reviews.freebsd.org/D42020

 usr.sbin/freebsd-update/freebsd-update.sh | 5 +++++
 1 file changed, 5 insertions(+)
Comment 4 commit-hook freebsd_committer freebsd_triage 2023-10-18 14:54:29 UTC 
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=c0f52443166ae7ecd512ab0350469d9c3648788c

commit c0f52443166ae7ecd512ab0350469d9c3648788c
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2023-09-12 02:59:30 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2023-10-18 14:48:58 +0000

    freebsd-update: handle directories changing to files

    Further to f6d37c9ca13f ("freebsd-update: handle file -> directory on
    upgrade"), handle the reverse case of a directory changing to a file.
    We may not encounter this case on upgradess (before freebsd-update is
    retired) but it is needed to support rollback.

    PR:             273950
    Reviewed by:    dim
    Sponsored by:   The FreeBSD Foundation
    Differential Revision: https://reviews.freebsd.org/D41945

 usr.sbin/freebsd-update/freebsd-update.sh | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
Comment 5 andriys 2023-11-10 12:26:19 UTC 
Do I understand correctly that this fix did not make it into the 14-RELEASE, and so the upgrade should only be considered if one is 100% certain the rollback will not be required?
Comment 6 commit-hook freebsd_committer freebsd_triage 2023-11-12 17:23:19 UTC 
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/doc/commit/?id=84ffee7a7c60059547834576fe53a57009faaeec

commit 84ffee7a7c60059547834576fe53a57009faaeec
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2023-11-10 15:31:17 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2023-11-12 17:22:03 +0000

    14.0 relnotes: freebsd-update cannot rollback across a major version

    PR:             273950
    Sponsored by:   The FreeBSD Foundation
    Differential Revision: https://reviews.freebsd.org/D42547

 website/content/en/releases/14.0R/relnotes.adoc | 1 +
 1 file changed, 1 insertion(+)
Comment 7 Ed Maste freebsd_committer freebsd_triage 2023-11-12 17:52:45 UTC 
(In reply to andriys from comment #5)
> Do I understand correctly that this fix did not make it into the 14-RELEASE

This is correct

> and so the upgrade should only be considered if one is 100% certain the rollback
> will not be required?

As far as I can tell this has always been the case. I have added a comment to this effect in the release notes.
Comment 8 commit-hook freebsd_committer freebsd_triage 2024-01-07 01:28:20 UTC 
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=1cf8528fec85a965baa5df2d8852166437af88a6

commit 1cf8528fec85a965baa5df2d8852166437af88a6
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2023-09-12 02:59:30 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2024-01-07 01:27:17 +0000

    freebsd-update: handle directories changing to files

    Further to f6d37c9ca13f ("freebsd-update: handle file -> directory on
    upgrade"), handle the reverse case of a directory changing to a file.
    We may not encounter this case on upgradess (before freebsd-update is
    retired) but it is needed to support rollback.

    PR:             273950
    Reviewed by:    dim
    Sponsored by:   The FreeBSD Foundation
    Differential Revision: https://reviews.freebsd.org/D41945

    (cherry picked from commit c0f52443166ae7ecd512ab0350469d9c3648788c)

 usr.sbin/freebsd-update/freebsd-update.sh | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
Comment 9 commit-hook freebsd_committer freebsd_triage 2024-01-07 21:26:44 UTC 
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=01e31d1d875f43a9d9c2d8f033409cbc7901c7ab

commit 01e31d1d875f43a9d9c2d8f033409cbc7901c7ab
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2023-09-12 02:59:30 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2024-01-07 20:41:48 +0000

    freebsd-update: handle directories changing to files

    Further to f6d37c9ca13f ("freebsd-update: handle file -> directory on
    upgrade"), handle the reverse case of a directory changing to a file.
    We may not encounter this case on upgradess (before freebsd-update is
    retired) but it is needed to support rollback.

    PR:             273950
    Reviewed by:    dim
    Sponsored by:   The FreeBSD Foundation
    Differential Revision: https://reviews.freebsd.org/D41945

    (cherry picked from commit c0f52443166ae7ecd512ab0350469d9c3648788c)
    (cherry picked from commit 1cf8528fec85a965baa5df2d8852166437af88a6)

 usr.sbin/freebsd-update/freebsd-update.sh | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
Comment 10 commit-hook freebsd_committer freebsd_triage 2024-01-07 21:26:46 UTC 
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=ec5e6781b3a1a3216535bc9aeedd746449cee6e6

commit ec5e6781b3a1a3216535bc9aeedd746449cee6e6
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2023-09-29 15:28:35 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2024-01-07 20:41:48 +0000

    freebsd-update: for rollback, first create directories

    rollback_files() tried to install files before creating the directories
    for those files.  In some cases this is due to special handling to
    install certain classes of files earlier than others.

    Just create all directories up front when performing rollback.

    PR:             273950
    Reviewed by:    dim
    Sponsored by:   The FreeBSD Foundation
    Differential Revision: https://reviews.freebsd.org/D42020

    (cherry picked from commit 6b27e1f2eaab9360a796c943da9d206515e1742b)
    (cherry picked from commit 2b79df0ad999df4bc22ed0e1eff446efa4055359)

 usr.sbin/freebsd-update/freebsd-update.sh | 5 +++++
 1 file changed, 5 insertions(+)
Comment 11 commit-hook freebsd_committer freebsd_triage 2024-01-08 14:46:40 UTC 
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=2b79df0ad999df4bc22ed0e1eff446efa4055359

commit 2b79df0ad999df4bc22ed0e1eff446efa4055359
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2023-09-29 15:28:35 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2024-01-07 19:40:06 +0000

    freebsd-update: for rollback, first create directories

    rollback_files() tried to install files before creating the directories
    for those files.  In some cases this is due to special handling to
    install certain classes of files earlier than others.

    Just create all directories up front when performing rollback.

    PR:             273950
    Reviewed by:    dim
    Sponsored by:   The FreeBSD Foundation
    Differential Revision: https://reviews.freebsd.org/D42020

    (cherry picked from commit 6b27e1f2eaab9360a796c943da9d206515e1742b)

 usr.sbin/freebsd-update/freebsd-update.sh | 5 +++++
 1 file changed, 5 insertions(+)
Comment 12 Mark Linimon freebsd_committer freebsd_triage 2024-01-10 04:52:39 UTC 
^Triage: assign to committer that resolved and MFCed.
Comment 13 Ed Maste freebsd_committer freebsd_triage 2024-01-10 15:22:44 UTC 
Note this is not fully fixed - my commits addressed a number of issues identified from this, but rollback still does not work.