Bug 289769 - libsys.so.7 not found when upgrading userland with legacy freebsd-update
Summary: libsys.so.7 not found when upgrading userland with legacy freebsd-update
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 15.0-STABLE
Hardware: Any Any
: --- Affects Many People
Assignee: Colin Percival
URL: https://reviews.freebsd.org/D52688
Keywords:
Depends on:
Blocks:
 
Reported: 2025-09-22 20:47 UTC by Graham Perrin
Modified: 2025-12-10 21:52 UTC (History)
11 users (show)

See Also:
grahamperrin: mfc-stable15+
grahamperrin: mfc-stable14+
grahamperrin: mfc-stable13+
grahamperrin: needs_errata+


Attachments
A screenshot of a test machine following the failure (68.75 KB, image/png)
2025-09-22 20:47 UTC, Graham Perrin
no flags Details
/bin/csh not usable for single user mode (43.51 KB, image/png)
2025-09-22 20:51 UTC, Graham Perrin
no flags Details
13.5-RELEASE-p3 userland upgraded to 15.0-ALPHA3 (65.45 KB, text/plain)
2025-09-24 21:39 UTC, Graham Perrin
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Graham Perrin 2025-09-22 20:47:01 UTC
Created attachment 264004 [details]
A screenshot of a test machine following the failure

Spun off from <https://www.reddit.com/r/freebsd/comments/1nndsd7/comment/nfnsbpt/?context=1>

1. FreeBSD-14.3-RELEASE-amd64-dvd1.iso

2. install the standard set (of base components) to UFS

3. boot the installed system

4. freebsd-update fetch

5. freebsd-update install

6. freebsd-update upgrade -r 15.0-ALPHA3

7. freebsd-update install

8. shutdown -r now

9. freebsd-update install

Expected: 

- installation of ALPHA3 userland

Actual result: 

ld-elf.so.1: Shared object "libsys.so.7" not found, required by "libc.so.7"
Comment 1 Graham Perrin 2025-09-22 20:51:02 UTC
Created attachment 264005 [details]
/bin/csh not usable for single user mode

Following shutdown -r now, the OS falls into a prompt for single user mode. 

Neither csh (pictured) nor sh is usable.
Comment 2 Colin Percival freebsd_committer freebsd_triage 2025-09-22 20:58:43 UTC
FWIW you can probably fix the system by manually extracting libsys using utilities in /rescue.  But that's obviously not a desired path.
Comment 3 Matthew D. Fuller 2025-09-22 21:12:20 UTC
Presumably (for those lucky enough not to have this stuff in their head):

- freebsd-update installs the libs in lexical order
- libc sorts before libsys
- New libc will be unhappy with the not-yet-existence of libsys
- freebsd-update needs to run things like gzip and install, and they'll be unhappy if libc doesn't work...
Comment 4 Colin Percival freebsd_committer freebsd_triage 2025-09-22 23:47:32 UTC
Yes, that sounds like a correct diagnosis.
Comment 5 Graham Perrin 2025-09-23 00:14:39 UTC
(In reply to Colin Percival from comment #2)

I attempted a few fixes. 

It seems easiest to use pkg, and then pkgbasify to work around quirks of FreeBSD-set-minimal-15.0.a2.20250919040617, however I understand that this will not satisfy people who prefer to continue with a freebsd-update(8) that is not yet pkg-centred.
Comment 6 Colin Percival freebsd_committer freebsd_triage 2025-09-23 07:03:43 UTC
Patch: https://reviews.freebsd.org/D52688
Comment 7 commit-hook freebsd_committer freebsd_triage 2025-09-23 21:06:02 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=7ece602e00e85195fc426a2401c49921cd39735e

commit 7ece602e00e85195fc426a2401c49921cd39735e
Author:     Colin Percival <cperciva@FreeBSD.org>
AuthorDate: 2025-09-23 06:55:08 +0000
Commit:     Colin Percival <cperciva@FreeBSD.org>
CommitDate: 2025-09-23 21:05:27 +0000

    freebsd-update: Library ordering

    Upgrading from 14.x to 15.x with freebsd-update broke because libc
    depends on the new libsys library; freebsd-update installed the new
    libc before creating libsys, and every step after that failed because
    all the tools (including gunzip and install) are dynamically linked
    and need a working libc.

    Enforce ordering when installing shared objects: First libsys, then
    libc, then libthr, and then all the rest of the shared object files.

    This is a candidate for an Errata Notice since the issue this fixes
    breaks upgrades.

    PR:             289769
    Reported by:    Graham Perrin
    Reviewed by:    kib
    MFC after:      3 days
    Sponsored by:   https://www.patreon.com/cperciva
    Differential Revision:  https://reviews.freebsd.org/D52688

 usr.sbin/freebsd-update/freebsd-update.sh | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
Comment 8 Graham Perrin 2025-09-24 21:39:24 UTC
Created attachment 264069 [details]
13.5-RELEASE-p3 userland upgraded to 15.0-ALPHA3

(In reply to commit-hook from comment #7)

FreeBSD-13.4-RELEASE-amd64-disc1.iso (end of life), I installed a full set of components. A minor update for patches, then a major upgrade to 13.5-RELEASE with patches. 

Restarted the OS, upgraded the userland to 13.5-RELEASE-p3, then a major upgrade to 15.0-ALPHA3. 

Restarted the OS, used script(1) for a transcript, upgraded the userland, then conversion with pkgbasify (479 packages). Notes attached.
Comment 9 Graham Perrin 2025-09-24 21:46:36 UTC
Incidentally, there's nothing exciting in the contents of /etc/master.passwd in the attachment at comment 8. The root password there is one that I use for throwaway test systems (never associated with data of any value).
Comment 10 commit-hook freebsd_committer freebsd_triage 2025-09-25 19:26:47 UTC
A commit in branch stable/15 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=8134e7f4b406dadb784d85e2cf3f9f4b578c7619

commit 8134e7f4b406dadb784d85e2cf3f9f4b578c7619
Author:     Colin Percival <cperciva@FreeBSD.org>
AuthorDate: 2025-09-23 06:55:08 +0000
Commit:     Colin Percival <cperciva@FreeBSD.org>
CommitDate: 2025-09-25 19:25:55 +0000

    freebsd-update: Library ordering

    Upgrading from 14.x to 15.x with freebsd-update broke because libc
    depends on the new libsys library; freebsd-update installed the new
    libc before creating libsys, and every step after that failed because
    all the tools (including gunzip and install) are dynamically linked
    and need a working libc.

    Enforce ordering when installing shared objects: First libsys, then
    libc, then libthr, and then all the rest of the shared object files.

    This is a candidate for an Errata Notice since the issue this fixes
    breaks upgrades.

    PR:             289769
    Reported by:    Graham Perrin
    Reviewed by:    kib
    MFC after:      3 days
    Sponsored by:   https://www.patreon.com/cperciva
    Differential Revision:  https://reviews.freebsd.org/D52688

    (cherry picked from commit 7ece602e00e85195fc426a2401c49921cd39735e)

 usr.sbin/freebsd-update/freebsd-update.sh | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
Comment 11 commit-hook freebsd_committer freebsd_triage 2025-09-25 19:27:48 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=e26928669f39c8683aea74040b9e2472e944c43a

commit e26928669f39c8683aea74040b9e2472e944c43a
Author:     Colin Percival <cperciva@FreeBSD.org>
AuthorDate: 2025-09-23 06:55:08 +0000
Commit:     Colin Percival <cperciva@FreeBSD.org>
CommitDate: 2025-09-25 19:26:58 +0000

    freebsd-update: Library ordering

    Upgrading from 14.x to 15.x with freebsd-update broke because libc
    depends on the new libsys library; freebsd-update installed the new
    libc before creating libsys, and every step after that failed because
    all the tools (including gunzip and install) are dynamically linked
    and need a working libc.

    Enforce ordering when installing shared objects: First libsys, then
    libc, then libthr, and then all the rest of the shared object files.

    This is a candidate for an Errata Notice since the issue this fixes
    breaks upgrades.

    PR:             289769
    Reported by:    Graham Perrin
    Reviewed by:    kib
    MFC after:      3 days
    Sponsored by:   https://www.patreon.com/cperciva
    Differential Revision:  https://reviews.freebsd.org/D52688

    (cherry picked from commit 7ece602e00e85195fc426a2401c49921cd39735e)

 usr.sbin/freebsd-update/freebsd-update.sh | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
Comment 12 commit-hook freebsd_committer freebsd_triage 2025-09-25 19:27:49 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=87eb52f1b061989a948d3eb08953c81a4e1281f0

commit 87eb52f1b061989a948d3eb08953c81a4e1281f0
Author:     Colin Percival <cperciva@FreeBSD.org>
AuthorDate: 2025-09-23 06:55:08 +0000
Commit:     Colin Percival <cperciva@FreeBSD.org>
CommitDate: 2025-09-25 19:27:28 +0000

    freebsd-update: Library ordering

    Upgrading from 14.x to 15.x with freebsd-update broke because libc
    depends on the new libsys library; freebsd-update installed the new
    libc before creating libsys, and every step after that failed because
    all the tools (including gunzip and install) are dynamically linked
    and need a working libc.

    Enforce ordering when installing shared objects: First libsys, then
    libc, then libthr, and then all the rest of the shared object files.

    This is a candidate for an Errata Notice since the issue this fixes
    breaks upgrades.

    PR:             289769
    Reported by:    Graham Perrin
    Reviewed by:    kib
    MFC after:      3 days
    Sponsored by:   https://www.patreon.com/cperciva
    Differential Revision:  https://reviews.freebsd.org/D52688

    (cherry picked from commit 7ece602e00e85195fc426a2401c49921cd39735e)

 usr.sbin/freebsd-update/freebsd-update.sh | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
Comment 13 commit-hook freebsd_committer freebsd_triage 2025-09-30 15:37:34 UTC
A commit in branch releng/14.3 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=978e04ff5bcf0a750cd3ef19561f5e0f67860620

commit 978e04ff5bcf0a750cd3ef19561f5e0f67860620
Author:     Colin Percival <cperciva@FreeBSD.org>
AuthorDate: 2025-09-23 06:55:08 +0000
Commit:     Gordon Tetlow <gordon@FreeBSD.org>
CommitDate: 2025-09-30 04:57:23 +0000

    freebsd-update: Library ordering

    Upgrading from 14.x to 15.x with freebsd-update broke because libc
    depends on the new libsys library; freebsd-update installed the new
    libc before creating libsys, and every step after that failed because
    all the tools (including gunzip and install) are dynamically linked
    and need a working libc.

    Enforce ordering when installing shared objects: First libsys, then
    libc, then libthr, and then all the rest of the shared object files.

    This is a candidate for an Errata Notice since the issue this fixes
    breaks upgrades.

    PR:             289769
    Reported by:    Graham Perrin
    Reviewed by:    kib
    MFC after:      3 days
    Sponsored by:   https://www.patreon.com/cperciva
    Differential Revision:  https://reviews.freebsd.org/D52688
    Approved by:    so
    Security:       FreeBSD-EN-25:18.freebsd-update

    (cherry picked from commit 7ece602e00e85195fc426a2401c49921cd39735e)
    (cherry picked from commit e26928669f39c8683aea74040b9e2472e944c43a)

 usr.sbin/freebsd-update/freebsd-update.sh | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
Comment 14 commit-hook freebsd_committer freebsd_triage 2025-09-30 15:37:35 UTC
A commit in branch releng/14.2 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=3447fea3523b56a235b9f8fe9234893f00795379

commit 3447fea3523b56a235b9f8fe9234893f00795379
Author:     Colin Percival <cperciva@FreeBSD.org>
AuthorDate: 2025-09-23 06:55:08 +0000
Commit:     Gordon Tetlow <gordon@FreeBSD.org>
CommitDate: 2025-09-30 04:58:01 +0000

    freebsd-update: Library ordering

    Upgrading from 14.x to 15.x with freebsd-update broke because libc
    depends on the new libsys library; freebsd-update installed the new
    libc before creating libsys, and every step after that failed because
    all the tools (including gunzip and install) are dynamically linked
    and need a working libc.

    Enforce ordering when installing shared objects: First libsys, then
    libc, then libthr, and then all the rest of the shared object files.

    This is a candidate for an Errata Notice since the issue this fixes
    breaks upgrades.

    PR:             289769
    Reported by:    Graham Perrin
    Reviewed by:    kib
    MFC after:      3 days
    Sponsored by:   https://www.patreon.com/cperciva
    Differential Revision:  https://reviews.freebsd.org/D52688
    Approved by:    so
    Security:       FreeBSD-EN-25:18.freebsd-update

    (cherry picked from commit 7ece602e00e85195fc426a2401c49921cd39735e)
    (cherry picked from commit e26928669f39c8683aea74040b9e2472e944c43a)

 usr.sbin/freebsd-update/freebsd-update.sh | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
Comment 15 commit-hook freebsd_committer freebsd_triage 2025-09-30 15:38:37 UTC
A commit in branch releng/13.5 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=ab91dd76ff72707d992cc54d2ebe1e424689d9cb

commit ab91dd76ff72707d992cc54d2ebe1e424689d9cb
Author:     Colin Percival <cperciva@FreeBSD.org>
AuthorDate: 2025-09-23 06:55:08 +0000
Commit:     Gordon Tetlow <gordon@FreeBSD.org>
CommitDate: 2025-09-30 04:58:26 +0000

    freebsd-update: Library ordering

    Upgrading from 14.x to 15.x with freebsd-update broke because libc
    depends on the new libsys library; freebsd-update installed the new
    libc before creating libsys, and every step after that failed because
    all the tools (including gunzip and install) are dynamically linked
    and need a working libc.

    Enforce ordering when installing shared objects: First libsys, then
    libc, then libthr, and then all the rest of the shared object files.

    This is a candidate for an Errata Notice since the issue this fixes
    breaks upgrades.

    PR:             289769
    Reported by:    Graham Perrin
    Reviewed by:    kib
    MFC after:      3 days
    Sponsored by:   https://www.patreon.com/cperciva
    Differential Revision:  https://reviews.freebsd.org/D52688
    Approved by:    so
    Security:       FreeBSD-EN-25:18.freebsd-update

    (cherry picked from commit 7ece602e00e85195fc426a2401c49921cd39735e)
    (cherry picked from commit 87eb52f1b061989a948d3eb08953c81a4e1281f0)

 usr.sbin/freebsd-update/freebsd-update.sh | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
Comment 16 Craig Leres freebsd_committer freebsd_triage 2025-10-18 22:44:28 UTC
I tried to go from 13.5 to 15.0-BETA2 and hit this. I don't see a recipe on how to recover so here's mine:

    # (I use csh)
    set path= ( /rescue $path )
    cd /tmp
    fetch https://download.freebsd.org/releases/amd64/15.0-BETA2/base.txz
    tar zxf ../base.txz lib/libsys.so.7
    mv -i lib/libsys.so.7 /lib
    freebsd-update install

I *think* my problem was that the 13.5 version freebsd-update is missing the fix from this PR?
Comment 17 ben 2025-12-03 11:47:09 UTC
I seem to have also encountered this issue, upgrading from 14.3-RELEASE to 15.0-RELEASE, and fixed it following Craig Leres' instructions.
Comment 18 Clay Ayers 2025-12-05 18:09:25 UTC
I can confirm I had the same issue on 14.3-RELEASE to 15.0-RELEASE on my server. Couldn't copy the files to /lib as I had no working su or sudo. Please advise.
Comment 19 ykla 2025-12-07 14:12:58 UTC
This issue still occurs in the update from FreeBSD 14.3-RELEASE to 15.0.
Comment 20 Lexi Winter freebsd_committer freebsd_triage 2025-12-10 08:22:46 UTC
(In reply to ykla from comment #19)
did you apply the freebsd-update errata (https://www.freebsd.org/security/advisories/FreeBSD-EN-25:18.freebsd-update.asc) before upgrading?
Comment 21 Kurt Jaeger freebsd_committer freebsd_triage 2025-12-10 08:54:24 UTC
(In reply to Clay Ayers from comment #18)

From my tests, the upgrade steps should be (from 14.3p6):

freebsd-update -r 15.0-RELEASE upgrade
freebsd-update install
freebsd-update install
shutdown -r now
pkg update
pkg upgrade
freebsd-update install
zpool upgrade ...

Because 14.3p6 to 15.0, with only one install followed by a reboot leads to an unusable system, see above.
Comment 22 Graham Perrin 2025-12-10 21:52:53 UTC
This report is closed, please make a separate report for things such as documentation. Thank you.