Bug 238314 - pkg not installed by dvd installer
Summary: pkg not installed by dvd installer
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 13.1-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: FreeBSD Release Engineering
URL: https://github.com/freebsd/freebsd-sr...
Keywords:
Depends on:
Blocks: 201485 266743
  Show dependency treegraph
 
Reported: 2019-06-03 18:28 UTC by Matthew Woehlke
Modified: 2022-11-17 00:29 UTC (History)
3 users (show)

See Also:


Attachments
Screenshot: bsdconfig fails to create /usr/local/sbin (30.70 KB, image/png)
2022-10-01 18:49 UTC, Graham Perrin
no flags Details
Fix broken pkg link in DVD1 repo (674 bytes, text/plain)
2022-11-07 20:13 UTC, Ian Smith
grahamperrin: maintainer-approval? (gjb)
Details
bsdconfig packages: fix musthavepkg.subr (468 bytes, text/plain)
2022-11-13 13:59 UTC, Ian Smith
no flags Details
bsdconfig packages: fix packages.subr (315 bytes, text/plain)
2022-11-13 14:13 UTC, Ian Smith
no flags Details
bsdconfig packages: fix index.subr (1.08 KB, text/plain)
2022-11-13 14:56 UTC, Ian Smith
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Matthew Woehlke 2019-06-03 18:28:37 UTC
After installing FreeBSD using the DVD image, pkg(8) is not installed. This makes it impossible to install additional software from the DVD image (i.e. using bsdconfig) without an internet connection, defeating the purpose of using the DVD image in the first place. Nor is bsdconfig able to install pkg from the DVD image.

Installing a package-managed OS without the package manager seems... odd in the first place, but at least bsdconfig should be able to install it (from the DVD image, without an internet connnection).
Comment 1 Graham Perrin freebsd_committer freebsd_triage 2022-10-01 18:26:25 UTC
Reproducible with FreeBSD-13.1-RELEASE-amd64-dvd1.iso
Comment 2 Graham Perrin freebsd_committer freebsd_triage 2022-10-01 18:49:02 UTC
Created attachment 236986 [details]
Screenshot: bsdconfig fails to create /usr/local/sbin

(In reply to Matthew Woehlke from comment #0)

> … without an internet connection, …


Alternatively, _with_ a connection: 

> Error: pkg
> 
> pkg: failed to extract pkg-static: Failed to create dir '/usr/local/sbin'
Comment 3 Ian Smith 2022-10-10 20:30:16 UTC
Graham, thanks for looking into these 'bsdconfig packages' issues. 

Re comment #2, /usr/local/sbin is created by bsdinstall and will be there already for any non-VM tests of bsdconfig so that's not an issue.

I could work with this one, or 195705 or 201485 or now 266743 but I'd prefer to consolidate if possible, happy to be guided by all those concerned as to where best to post.

That said, I have worked through this over many months, and have posted the steps needed to get bsdconfig to work correctly to install so far xorg, KDE and a few utilities from the 12.3-R dvd1, without once contacting the online freebsd.org pkg server, here:

https://forums.freebsd.org/threads/installing-freebsd-12-3-from-dvd1-usb-without-network.84838/

Very likely far too detailed and wordy for some, I will try being more succinct here - but the bottom line is that

 a) for 12.x at least, pkg bootstrap _cannot_ succeed due to a broken symlink, details in the forum post.

    This needs fixing soon so that early builds of 12.4 can use offline packages, once the other issues below are fixed.  As it stands even the workarounds in e.g. the 10.x errata can't work as pkg-static can't find the ../All/pkg-*.pkg to install it, as Matthew Woehlke describes above.

    I haven't found in src/release where that and the other .txz -> .pkg symlinks are made, help please?

 b) since perhaps 9.x, asynchronous changes to bsdconfig and pkg and some apparent personal differences have gummed up the works.  I don't care for the politics: the solution is purely technical.

Once seen to be working for intended purpose then updates to the docs can follow, describing how to use it rather than suggesting workarounds.

The three patches in my forum post are against the installed sh(1) code in /usr/share/bsdconfig/packages/* which are just copies of the sources in src/usr.sbin/bsdconfig/share/packages/ but I'll update those for that soon, removing gratuitous comments.

There's still an issue with one of my patches re which sqlite database to use, that needs a selection mechanism between local and (later, normal) remote pkg repos; perhaps some ideas might emerge?
Comment 4 Ian Smith 2022-10-11 20:49:06 UTC
(In reply to Graham Perrin from comment #2)

Graham, sorry, I hadn't twigged that you meant running bsdconfig *while still in the installer*, before rebooting into the installed system; it didn't occur to me.

I'm not sure whether Matthew also meant that, or running bsdconfig later - which also doesn't work due to a) the broken symlink to the pkg-*.pkg I detailed, in 12.x at least, and b) at least 2 of the 3 error fix patches required to first bootstrap and then use pkg(8), similarly.

This PR should still reference 12.x too.

Therefore running bsdconfig from the newly installed system with dvd or stick mounted is the way to go, needing release notes and handbook supporting that.  Adding a dialog note outlining that at the end of bsdinstall should be simple; I think I could do that, monkey see, monkey do.

I'd still like to see exactly what are pointed to by the now 2 symlinks in /dist/packages/FreeBSD:13:amd64/Latest ?
Comment 5 Graham Perrin freebsd_committer freebsd_triage 2022-10-17 14:39:39 UTC
For subscribers here who are not subscribers to freebsd-questions: 

<https://lists.freebsd.org/archives/freebsd-questions/2022-October/002087.html> 

* ignore the first of my responses
* the second was relevant.
Comment 6 Ian Smith 2022-11-07 20:06:23 UTC
This fix is needed on 12.4 so that:

a) pkg bootstrap can find the packages included with the dvd1, in particular the ../All/pkg-*.pkg that pkg(7) needs for pkg(8) update, the cause of all these "can't find pkg" failures.

b) subsequently bsdconfig can be used to install dvd1 packages as intended, after applying three one-line patches that resolve the remaining issues, seemingly consequences of changes to pkg(8) over recent years.

c) bsdconfig can also later browse, select and install packages from the FreeBSD repository, after provision of a fairly concise howto and/or a later patch for swapping repos.

d) bsdinstall(8) can install the docs, which also hasn't worked offline so far on 12.x dvd1, maybe disc1 too?

This has been broken since after 11.3-R, when packages in repos including those on the dvd were renamed from *.txz to *.pkg, as on all? the 12.x releases.

The patch is shown as a diff, but in fact just reflects the version in 13.1-R being merged to 12-STABLE and/or 12.4-R before the release. This points the main symlink to the full ../All/pkg-*.pkg while maintaining another symlink called pkg.txz for backward compatibility.

If this can go ahead I'll conplete uploading and documenting the three other essential patches for bsdconfig - otherwise there'd be little point.
Comment 7 Ian Smith 2022-11-07 20:13:06 UTC
Created attachment 237927 [details]
Fix broken pkg link in DVD1 repo
Comment 8 Graham Perrin freebsd_committer freebsd_triage 2022-11-13 02:00:43 UTC
Comment on attachment 237927 [details]
Fix broken pkg link in DVD1 repo

Glen's kind offer (in response to email from Ian Smith) 

… plus <https://www.freebsd.org/releases/12.4R/> a release note, if committed.
Comment 9 Ian Smith 2022-11-13 13:59:26 UTC
Created attachment 238052 [details]
bsdconfig packages: fix musthavepkg.subr

bsdconfig.debug.1-DEBUG: f_musthavepkg_init: retval=0 <output below>
bsdconfig.debug.1-pkg-static: Invalid type for environment variable ASSUME_ALWAYS_YES, got 1, while
expecting a boolean
bsdconfig.debug.1-pkg: Invalid type for environment variable ASSUME_ALWAYS_YES, got 1, while expecti
ng a boolean
Comment 10 Ian Smith 2022-11-13 14:13:56 UTC
Created attachment 238053 [details]
bsdconfig packages: fix packages.subr

This was the last patch needed to make bsdconfig have pkg(8) install browsed and selected packages, again due to the change of package suffixes.
Comment 11 Ian Smith 2022-11-13 14:56:09 UTC
Created attachment 238055 [details]
bsdconfig packages: fix index.subr

This patch, thanks to bapt@ for method, works fine for me but needs testing by others, as opposed to the above two that are uncontroversial.

You can still set REPOS_DIR (except when choosing type CDROM, which overrides it) but I find it more convenient to use the standard method of having FreeBSD.conf or FreeBSD_install_cdrom.conf (or other) chosen by pkg(8) by placing the desired .conf in /usr/local/etc/pkg/repos, for the duration of one or more sessions of browsing and/or installing packages.

Clearly this needs better documenting.
Comment 12 Ian Smith 2022-11-13 15:14:52 UTC
I haven't marked these three patches as such; I'm sure they're not in the correct format (just diff -u against copied .orig files).

Further, as sources they belong in /usr/src/usr.sbin/bsdconfig/share/packages but are identical to the object code at /usr/share/bsdconfig/packages.

I hope to provide a single script to patch the object code for users before these might make it into the tree, perhaps 13.2?
Comment 13 Glen Barber freebsd_committer freebsd_triage 2022-11-15 19:17:40 UTC
At this point in the release cycle, I have opted to go with the most least intrusive path forward (related to PR 263574, committed as f398c2d8323c to stable/12, pending approval from another re@ member).
Comment 14 Graham Perrin freebsd_committer freebsd_triage 2022-11-16 00:16:35 UTC
Thanks

releng/12.4 7b05f19e97087b2630e1142b13edef57f0540076 cherry-picked from f398c2d
Comment 15 commit-hook freebsd_committer freebsd_triage 2022-11-17 00:29:17 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/doc/commit/?id=027810211e9fffec16ebc5c0cdbc5ccc64553828

commit 027810211e9fffec16ebc5c0cdbc5ccc64553828
Author:     Graham Perrin <grahamperrin@FreeBSD.org>
AuthorDate: 2022-11-17 00:15:42 +0000
Commit:     Graham Perrin <grahamperrin@FreeBSD.org>
CommitDate: 2022-11-17 00:15:42 +0000

    12.4 release note: fix for bugs 263574 and 238314

    Bug 238314
    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238314
    was fixed by a commit with reference to bug 263574
    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263574

    PR:                    263574, 238314
    Approved by:           gjb (mentor)
    Differential revision: https://reviews.freebsd.org/D37402

 website/content/en/releases/12.4R/relnotes.adoc | 2 ++
 1 file changed, 2 insertions(+)