Bug 249284 - sysutils/e2fsprogs: installs files outside ${PREFIX}
Summary: sysutils/e2fsprogs: installs files outside ${PREFIX}
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Matthias Andree
Depends on: 249374
  Show dependency treegraph
Reported: 2020-09-12 21:35 UTC by Pawel Biernacki
Modified: 2021-08-04 21:29 UTC (History)
7 users (show)

See Also:
mandree: maintainer-feedback+

proposed fixed port, also fixes some borderline cases - see changes_4 (52.23 KB, text/plain)
2020-09-16 19:20 UTC, Matthias Andree
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Pawel Biernacki freebsd_committer 2020-09-12 21:35:57 UTC
I noticed that this port installs additional files in system /sbin (!), which came as a surprise.  Can the installation of /sbin/fsck_ext2fs etc… be made a port configuration option?  Not everyone expects (and wants) ports to touch system directories :-(
I understand the use case, just want an option :-)
Comment 1 Matthias Andree freebsd_committer 2020-09-13 08:33:11 UTC
Pawel, I won't add options that allow people shoot holes in their feet and jeopardize their boot sequence.

The Linux-based ext* file systems have become first-class citizen in FreeBSD, with even a BSD-licensed kernel driver, so we need full tool support so that people can use them properly from fstab.  That isn't "optional".  A deinstallation of e2fsprogs will remove the files from /sbin again.

So, in order to use ext2/ext3/ext4 from fstab, we need support for fsck before /usr is mounted, meaning we need a self-contained fsck tool and wrapper in /sbin/fsck_ext2fs and the actual tool is /sbin/e2fsck.
Comment 2 Matthias Andree freebsd_committer 2020-09-13 08:34:24 UTC
Oh, and the proliferation of options and required testing isn't what I intend to spend (or should I spell that "waste") time on. We have too many options already and for FreeBSD-provided packages they don't apply anyways.

Feel free to hack your pkg-install script locally.
Comment 3 Baptiste Daroussin freebsd_committer 2020-09-14 07:26:39 UTC
While this is something strongly not recommended to touch anything out of base, and the first class citizen nature of ext* on FreeBSD is imho deattable but that is another topic.

But the pkg-install script is for sure an issue:

Regarding this ticket specifically:
- e2fsck should be installed in base and deinstall in base using regular plist mecanism and using post-install target from the Makefile. I see the reason why it is done in script is trying to hardlink it first, this seems an unnecessary optimisation why not in that case just install in /sbin.
- from quick glance mke2fs.conf dealing seems like it could benefit from a swicth to @sample

The following part:
is useless

Note that I still strongly think the port should be installed outside of localbase as the only point for it is for people having / and/or /usr/local running on ext* given the quality of our support I strongly doubt that it is the case at all
Comment 4 Matthias Andree freebsd_committer 2020-09-14 18:52:09 UTC
(In reply to Baptiste Daroussin from comment #3)
Bapt, quick glances don't help. The script is ages old and has several special purposes: the ext4 name changed over time (long ago though), and it contains migration code to handle that transition and configuration file hacks.

You don't lay out why pkg-plist is better than post-install. @sample is only a macro in the end that adds scripts, too.

This is all nit-picking and nothing about malfunctions that would require fixing.
Comment 5 Baptiste Daroussin freebsd_committer 2020-09-14 19:01:06 UTC
Looking more carefully I keep my comment!

Keywords are usually better because we can modify/fix them in a single place without the need for a sweep change.

For example @sample will soon be changed to lua, which will automatically make all packages using it run it in a capsicum sandbox! have that part be cross installation friendly and have that rootdir friendly.

For instance the current e2fsprogs is not rootdir friendly at all because of the cp in / and the config file handling
Comment 6 Matthias Andree freebsd_committer 2020-09-14 19:20:05 UTC
'nuff of that useless talk. I've unsubscribed from this PR.
Comment 7 Matthias Andree freebsd_committer 2020-09-14 19:48:05 UTC
portmgr@ would also do good to post the upcoming changes, migration plans, necessities in prominent places. Maybe a new section in the porter's handbook, maybe at least CHANGES, and playing dumb looking for "rootdir", in those two places, I don't find references.

And I don't mean to make changes that incur new maintenance and testing debt without real good reason.

For stuff that doesn't need a migration, @sample might be possible (for one of the files), but that should then happen with the next port update when I need to test things anyways, not out of the blue "just because".

So what is this rootdir thing that you're talking about, where is the documentation, and why does a port need to care?
Comment 8 Matthias Andree freebsd_committer 2020-09-16 14:06:21 UTC
This can't be solved without fixing up pkg first. See blocking PR.
Comment 9 Matthias Andree freebsd_committer 2020-09-16 17:46:46 UTC
Packing the hard links directly so they are exposed through pkg-plist doesn't work everywhere, as seen in Bug#249374, and the extra size that results in packing copies of two files to overcome such situations is not absorbed by compression gains:

-rw-r--r--  2 mandree  wheel  1580164 16 Sep. 19:39 e2fsprogs-1.45.6_4copy.txz
-rw-r--r--  1 mandree  wheel  1388016 16 Sep. 19:38 e2fsprogs-1.45.6_4links.txz

That's the e2fsck (899k) and fsck_ext2fs (15 k) packed twice in the "copy" file.

So the only alternative left would be to pack the files into /sbin (that's where they belong if needed for boot-up) and create symbolic links from ${PKG_PREFIX} or ${PREFIX}. Someone may be annoyed...
Comment 10 Matthias Andree freebsd_committer 2020-09-16 19:20:25 UTC
Created attachment 218010 [details]
proposed fixed port, also fixes some borderline cases - see changes_4

kaktus@ can you try this and report back?
Comment 11 Matthias Andree freebsd_committer 2020-09-16 22:23:59 UTC
Note the "fixed port" will keep installing into /sbin but properly register those files so as to expose them to pkg which and thereabouts, and ${PREFIX} (not ${LOCALBASE} - that's for "input to the port", $PREFIX is for the directories that the port's files go into) will symlink (if cross file system) or hardlink (else, from pkg-install) to /sbin.

That's the best we can do with the current state of our infrastructure.
Comment 12 Pawel Biernacki freebsd_committer 2020-09-17 10:39:47 UTC
It doesn't fix the issue described.

In the meantime, I was thinking about what brooks@ proposed on freebsd-arch@: to change the fsck and mount commands to look for binaries in other directories too, that would made this port install fsck_extfs2 into ${LOCALBASE}/sbin, and I believe that this is the way to go.
Comment 13 Matthias Andree freebsd_committer 2020-09-17 17:51:19 UTC
(In reply to Pawel Biernacki from comment #12)
Thanks for testing the change. 

It is not intended or documented to fix the outside-LOCALBASE installation, and it cannot, it needs to be available before /usr is mounted.  It should however make "pkg which /sbin/e2fsck" tell you that e2fsprogs brought it and be more careful about system resources.

The traditional partition layout keeps /usr separate thus anything there isn't available early in the boot, and the traditional LOCALBASE is /usr/local, so we can't make that requested change before all versions that have had traditional partitioning options (/ separate from /usr) have been EOL for a while.

You are noticing that I am pushing back pretty hard on adding options to ports, because that would incur additional testing overhead I cannot currently invest.
Comment 14 Matthias Andree freebsd_committer 2020-09-18 09:43:33 UTC
bapt@ swills@ just to defang any potential "anyone can maintain the port" I'd recommend reading through https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242798 and other hard to debug closed PRs.
Comment 15 commit-hook freebsd_committer 2020-09-23 12:45:11 UTC
A commit references this bug:

Author: mandree
Date: Wed Sep 23 12:44:18 UTC 2020
New revision: 549723
URL: https://svnweb.freebsd.org/changeset/ports/549723

  - make /sbin/e2fsck and /sbin/fsck_ext2fs visible in pkg catalog/"pkg which",
    so that "pkg which /sbin/e2fsck" yields the proper result.

    * this entails symlinking from $PREFIX/sbin to /sbin, and the pkg-install
      script will attempt to replace the symlinks by hard links if possible.
      pkg 1.15.4 cannot deal with packaged hard links and will fail.

    * Note that it is unavoidable that these be in /sbin because /usr/local or
      /usr may not be mounted and consequently ext2 file systems could not
      be fsck-ed or mounted from /etc/fstab.

    There will be no port option to avoid /sbin installs for now. We have too
    many options already and the testing effort increases exponentially.

  - make sure pkg-message appears on both install and upgrade

  - clean up and document/comment pkg-install so that armchair experts
    will not pester me with meaningless change requests

  - bugfix/change: logic of mke2fs.conf upgrade handling to present less work for
    users on port/package upgrades


  PR:	249284 (related)

Comment 16 Matthias Andree freebsd_committer 2020-09-23 12:45:23 UTC
(edit Summary LOCALBASE -> PREFIX because LOCALBASE contains port requisites, and PREFIX is where the port is supposed to install)
Comment 17 Oleg Ginzburg 2021-06-02 08:39:46 UTC
Unfortunately this port (and everything that depends on it, e.g: 'misc/mc' ) can no longer be installed to jail when using a read-only overlay for the base:

pkg install misc/mc
pkg: Fail to create temporary file: /sbin/.pkgtemp.e2fsck.2kw95CtFaizZ:Read-only file system

env BATCH=no make -C /usr/ports/sysutils/e2fsprogs install
===>  Installing for e2fsprogs-1.46.2
===>  Checking if e2fsprogs is already installed
===>   Registering installation for e2fsprogs-1.46.2

pkg-static: Fail to create temporary file: /sbin/.pkgtemp.e2fsck.HfpUwx78yufu:Read-only file system
cp: /usr/local/etc/mke2fs.conf.dist: No such file or directory
pkg-static: POST-INSTALL script failed
*** Error code 1

make: stopped in /usr/ports/sysutils/e2fsprogs

I don't need e2fsprogs in the jail, but I need a 'mc' ;)
Comment 18 Matthias Andree freebsd_committer 2021-06-02 18:35:39 UTC
Comment 19 Matthias Andree freebsd_committer 2021-06-02 18:43:46 UTC
Considering how to best solve this. Ideas:
- split port into one that only does $PREFIX and a new one that does the /sbin and other base system stuff (links)
- create option/flavors to turn base install off (but default to install because you need /sbin/fsck_ext2fs to avoid user astonishment).

Anyone with reasons in favor of either?
Comment 20 Oleg Ginzburg 2021-06-02 19:07:14 UTC
(In reply to Matthias Andree from comment #19)

It seems that the option/flavors is the most suitable compromise.
Comment 21 Michael Osipov 2021-06-22 09:24:04 UTC
(In reply to Oleg Ginzburg from comment #17)

Same here!
Comment 22 commit-hook freebsd_committer 2021-06-24 08:04:45 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=4eb82883317e97c3851e041c415fca854bf062fc

commit 4eb82883317e97c3851e041c415fca854bf062fc
Author:     Alexey Dokuchaev <danfe@FreeBSD.org>
AuthorDate: 2021-06-24 08:02:58 +0000
Commit:     Alexey Dokuchaev <danfe@FreeBSD.org>
CommitDate: 2021-06-24 08:03:15 +0000

    misc/mc: the port had been improved (+)

    - By popular demand, disable EXTATTR option by default: the benefits
      it provides are outweighed by having to pull `sysutils/e2fsprogs'
      port as dependency and various troubles people are having with it.
      While here, adjust the description as it was is a bit misleading:
      it is not limited exclusively to ext2fs, but can also manage UFS-
      specific flags like append-only, etc. [1]

    - Fix ZIP/UNZIP program detection and add missing dependency on the
      `archivers/zip' as FreeBSD does not provide native zip(1) program.
      This bug had been present since late 2014: when fixing PR 193766,
      an incomplete patch had been committed; it went unnoticed because
      apparently users rarely create ZIP archives, and extraction worked
      because `archivers/unzip' is very commonly installed package [2]

    PR:     249284, 256766 [1]
            256546, 193766 [2]

 misc/mc/Makefile | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
Comment 23 Matthias Andree freebsd_committer 2021-08-04 21:29:57 UTC
since the v1.46.3 updates (only in FreeBSD-latest for now), there is a nobootfsck FLAVOR of e2fsprogs available (sysutils/e2fsprogs@nobootfsck, or e2fsprogs-nobootfsck) which avoids installing into /sbin.