Bug 221349

Summary: When COMPRESS_EXT is anything other than .gz, plist may break
Product: Ports & Packages Reporter: Mikhail T. <mi>
Component: Ports FrameworkAssignee: Port Management Team <portmgr>
Status: Open ---    
Severity: Affects Some People CC: mi, ports-bugs
Priority: ---    
Version: Latest   
Hardware: Any   
OS: Any   

Description Mikhail T. 2017-08-08 22:04:48 UTC
I set COMPRESS_CMD to "xz -9c" on my system and, correspondingly, COMPRESS_EXT -- to ".xz".

Things under src/ rebuilt nicely, as expected. Most things under ports/ seem to ignore the setting, which is mildly disappointing, but acceptable.

The ports-mgmt/portupgrade, however, tries to observe it, but fails:

===> Parsing plist
===> Checking for items in STAGEDIR missing from pkg-plist
Error: Orphaned: man/man1/pkg_deinstall.1.xz.gz
Error: Orphaned: man/man1/pkg_fetch.1.xz.gz
Error: Orphaned: man/man1/pkg_glob.1.xz.gz
Error: Orphaned: man/man1/pkg_sort.1.xz.gz
Error: Orphaned: man/man1/pkg_which.1.xz.gz
Error: Orphaned: man/man1/pkgdb.1.xz.gz
Error: Orphaned: man/man1/pkgdu.1.xz.gz
Error: Orphaned: man/man1/portcvsweb.1.xz.gz
Error: Orphaned: man/man1/portinstall.1.xz.gz
Error: Orphaned: man/man1/ports_glob.1.xz.gz
Error: Orphaned: man/man1/portsclean.1.xz.gz
Error: Orphaned: man/man1/portsdb.1.xz.gz
Error: Orphaned: man/man1/portsvnweb.1.xz.gz
Error: Orphaned: man/man1/portupgrade.1.xz.gz
Error: Orphaned: man/man1/portversion.1.xz.gz
Error: Orphaned: man/man5/pkgtools.conf.5.xz.gz
===> Checking for items in pkg-plist which are not in STAGEDIR
Error: Missing: man/man1/pkg_deinstall.1.gz
Error: Missing: man/man1/pkg_fetch.1.gz
Error: Missing: man/man1/pkg_glob.1.gz
Error: Missing: man/man1/pkg_sort.1.gz
Error: Missing: man/man1/pkg_which.1.gz
Error: Missing: man/man1/pkgdb.1.gz
Error: Missing: man/man1/pkgdu.1.gz
Error: Missing: man/man1/portcvsweb.1.gz
Error: Missing: man/man1/portinstall.1.gz
Error: Missing: man/man1/ports_glob.1.gz
Error: Missing: man/man1/portsclean.1.gz
Error: Missing: man/man1/portsdb.1.gz
Error: Missing: man/man1/portsvnweb.1.gz
Error: Missing: man/man1/portupgrade.1.gz
Error: Missing: man/man1/portversion.1.gz
Error: Missing: man/man5/pkgtools.conf.5.gz
===> Error: Plist issues found.
*** Error code 1

The logic handling man-page extensions in bsd.ports.mk (and elsewhere?) needs fixing -- it seems to consider only .gz as the only valid extension for man-pages, which just is not correct any more. Even on 10.x man(1) is perfectly capable of handling xz-compressed texts...
Comment 1 Baptiste Daroussin freebsd_committer 2019-09-05 13:29:23 UTC
mandoc can only work with .gz compressed manpages as it uses direcly zlib to read them.
While man itself can still work with other compression format due to the fact we do still use the shell wrapper on top of mandoc, other tools cannot, so supporting anything else than .gz in ports as no sense.
Comment 2 Mikhail Teterin freebsd_committer 2019-09-05 18:14:54 UTC
If it is possible to specify a different extension for the compressed man-pages -- if the COMPRESS_EXT knob is exposed -- then it should be respected. The truth of this statement seems self-evident.

And it is respected in the src/ part of FreeBSD -- and has been for many years.

The ports/ part, however, ignores the knob. The ".gz" is hardcoded in multiple parts of bsd.port.mk. This is wrong regardless of whether or not mandoc -- who calls it directly anyway? -- can or cannot read files compressed by anything other than libz itself.

> While man itself can still work with other compression formats due to the
> fact we do still use the shell wrapper on top of mandoc, other tools
> cannot, so supporting anything else than .gz in ports as no sense.

I'm not sure, what "other tools" you have in mind here, but chances are, by the time we replace the man-script with anything else (what? why?), that something else will be able to read the other formats the way zcat already can.

Since filing this ticket two years ago, I switched all my machines to compress man-pages with "xz -9" (COMPRESS_CMD), while still using the ".gz" extension (COMPRESS_EXT). This creates strangely-named (perhaps MISnamed) files:

    % file /opt/man/man1/pkg_glob.1.gz /usr/share/man/man1/ls.1.gz 
    /opt/man/man1/pkg_glob.1.gz: XZ compressed data
    /usr/share/man/man1/ls.1.gz: XZ compressed data

but otherwise works fine -- the texts are compressed most efficiently, while working around the bug in bsd.port.mk.

I understand, that fixing it is not trivial, because some of the ported software compresses man-pages (with gzip) on its own. Ideally, these would be uncompressed and then redone with the specified COMPRESS_CMD.

Alternatively, you could declare, that COMPRESS_CMD (and COMPRESS_EXT) are not applicable to and ignored by the ports/ world. But then you'll need to change/patch the few ports, that currently respect the two knobs (or try to) -- ports-mgmt/pkg itself included. (I do not like this option, but it would, at least, be consistent.)

At any rate, the resolution "Works As Intended" is wrong -- because it does not. If portmgr@ does not want to fix this -- but still wishes the ticket closed for some reason -- something like "Not Accepted" should be the resolution.
Comment 3 Mikhail Teterin freebsd_committer 2019-09-05 18:26:45 UTC
> ports-mgmt/pkg itself included

Sorry, the pkg already uses gzip unconditionally. It is portupgrade, that installs pkg_glob from my example -- and respects COMPRESS_CMD.

Other examples (from my desktop) include:

 * tradcpp
 * OpenSSH-askpass
 * bsdgames
Comment 4 Baptiste Daroussin freebsd_committer 2019-09-06 08:44:11 UTC
by other tools I mean makewhatis, apropos etc. mandoc only populates its db with uncompressed or gzip compressed files.

IMHO the mechanism to switch the compression in the ports tree should be entirely removed due to that.

I am not saying we will replace right now man(1) with something else (but most of the OS who switched to mandoc did it, so it is a plausible future).