Bug 227922 - man -k/-f does not search man pages/mandoc.db from packages/ports (MANPATH problem)
Summary: man -k/-f does not search man pages/mandoc.db from packages/ports (MANPATH pr...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 11.1-STABLE
Hardware: amd64 Any
: --- Affects Many People
Assignee: Yuri Pankov
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-05-02 12:39 UTC by Jeremy Chadwick
Modified: 2024-01-10 07:20 UTC (History)
6 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jeremy Chadwick 2018-05-02 12:39:05 UTC
It appears that man -k (a.k.a. apropos) and man -f (a.k.a. whatis) have been broken for several years now.  Rather than just give some terse explanation, I'm going to provide proof/evidence first, followed by the fixes last.

Looks broken:

$ man -k IPC::Open3
apropos: nothing appropriate
$ man -f IPC::Open3
whatis: nothing appropriate
$ apropos -k IPC::Open3
apropos: nothing appropriate
$ whatis -f IPC::Open3
whatis: nothing appropriate

Man page does exist:

$ find /usr/local/lib/perl5 -name "IPC::Open3*" -print
/usr/local/lib/perl5/5.26/perl/man/man3/IPC::Open3.3.gz

What does manpath say?

$ manpath
/usr/share/man:/usr/local/man:/usr/share/openssl/man:/usr/local/lib/perl5/site_perl/man:/usr/local/lib/perl5/5.26/perl/man

Looks correct.  Double verification that the perl pkg did the right thing:

$ cat /usr/local/etc/man.d/perl5.conf
MANPATH /usr/local/lib/perl5/site_perl/man
MANPATH /usr/local/lib/perl5/5.26/perl/man

And mandoc.db from makewhatis?  Looks OK...

$ find /usr/local -name "mandoc.db" -print 2>/dev/null
/usr/local/lib/perl5/site_perl/man/mandoc.db
/usr/local/lib/perl5/5.26/perl/man/mandoc.db
/usr/local/man/mandoc.db

What if we specify the MANPATH directly?  Wow, it works:

$ apropos -M /usr/local/lib/perl5/5.26/perl/man -k IPC::Open3
IPC::Open3(3) - open a process for reading, writing, and error handling using open3()
$ whatis -M /usr/local/lib/perl5/5.26/perl/man -f IPC::Open3
IPC::Open3(3) - open a process for reading, writing, and error handling using open3()

Okay, let's dig around with truss and see what's happening in apropos and whatis, since they're C programs (man is not -- man is a shell script):

$ file /usr/bin/man /usr/bin/apropos /usr/bin/whatis
/usr/bin/man:     POSIX shell script, ASCII text executable
/usr/bin/apropos: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.1 (1101515), FreeBSD-style, not stripped
/usr/bin/whatis:  ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.1 (1101515), FreeBSD-style, not stripped

$ truss -f -- apropos -k IPC::Open3 2>&1 | grep man.d
$ truss -f -- whatis -f IPC::Open3 2>&1 | grep man.d
$ truss -f -- whatis -f IPC::Open3 2>&1 | grep opendir
$ truss -f -- apropos -k IPC::Open3 2>&1 | grep opendir
$ truss -f -- apropos -k IPC::Open3 2>&1 | egrep 'chdir|mandoc.db'
 6011: chdir("/usr/share/man")                   = 0 (0x0)
 6011: openat(AT_FDCWD,"mandoc.db",O_RDONLY,00)  = 3 (0x3)
 6011: chdir("/usr/local/man")                   = 0 (0x0)
 6011: openat(AT_FDCWD,"mandoc.db",O_RDONLY,00)  = 3 (0x3)
 6011: chdir("/home/jdc")                        = 0 (0x0)
$ truss -f -- whatis -f IPC::Open3 2>&1 | egrep 'chdir|mandoc.db'
 6018: chdir("/usr/share/man")                   = 0 (0x0)
 6018: openat(AT_FDCWD,"mandoc.db",O_RDONLY,00)  = 3 (0x3)
 6018: chdir("/usr/local/man")                   = 0 (0x0)
 6018: openat(AT_FDCWD,"mandoc.db",O_RDONLY,00)  = 3 (0x3)
 6018: chdir("/home/jdc")                        = 0 (0x0)
$ truss -f -- apropos -k IPC::Open3 2>&1 | grep \.conf
 5594: lstat("/etc/libmap.conf",{ mode=-rw-r--r-- ,inode=1926215,size=107,blksize=32768 }) = 0 (0x0)
 5594: openat(AT_FDCWD,"/etc/libmap.conf",O_RDONLY|O_CLOEXEC,00) = 3 (0x3)
 5594: readlink("/etc/malloc.conf",0x7fffffffd520,1024) ERR#2 'No such file or directory'
 5594: open("/etc/man.conf",O_RDONLY,0666)       ERR#2 'No such file or directory'
$ truss -f -- whatis -f IPC::Open3 2>&1 | grep \.conf
 5623: lstat("/etc/libmap.conf",{ mode=-rw-r--r-- ,inode=1926215,size=107,blksize=32768 }) = 0 (0x0)
 5623: openat(AT_FDCWD,"/etc/libmap.conf",O_RDONLY|O_CLOEXEC,00) = 3 (0x3)
 5623: readlink("/etc/malloc.conf",0x7fffffffd520,1024) ERR#2 'No such file or directory'
 5623: open("/etc/man.conf",O_RDONLY,0666)       ERR#2 'No such file or directory'

I see no evidence that apropos or whatis understand /usr/local/etc/man.d/*.conf.  The man pages for these utilities confirm it:

$ man apropos
...
NAME
     apropos, whatis – search manual page databases

...
     By default, apropos searches for makewhatis(8) databases in the default
     paths stipulated by man(1) and uses case-insensitive substring matching
     (the = operator) over manual names and descriptions (the Nm and Nd macro
     keys).  Multiple terms imply pairwise -o.
...
FILES
     mandoc.db      name of the makewhatis(8) keyword database
     /etc/man.conf  default man(1) configuration file
...

What about man's own man page?

$ man man
...
     -d      Print extra debugging information.  Repeat for increased
             verbosity.  Does not display the manual page.
...
FILES
     /etc/man.conf
             System configuration file.
     /usr/local/etc/man.d/*.conf
             Local configuration files.
...

So it should be working, but isn't.  Let's try that debugging flag:

$ man -dddddd -k IPC::Open3
apropos: nothing appropriate
$ man -dddddd -f IPC::Open3
whatis: nothing appropriate

Not helpful.

At this stage I'm thinking: do I delve into 20KB+ of shell script for man, or do I start looking at C code for apropos and whatis?  I decided to look at man first.

What I found:

1. Use of man -k or -f ends up calling do_man (because argv[0] is man, not apropos or whatis like it used to be in old days).

2. do_man immediately calls man_parse_args, which uses shell getopt to parse argv flags and so on.  Within this same code: if -f is set, run do_whatis with arguments.  If -k is set, run do_apropos with arguments.

3. Looking at do_apropos we find this magical beast:

 957 do_apropos() {
 958         [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/apropos) ] && \
 959                 exec apropos "$@"
 960         search_whatis apropos "$@"
 961 }

That's... interesting.  What is with the stat calls?  Why are we comparing inode numbers of these two binaries, and then call exec on apropos if they don't match?

213317     gordon do_apropos() {
285836       bapt       [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/apropos) ] && \
285835       bapt               exec apropos "$@"
213317     gordon       search_whatis apropos "$@"
213317     gordon }

What are these two commits?

r285836 | bapt | 2015-07-24 02:20:02 -0700 (Fri, 24 Jul 2015) | 4 lines

inode should be different to actually mean mandocdb is in used

Sponsored by:   gandi.net

------------------------------------------------------------------------
r285835 | bapt | 2015-07-24 02:10:03 -0700 (Fri, 24 Jul 2015) | 9 lines

Fix man -k with mandocdb

If apropos(1) and whatis(1) are not hardlinks to man(1) that means the system is
using mandocdb, then man -k should spawn apropos(1) and/or whatis(1) directly

Reported by:    kevlo
Tested by:      kevlo
Sponsored by:   gandi.net

So they happened over 3 years ago.  There's no PR/GNATS/bug number in the commit.  Should I search mailing lists?  At this point I opt to say no.  The commit message says that the intended goal was to "fix man -k", so did this break things or did something else cause the breakage (ex. /usr/local/etc/man.d/*.conf introduction?).  My guess is the Gandi folks are building their systems with WITHOUT_MANDOCDB=true or WITHOUT_MAN_UTILS=true (not sure which) in /etc/src.conf.  Anyway, let's look into the man.1 man page to see if we can find out when THAT feature got added.

213317     gordon .Sh FILES
213317     gordon .Bl -tag -width indent -compact
213317     gordon .It Pa /etc/man.conf
213317     gordon System configuration file.
213317     gordon .It Pa /usr/local/etc/man.d/*.conf
213317     gordon Local configuration files.
213317     gordon .El

Long before r285835 and r285836, so /usr/local/etc/man.d/*.conf is not responsible for this.  No wonder I didn't see this breakage in stable/9.

So now that we know who to blame and understand WHY there's breakage, let's figure out how to fix it.

apropos and whatis both comprehend the MANPATH environment variable if present.  On FreeBSD it isn't by default.  FreeBSD has the manpath command, esp. manpath -q which is used in periodic scripts like weekly/320.whatis.  man's man page says it comprehends /usr/local/etc/man.d/*.conf, so surely it has the smarts somewhere, but it isn't getting called when using man -k or man -f.  I changed #! /bin/sh to #! /bin/sh -x and witnessed proof of this.

So back to the man shell script we go, starting from step 4...

4. In do_man(), if man -k or man -f aren't used, the do_apropos and do_whatis aren't called.  So what happens in that case?  Interesting function name:

 969         man_setup

5. man_setup() does some magical things relating to environment variables within the script itself (NOTHING is export-ed).  Near the end of it, I see:

 628         build_manpath
 629         man_setup_locale
 630         man_setup_width

Hey, that build_manpath() looks like it might be relevant.

6. build_manpath() parse all kinds of fun stuff.  Most importantly, it calls a function called parse_configs, which does in fact comprehend /usr/local/etc/man.d/*.conf and so on.  Near the end, build_manpath() sets the env var MANPATH to what should be a proper manpath.

What if we were to call build_manpath ourselves and export MANPATH?  Let's try this code in do_apropos() and then try man -k:

 957 do_apropos() {
 958         build_manpath
 959         export MANPATH
 960         [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/apropos) ] && \
 961                 exec apropos "$@"
 962         search_whatis apropos "$@"
 963 }

$ ./man.sh -k IPC::Open3
IPC::Open3(3) - open a process for reading, writing, and error handling using open3()

It works!  Mostly.

There is some locale-based support in man.sh but it's mainly used by manpath (which is also this script) and the OLD whatis used to do: call search_whatis() back in the old days.

7. search_whatis() does this right at the start:

 817 search_whatis() {
 ...
 825         build_manpath
 826         build_manlocales
 827         setup_pager

Very similar.  build_manlocales() sets an env var called MANLOCALES that is only used within search_whatis() itself.  setup_pager() isn't relevant to man -k output.

So while I can't get the locale bits to work for this because some code from search_whatis() would need to be made its own function (or maybe apropos/whatis modified to support MANLOCALE env var), what I CAN do is get man -f and man -k working with this simple change:

$ svnlite diff
Index: man.sh
===================================================================
--- man.sh      (revision 333170)
+++ man.sh      (working copy)
@@ -955,6 +955,8 @@ whatis_usage() {

 # Supported commands
 do_apropos() {
+       build_manpath
+       export MANPATH
        [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/apropos) ] && \
                exec apropos "$@"
        search_whatis apropos "$@"
@@ -992,6 +994,8 @@ do_manpath() {
 }

 do_whatis() {
+       build_manpath
+       export MANPATH
        [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/whatis) ] && \
                exec whatis "$@"
        search_whatis whatis "$@"

Confirmation:

$ ./man.sh -k IPC::Open3
IPC::Open3(3) - open a process for reading, writing, and error handling using open3()
$ ./man.sh -f IPC::Open3
IPC::Open3(3) - open a process for reading, writing, and error handling using open3()

In summary: I'm amazed that man -k/-f has essentially been broken for almost 3 years (for ports/pkgs, not base system) and nobody's complained.  Like, TRULY amazed/baffled.

So what are apropos/whatis now?  Well, they're from OpenBSD, and the source is in src/contrib/mdocml.  This is code for mandoc, makewhatis, apropos, and whatis.  I just don't see a native way to make this code work with /usr/local/etc/man.d/*.conf so man (the shell script) will have to be the "gateway" for now.

Also: I just want to give a shoutout to Bug 223555 because it looks like apropos/whatis may on stable/10 been the old shell script version which users have come to rely on.  Maybe the tail end of this PR will help explain that.
Comment 1 Jeremy Chadwick 2018-05-02 12:40:34 UTC
Adding relevant committers/people involved in past commits.  I don't know who to assign this to.
Comment 2 Gordon Tetlow freebsd_committer freebsd_triage 2018-05-02 20:59:34 UTC
Over to bapt. You broke it, you bought it. :-)

The patch proposed seems easy enough (at least for the man -k/-f) behavior. Unsure how to get the binaries to deal with it properly.
Comment 3 Jeremy Chadwick 2018-05-05 01:34:06 UTC
A workaround for this bug that doesn't involve modifying /usr/bin/man would be to use this in one of your dotfiles (ex. .bashrc): export MANPATH=`manpath -q`
Comment 4 Yuri Pankov 2018-10-07 07:28:24 UTC
Using manpath(1) output to populate the defpaths for mandoc seems to be the easiest way without doing massive changes to vendor code, see review D17454.
Comment 5 commit-hook freebsd_committer freebsd_triage 2018-10-16 17:17:21 UTC
A commit references this bug:

Author: yuripv
Date: Tue Oct 16 17:17:12 UTC 2018
New revision: 339385
URL: https://svnweb.freebsd.org/changeset/base/339385

Log:
  apropos/whatis: use output of manpath(1) to set defpaths if -M is not
  specified.  This fixes searching the paths specified in
  /usr/local/etc/man.d/*.conf, as currently apropos/whatis from mandoc
  suite aren't aware about them.

  PR:		227922
  Reviewed by:	bapt
  Approved by:	re (gjb), kib (mentor)
  Differential Revision:	https://reviews.freebsd.org/D17454

Changes:
  head/contrib/mandoc/main.c
Comment 6 Jeremy Chadwick 2020-05-07 04:33:57 UTC
Can r339385 please be backported to stable/11 ?  The relevant source bits are in contrib/mdocml rather than contrib/mandoc.  It looks like it should patch cleanly with a larger line offset.  stable/11 is still officially supported as of this writing.

Thanks!
Comment 7 Mark Linimon freebsd_committer freebsd_triage 2024-01-10 07:20:58 UTC
^Triage: committed back in 2018, now in all supported branches.