Bug 197317 - libncurses: Split into separate libncurses and libtinfo
Summary: libncurses: Split into separate libncurses and libtinfo
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: Normal Affects Some People
Assignee: Baptiste Daroussin
URL: https://reviews.freebsd.org/D32098
Depends on:
Reported: 2015-02-04 11:09 UTC by Vitaly Magerya
Modified: 2021-10-27 00:04 UTC (History)
6 users (show)

See Also:
koobs: mfc-stable13?
koobs: mfc-stable12?
koobs: mfc-stable11-


Note You need to log in before you can comment on or make changes to this bug.
Description Vitaly Magerya 2015-02-04 11:09:53 UTC
Some time ago I (and others as well) have encountered a problem
with Python 'curses' module, the solution to which requires a
modification to how we build ncurses in base.

Here's the background of the problem.

1. Our build scripts for ncurses in base (see lib/ncurses and
contrib/ncurses) produce one big library with all the curses
functions, libncurses.so, and then symlinks smaller libraries
like libtermcap.so and libtinfo.so to libncurses.so. They also
produce a Unicode version of that library, libncursesw.so, with
a corresponding set of symlinks.

2. Other libraries like libreadline.so (from devel/readline, but
previously it was in the base as well), use the 'tgetent' family
of functions contained in libtermcap (or libtinfo, which provides
those functions as well). Since libtermcap.so is a symlink,
libreadline.so gets linked to libncurses.so.

3. Python has a module 'readline' that uses libreadline.so, and
another module 'curses' that uses the Unicode version of ncurses,
libncursesw.so. Some programs may have both of these modules
loaded, which leads to both libncurses.so (through libreadline)
and libncursesw.so loaded into the same process.

4. Since both libncurses and libncursesw have identical set of
functions, linking to them both may lead to problems. In fact,
Python test suite was crashing due to this interaction; for this
reason a workaround was added to Python build scripts that checked
if libreadline.so is linked with libncurses.so, and if so, it
would link Python's 'curses' module with libncurses.so, instead
of libncursesw.so. This prevented crashing, but made displaying
Unicode text via the 'curses' module impossible. See Python issue
7384 [1] for how this came to pass.

Due to all of this, Python's 'curses' module does not display
Unicode characters correctly on FreeBSD (see PR 171246 [2] for
a test case).

The fix for this problem is to split ncurses into separate
libtinfo.so (not a symlink!) and libncurses.so, with the latter
linking with the former. This is what the upstream ncurses does
(also, devel/ncurses). This is also what most Linux distros do,
and Python works well in this setup. Note that with this change
libncurses.so will still provide the same set of functions, so
no regression will occur.

Unfortunately we don't use native ncurses build scripts in base,
and I don't know how to modify our custom ones to this effect,
which is why this PR is without a patch. So, if anyone knows a
way to make our build scripts for ncurses to install a separate
libtinfo.so, that would be very much appreciated.

[1] http://bugs.python.org/issue7384
[2] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=171246
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2015-02-04 11:14:38 UTC
Lovely issue report, thank you for following up on this :)

CC'ing bapt since he is quite familiar with our situation with ncurses
Comment 2 Jilles Tjoelker freebsd_committer 2015-02-08 21:32:21 UTC
Make sure to move lib/libedit from ncursesw to tinfo as well, to avoid slowing down fork() in sh and other programs. Programs that do use ncurses or ncursesw will start and fork() slightly more slowly but I think that is acceptable.
Comment 3 commit-hook freebsd_committer 2017-06-28 02:38:38 UTC
A commit references this bug:

Author: dbaio
Date: Wed Jun 28 02:37:54 UTC 2017
New revision: 444534
URL: https://svnweb.freebsd.org/changeset/ports/444534

  lang/python{27,33,34,35,36}: Make Python curses module work with Unicode

  Use readline from ports (USES= readline:port) and patch
  setup.py to ignore readline from base. The patch is necessary for
  FreeBSD < 1100000, as after this the readline library became an
  INTERNALLIB, see base r268461 [1]

  Link devel/readline against termcapw instead of termcap is part of
  this change, see ports r444463 [2]

  Note that this is the **ports** approach for getting Python curses
  module working with Unicode. The other way is splitting libncurses
  into separate libncurses and libtinfo in base, for which an open
  issue exists [3].

  Apart from Python language ports, at least www/rtv and
  sysutils/py-ranger ports have been tested to work correctly
  (display Unicode) after this change.

  [1] https://svnweb.freebsd.org/changeset/base/268461
  [2] https://svnweb.freebsd.org/changeset/ports/444463
  [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197317

  PR:		171246, 197317
  Reported by:	Vitaly Magerya <vmagerya gmail com>
  Reviewed by:	garga, koobs, miwi, sunpoet
  Approved by:	garga (mentor), sunpoet (python, with hat)
  Differential Revision:	https://reviews.freebsd.org/D11127

Comment 4 Baptiste Daroussin freebsd_committer 2021-09-24 13:53:24 UTC
It is going to finally happen: https://reviews.freebsd.org/D32098
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2021-09-25 22:58:33 UTC