Some time ago I (and others as well) have encountered a problem with Python 'curses' module, the solution to which requires a modification to how we build ncurses in base. Here's the background of the problem. 1. Our build scripts for ncurses in base (see lib/ncurses and contrib/ncurses) produce one big library with all the curses functions, libncurses.so, and then symlinks smaller libraries like libtermcap.so and libtinfo.so to libncurses.so. They also produce a Unicode version of that library, libncursesw.so, with a corresponding set of symlinks. 2. Other libraries like libreadline.so (from devel/readline, but previously it was in the base as well), use the 'tgetent' family of functions contained in libtermcap (or libtinfo, which provides those functions as well). Since libtermcap.so is a symlink, libreadline.so gets linked to libncurses.so. 3. Python has a module 'readline' that uses libreadline.so, and another module 'curses' that uses the Unicode version of ncurses, libncursesw.so. Some programs may have both of these modules loaded, which leads to both libncurses.so (through libreadline) and libncursesw.so loaded into the same process. 4. Since both libncurses and libncursesw have identical set of functions, linking to them both may lead to problems. In fact, Python test suite was crashing due to this interaction; for this reason a workaround was added to Python build scripts that checked if libreadline.so is linked with libncurses.so, and if so, it would link Python's 'curses' module with libncurses.so, instead of libncursesw.so. This prevented crashing, but made displaying Unicode text via the 'curses' module impossible. See Python issue 7384 [1] for how this came to pass. Due to all of this, Python's 'curses' module does not display Unicode characters correctly on FreeBSD (see PR 171246 [2] for a test case). The fix for this problem is to split ncurses into separate libtinfo.so (not a symlink!) and libncurses.so, with the latter linking with the former. This is what the upstream ncurses does (also, devel/ncurses). This is also what most Linux distros do, and Python works well in this setup. Note that with this change libncurses.so will still provide the same set of functions, so no regression will occur. Unfortunately we don't use native ncurses build scripts in base, and I don't know how to modify our custom ones to this effect, which is why this PR is without a patch. So, if anyone knows a way to make our build scripts for ncurses to install a separate libtinfo.so, that would be very much appreciated. [1] http://bugs.python.org/issue7384 [2] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=171246
Lovely issue report, thank you for following up on this :) CC'ing bapt since he is quite familiar with our situation with ncurses
Make sure to move lib/libedit from ncursesw to tinfo as well, to avoid slowing down fork() in sh and other programs. Programs that do use ncurses or ncursesw will start and fork() slightly more slowly but I think that is acceptable.
A commit references this bug: Author: dbaio Date: Wed Jun 28 02:37:54 UTC 2017 New revision: 444534 URL: https://svnweb.freebsd.org/changeset/ports/444534 Log: lang/python{27,33,34,35,36}: Make Python curses module work with Unicode Use readline from ports (USES= readline:port) and patch setup.py to ignore readline from base. The patch is necessary for FreeBSD < 1100000, as after this the readline library became an INTERNALLIB, see base r268461 [1] Link devel/readline against termcapw instead of termcap is part of this change, see ports r444463 [2] Note that this is the **ports** approach for getting Python curses module working with Unicode. The other way is splitting libncurses into separate libncurses and libtinfo in base, for which an open issue exists [3]. Apart from Python language ports, at least www/rtv and sysutils/py-ranger ports have been tested to work correctly (display Unicode) after this change. [1] https://svnweb.freebsd.org/changeset/base/268461 [2] https://svnweb.freebsd.org/changeset/ports/444463 [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197317 PR: 171246, 197317 Reported by: Vitaly Magerya <vmagerya gmail com> Reviewed by: garga, koobs, miwi, sunpoet Approved by: garga (mentor), sunpoet (python, with hat) Differential Revision: https://reviews.freebsd.org/D11127 Changes: head/lang/python27/Makefile head/lang/python33/Makefile head/lang/python34/Makefile head/lang/python35/Makefile head/lang/python36/Makefile
It is going to finally happen: https://reviews.freebsd.org/D32098
Hoorah!