Bug 275304 - ports-mgmt/portconfig garbled output (after bsddialog update?)
Summary: ports-mgmt/portconfig garbled output (after bsddialog update?)
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Baptiste Daroussin
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-11-24 11:30 UTC by jakub_lach
Modified: 2024-02-28 16:12 UTC (History)
8 users (show)

See Also:
bugzilla: maintainer-feedback? (bapt)


Attachments
locale -a (2.71 KB, text/plain)
2023-11-24 13:46 UTC, jakub_lach
no flags Details
vim (15.94 KB, image/png)
2023-11-24 17:34 UTC, jakub_lach
no flags Details
menu (6.58 KB, image/png)
2023-11-24 17:36 UTC, jakub_lach
no flags Details
Error in X with XFCE terminal and encode single byte chars. (137.80 KB, image/png)
2023-11-28 00:06 UTC, Alfonso S. Siciliano
no flags Details
Remove 'setlocale(LC_ALL, "");' line in portconfig.c (296 bytes, patch)
2024-02-10 20:44 UTC, Joshua Kinard
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description jakub_lach 2023-11-24 11:30:15 UTC
FreeBSD 14.0-STABLE #0 stable/14-a2bb07585: Fri Nov 24 03:29:20 

https://bz-attachments.freebsd.org/attachment.cgi?id=246479 

$ bsddialog --version
Version: 0.4

$ /usr/local/bin/bsddialog --version
mksh: /usr/local/bin/bsddialog: inaccessible or not found

(however, the problem persist after installing devel/bsddialog).

I use xterm and x11-fonts/terminus-font. Problem does not occur in vt (albeit since some time the colors have changed there - previously they were the same as in X).
Comment 1 jakub_lach 2023-11-24 11:32:13 UTC
(In reply to jakub_lach from comment #0)

Please see also bug#274472 for additional test cases.
Comment 2 Baptiste Daroussin freebsd_committer freebsd_triage 2023-11-24 12:32:35 UTC
can you show more about your environement, like locale -a (within your terminal) I am also interested in $TERM output.
Comment 3 jakub_lach 2023-11-24 13:46:15 UTC
Created attachment 246538 [details]
locale -a
Comment 4 jakub_lach 2023-11-24 13:47:19 UTC
(In reply to Baptiste Daroussin from comment #2)

$ print $TERM
xterm
$ xterm -v
XTerm(388)

Thanks for taking an interest!
Comment 5 Alfonso S. Siciliano freebsd_committer freebsd_triage 2023-11-24 16:17:38 UTC
Hello,

From bug #bug 274472  we know bsddialog(1) is not affected by the problem.
So we have to start some test only for portconfig.

To recap: vt is ok, the problem occurs in X with xterm/mksh/x11-fonts/terminus-font in X. Is it Right?

Please could you run:
# pkg delete portconfig
# pkg delete bsddialog
# cd /usr/ports/devel/bsddialog
# make clean
# make install clean
# cd /usr/ports/ports-mgmt/portconfig
# make clean
# make install clean
# rehash

Now, what is the output of:
# /usr/local/bin/bsddialog --version
# /usr/local/bin/bsddialog --menu test 0 0 0 A aaa B bbb
# cd /usr/ports/editors/vim
# make config
# portconfig -v

(Now bsddialog (--menu) and portconfig (make config) should draw the same box-chars because are using the same library).
Comment 6 jakub_lach 2023-11-24 17:34:36 UTC
Created attachment 246543 [details]
vim
Comment 7 jakub_lach 2023-11-24 17:35:45 UTC
(In reply to Alfonso S. Siciliano from comment #5)

$ /usr/local/bin/bsddialog --version                                                                                                         
Version: 1.0.1
Comment 8 jakub_lach 2023-11-24 17:36:01 UTC
Created attachment 246544 [details]
menu
Comment 9 jakub_lach 2023-11-24 17:37:51 UTC
(In reply to jakub_lach from comment #7)

$ portconfig -v
portconfig version: 0.6.1 (libbsddialog: 1.0.1).

(I've did the reinstall according to #5)
Comment 10 jakub_lach 2023-11-24 17:47:25 UTC
(In reply to jakub_lach from comment #9)

Tried running X without .Xdefaults - same problem.
Comment 11 firk 2023-11-26 19:54:45 UTC
I have similar problem.

portconfig & bsddialog installed just 1 hour ago (previously used dialog4ports), an it shows (suspectedly) utf-8 pseudographics even in 8-bit ANSI console.
I didn't run portconfig directly, I run make config for some port.

TERM=xterm
LANG=
LC_ALL=
other LC_*=C


bsddialog --msgbox xxx 0 0

works fine (even without --ascii-lines option).

Don't know if it worked before 1.0.1 update.
Comment 12 firk 2023-11-26 19:59:12 UTC
Additional info:

bsddialog starts to behave badly if I set LANG=C.UTF-8 and not using --ascii-lines.

It looks like portconfig (or its caller) somehow modifying/ignoring LANG= env var and using some another value for it.
Comment 13 Alfonso S. Siciliano freebsd_committer freebsd_triage 2023-11-27 10:20:23 UTC
(In reply to firk from comment #11)

What FreeBSD version are you using? Could you upload a screenshot?
Comment 14 Alfonso S. Siciliano freebsd_committer freebsd_triage 2023-11-27 11:09:10 UTC
(In reply to jakub_lach from comment #10)

Thank you for testing and screenshots.

So, "menu" screenshoot is ok. Looking
> https://bz-attachments.freebsd.org/attachment.cgi?id=246479 
it seems portconfig does not link to ncurses correctly. 

Please, could you test:

install devel/bsddialog if not installed, then
% git clone https://gitlab.com/alfix/portconfig.git
% cd portconfig
% cc -I/usr/local/include portconfig.c -L/usr/local/lib -o portconfig_test -lbsddialog
Add portconfig_test to /etc/make.conf, that is:
DIALOG4PORTS?=/<absolute>/<path>/<to>/portoptscli_test
Now you can retest:
# cd /usr/ports/editors/vim
# make config

Output?
Comment 15 firk 2023-11-27 13:38:02 UTC
> What FreeBSD version are you using? Could you upload a screenshot?

12.4, and no screenshot, it is native text mode. Anyway, Eugene already figured out the cause: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275364
Comment 16 jakub_lach 2023-11-27 15:00:37 UTC
(In reply to Alfonso S. Siciliano from comment #14)

No change, output is the same.

(I've pulled git src, compiled portconfig_test and added its path to make.conf)
Comment 17 Alfonso S. Siciliano freebsd_committer freebsd_triage 2023-11-28 00:06:43 UTC
Created attachment 246622 [details]
Error in X with XFCE terminal and encode single byte chars.

(In reply to jakub_lach from comment #16)

Finally, I can reproduce the problem in X setting xfce terminal to encode 8859-1 single-byte chars (screenshot).

For temporary fix you could add to /etc/make.conf:
D4PASCIILINES=Y
(to use ascii box-drawing chars).

Of course a proper solution is described in bug #275364; just a while to discuss with others.
Comment 18 Peter Much 2024-01-05 12:45:22 UTC
There is more to it:

The thing does mis-handle curses and does not know how to scroll correctly.

Ctrl-L (which should redraw the screen in such case) seems not even implemented.

And, BTW, the mouse support seems also to have disappeared.

Photos and discussion here, please: 
https://forums.freebsd.org/threads/2024q1-portbuild-trouble.91770/#post-636922

This is not a training ship!
Comment 19 Joshua Kinard 2024-02-10 20:44:59 UTC
Created attachment 248336 [details]
Remove 'setlocale(LC_ALL, "");' line in portconfig.c

I am having this problem on a Windows system that is running PuTTY v0.80.  On my Linux/BSD systems, I use charset=ISO-8859-15, lang=en_US.ISO8859-15, and in PuTTY, set the window translation to "ISO-8859-15:1999".  I do this mostly to get enough coverage for some international characters, but to also avoid the "curly" (smart) quotes appearing in my terminal (compiler output is notorious for switching to curly quotes if UTF8 is enabled).

It seems that the culprit in portconfig's source for causing the garbled output when using a non-UTF8 locale is the "setlocale(LC_ALL, "");" on line 200 in portconfig.c.  Removing this line causes the correct line-drawing characters to be used in both UTF8 and non-UTF8 cases.  Only tested with PuTTY, so could use some testing against other terminal emulators.
Comment 20 Alfonso S. Siciliano freebsd_committer freebsd_triage 2024-02-11 23:59:41 UTC
(In reply to Joshua Kinard from comment #19)

Thanks for the report,

please could you post output:

% portconfig -v
% /usr/local/bin/bsddialog --version

(I suspect it is an old (fixed) problem with putty/ncurses).
Comment 21 Joshua Kinard 2024-02-12 03:06:06 UTC
(In reply to Alfonso S. Siciliano from comment #20)
> Thanks for the report,
> 
> please could you post output:
Sure, see below:
# portconfig -v
portconfig version: 0.6.1 (libbsddialog: 1.0.1).

# /usr/local/bin/bsddialog --version
Version: 1.0.1

# freebsd-version
14.0-RELEASE-p4

> (I suspect it is an old (fixed) problem with putty/ncurses).
I am running the latest PuTTY release (0.80), so not too sure about that.  $TERM is "xterm-256color", too, which on FreeBSD seems to work pretty well w/ PuTTY (On Linux, it's a bit of a different story).  I tested my fix for the cases where "charset" and "lang" were both set to ISO8859-15 in /etc/login.conf and PuTTY set to the same, as well as all being set to UTF8, and could not reproduce the garbled line-drawing output.

The garbled line output also started fairly recently, too.  Unsure if that's due to the initial changeover to bsddialog, though.  I tried looking back through the commit history of portconfig, but I can't quickly find the point in time where that setlocale() call was added (github's UI is rather terrible for digging through commit histories).
Comment 22 Alfonso S. Siciliano freebsd_committer freebsd_triage 2024-02-12 13:11:46 UTC
(In reply to Joshua Kinard from comment #21)

Thank you for testing and the answer. Of course, reading your comments,
also your problem is related to bug #275364.
Actually we should merge these PRs.

Briefly, we know the cause and some possible solution exist (commenting setlocale() could be), we are thinking about the better solution.


> I am running the latest PuTTY release (0.80), so not too sure about that.  
Actually it is not a bug, it is a ncurses feature NCURSES_NO_UTF8_ACS (bug #274472 and review D42380). Anyway you are portconfig 0.6.1 (libbsddialog: 1.0.1), so you can ignore this env as I changed borders chars from utf-8 to wide-chars. This is not the current problem.

> "setlocale(LC_ALL, "");
> Removing this line causes the correct line-drawing characters to be used
> in both UTF8 and non-UTF8 cases.
Of course, this is a solution, not the better: using always C/POSIX locale.
When portconfig starts (like other programs) its locale is C (aka POSIX), that is ASCII chars. Then ncurses handles this setting automatically and correctly. 
portconfig calls setlocale(LC_ALL, "") to set the locale like the current environment. So if it is unicode, ncurses handles automatically the chars set. Example arrows, ASCII: v ^ < > become unicode: ↓ ↑ ← →.

Recap.
The problem is: ports framework sets always UTF-8 -> portconfig calls setlocale() -> ncurses uses unicode chars -> if the current console has not UTF-8 chars -> garbled output.

If we delete setlocale() in portconfig, the port framework UTF-8 setting will be ignored (always C Locale).

Yesterday I added a new env to fix, avoiding to set always UTF-8 or C Locale https://gitlab.com/alfix/portconfig/-/commit/725a1e9cec2e042af1268d31c083faaf1b584f31 (it requires also a change to the port framework). I'll contact the portmgr team to find the better solution.