I use x11-fonts/terminus-font. After the update, bold text output has extra width added, similar second row in https://i.stack.imgur.com/DEn6D.png Downgrade to 327 solves the issue. Recent changes related in any way to bold fonts were in 326, so I believe this behaviour might by unintended.
Added author as CC.
thanks - I can see (now that I'm looking) a problem with the bold fonts. I'll see what I can fix.
hmm - the problem I see is not related (the threshold for the fontWarnings resource was lost in my reorganization, which causes extra warnings). The jaggy effect would happen if xterm's not able to get the exact font requested, and the X server scales the font. At the moment I don't have an up-to-date FreeBSD with X (so I'll have to work on that). However, displaying with the MacPorts terminus-fonts shows that effect for the first font (using KOI8RXTerm.ad as a convenient testcase), but displaying directly on my Debian 7 "desktop" shows no problem with fonts for any of those selections. I assume that what I'm looking for is some case where the XLFD strings inside xterm aren't identical to that in #327. So... it would help if I knew what your font-resources are, so that once I've got FreeBSD X setup (which release-- 11?) I'll be able to reproduce the problem.
https://ibb.co/gXBB7a top is #328, bottom is a #327 FreeBSD 11.1-PRERELEASE #0 r319591 amd64 $ pkg info | grep font GentiumBasic-1102 Gentium Basic and Gentium Book Basic TrueType fonts croscorefonts-1.31.0 Fonts created from by Google for ChromeOS to replace MS TTF encodings-1.0.4_3,1 X.Org Encoding fonts font-adobe-100dpi-1.0.3_3 X.Org Adobe 100dpi font font-alias-1.0.3_3 X.Org Font aliases font-bh-ttf-1.0.3_3 X.Org Bigelow & Holmes TTF font font-misc-ethiopic-1.0.3_3 X.Org miscellaneous Ethiopic font font-misc-meltho-1.0.3_3 X.Org miscellaneous Meltho font font-misc-misc-1.1.2_3 X.Org miscellaneous Misc fonts font-util-1.3.1 Create an index of X font files in a directory fontconfig-2.12.1,1 XML-based font configuration API for X Windows fontsproto-2.1.3,1 Fonts extension headers freetype2-2.8 Free and portable TrueType font rendering engine libXfont-1.5.2,2 X font library libXft-2.3.2_1 Client-sided font API for X applications libfontenc-1.1.3_1 The fontenc Library linux-c7-fontconfig-2.10.95_2 XML-based font configuration API for X Windows (Linux CentOS 7.3.1611) linuxlibertine-g-20120116_1 Linux Libertine G and Linux Biolinum G fonts mkfontdir-1.0.7 Create an index of X font files in a directory mkfontscale-1.1.2 Creates an index of scalable font files for X noto-1.0.5_1 Google font family terminus-font-4.40 Terminus Font - a clean fixed width font webfonts-0.30_13 TrueType core fonts for the Web xorg-fonts-truetype-7.7_1 X.Org TrueType fonts Let me know If there's anything more I can help.
that's a package list - but I was asking about the font resource values. Not all fonts provide a bold variant for each size of the font (8x13 comes to mind in the usual ISO-8859-1 font). xterm has a resource for bold font, which is used only for the default fontsize.
(In reply to dickey from comment #5) #327 $ xterm -report-fonts Loaded VTFonts(default) fNorm: -*-terminus-medium-r-normal-*-14-*-*-*-*-*-iso8859-2 all chars: no default char: 46 direction: 0 ascent: 12 descent: 2 first char: 0 last char: 255 maximum-chars: 256 missing-chars: 32 present-chars: 224 min_byte1: 0 max_byte1: 0 properties: 22 min_bounds: lbearing: 0 rbearing: 0 width: 8 ascent: -1 descent: -11 max_bounds: lbearing: 5 rbearing: 8 width: 8 ascent: 12 descent: 2 fBold: -*-terminus-bold-r-*-*-14-*-*-*-*-*-iso8859-2 all chars: no default char: 46 direction: 0 ascent: 12 descent: 2 first char: 0 last char: 255 maximum-chars: 256 missing-chars: 32 present-chars: 224 min_byte1: 0 max_byte1: 0 properties: 22 min_bounds: lbearing: 0 rbearing: 0 width: 8 ascent: 0 descent: -10 max_bounds: lbearing: 4 rbearing: 8 width: 8 ascent: 12 descent: 2 $ xrdb -q| grep -i font xterm*font: -*-terminus-medium-r-normal-*-14-*-*-*-*-*-iso8859-2 xterm*boldFont: -*-terminus-bold-r-*-*-14-*-*-*-*-*-iso8859-2 !XTerm*font: -misc-fixed-medium-r-normal-*-15-*-*-*-*-*-iso8859-2 !XTerm*boldFont: -misc-fixed-bold-r-*-*-15-*-*-*-*-*-iso8859-2 #328 $ xterm -report-fonts Loaded VTFonts(default) fNorm: -*-terminus-medium-r-normal-*-14-*-*-*-*-*-iso8859-2 all chars: no default char: 46 direction: 0 ascent: 12 descent: 2 first char: 0 last char: 255 maximum-chars: 256 missing-chars: 32 present-chars: 224 min_byte1: 0 max_byte1: 0 properties: 22 min_bounds: lbearing: 0 rbearing: 0 width: 8 ascent: -1 descent: -11 max_bounds: lbearing: 5 rbearing: 8 width: 8 ascent: 12 descent: 2 fBold: -xos4-Terminus-bold-R-*-*-14-140-72-72-C-80-ISO8859-2 all chars: no default char: 46 direction: 0 ascent: 12 descent: 2 first char: 0 last char: 255 maximum-chars: 256 missing-chars: 32 present-chars: 224 min_byte1: 0 max_byte1: 0 properties: 22 min_bounds: lbearing: 0 rbearing: 0 width: 8 ascent: 0 descent: -10 max_bounds: lbearing: 4 rbearing: 8 width: 8 ascent: 12 descent: 2 $ xrdb -q| grep -i font xterm*font: -*-terminus-medium-r-normal-*-14-*-*-*-*-*-iso8859-2 xterm*boldFont: -*-terminus-bold-r-*-*-14-*-*-*-*-*-iso8859-2 !XTerm*font: -misc-fixed-medium-r-normal-*-15-*-*-*-*-*-iso8859-2 !XTerm*boldFont: -misc-fixed-bold-r-*-*-15-*-*-*-*-*-iso8859-2
I also am experiencing issues with xterm-328 in vim and zsh. The same resources I use on Arch Linux do not have this issue. I am attaching before and after images of the status lines in vim generated by powerline (https://github.com/powerline/powerline). In zsh, the issue appears if I type two gibberish words then go backwards over the line using the left arrow key. The cursor disappears. Notes: - I see this error on FreeBSD but not Linux: xterm: cannot load font "-Misc-Fixed-bold-R-*-*-18-120-100-100-C-180-ISO10646-1" - I am unable to set the wideBoldFont using -fwb; it appears to be ignored as the error is still shown with it. - XTerm font resources: XTerm*font: -misc-fixed-medium-r-normal-*-18-120-100-100-c-90-iso10646-1 XTerm*wideFont: -misc-fixed-medium-r-normal-*-18-120-100-100-c-180-iso10646-1
Created attachment 183269 [details] xterm-327 vim status line
Created attachment 183270 [details] xterm-328 vim status line
thanks - Jakub has given plenty of information. I'm working to reproduce the problem (which as you see from my tests with MacOS and Debian, the package contents may hide the problem from my usual testing).
I'm able to reproduce the problem, and see that it's in the last set of changes which I made for fonts. I'll study the debug-traces and see what the proper fix would be.
Created attachment 183341 [details] fix for the bold/widebold resource bug The problem was that in replacing old/new structures, I had a special case where there was a structure used to replace a return-value from a function that usually provides only the normal font, and my replacement used just the normal font. This change provides an analogous structure. (I'll put out #328 in a day or so, after some testing).
Thank you Thomas. I'll update the port as soon as the new version is announced. Meanwhile, Jakub and Sean, it would be good if you could give Thomas feedback on the patch.
(In reply to Emanuel Haupt from comment #13) The patch has fixed my problem. Thanks for all involved!
Unfortunately, the patch did not fix my issues. Mine could be a separate bug or could be my configuration (sensitive to xterm-328). I am still suspicious of the error that I reported before about a missing font, but it too may not be the cause. Just in case, here are my build options: ===> The following configuration options are available for xterm-328: 256COLOR=on: Enable 256-color support DABBREV=off: Enable support for dabbrev-expand DECTERM=off: Enable DECterm Locator support GNOME=off: GNOME desktop environment support LOGGING=on: Enable logging terminal I/O to a file LUIT=on: Use LUIT for locale convertion from/to UTF-8 PCRE=off: Use Perl Compatible Regular Expressions REGIS=on: Enable ReGIS graphics support SCRNDUMP=on: Enable XHTML and SVG screen dumps SIXEL=on: Enable Sixel graphics support WCHAR=on: Enable wide-character support XINERAMA=on: X11 Xinerama extension support ====> Options available for the radio 3D: you can only select none or one of them XAW3D=off: Link with Xaw 3d library XAW3DXFT=off: Link with Xaw 3d xft (extended fonts) library NEXTAW=off: Link with neXT Athena library ===> Use 'make config' to modify these settings
I set up the powerline stuff, and can see a problem to investigate (with FreeBSD: for some reason the same configuration works fine with Debian 7).
The explanation was that a problem in the FreeBSD wcwidth() exposed a problem in the logic which I added for soft-hyphen. The FreeBSD wcwidth returns -1 for 0xa0 (non-breaking space). In xterm, it was scanning through buffer (which happened to contain only this character) and because wcwidth said it was actually nonprinting, discarded it. That caused a later check to fail (which asked if the scan ran into the right-margin: the only reason why it should have failed to add the character...). I'll add a workaround for this.
(In reply to dickey from comment #17) I am just checking. The FreeBSD wcwidth() is incorrect or something on how it is used in xterm specifically on FreeBSD is incorrect? I ask because I tried a small test and received -1 on FreeBSD and Linux with the following program: #include <stdio.h> #define __USE_XOPEN #include <wchar.h> int main() { return (printf("%d\n", wcwidth(0xa0))); } Thank you for doing so much for xterm. It is my favorite terminal.
I've updated the port to 329. Sean, did the workaround solve the powerline issue for you?
(In reply to Emanuel Haupt from comment #19) I saw no change in its behavior between 328 and 329 for zsh; the cursor still disappears when going backwards with the arrow key. For vim with powerline, it is a slight improvement. The last line that shows ""/etc/motd" [readonly] 21L, 933C" is correct, but the preceding line is a bit compressed with a lot of whitespace missing between elements. Thomas, would you explain more about the issue regarding wcwidth(), as noted in an earlier comment of mine? For me, Linux and FreeBSD return the same result for wcwidth(0xa). To be more specific about zsh, I am https://github.com/zsh-users/zsh-syntax-highlighting. Without that, I do not see a problem in zsh.
I said "0xa0" (decimal 160), not "0xa" (control/J aka "newline"). In ISO-8859-1, codes 160-255 are printable characters. My change for soft-hyphen used wcwidth() because (at least with Linux), that function has been made locale-sensitive, returning -1 for the soft-hyphen character in UTF-8 locales. I wasn't expecting other characters to become non-printing, which was the case with FreeBSD and non-breaking-space.
Regarding the loss of whitespace: I did consider just ignoring the (probably incorrect...) return from wcwidth() for non-breaking-space, but didn't think that would be an improvement. I noticed that powerline put a half-dozen or so non-breaking-spaces into the vim status line. Regarding zsh: I didn't dig into that (probably should have, but ran short of time relative to when I thought a bug-fix should be available). In your test-program, since you did not call setlocale(LC_ALL, ""), then it was using the POSIX character set (codes 0 to 127). Codes 128 to 255 are nonprinting in the POSIX locale.
I'll be working on #329 this week (mostly bug-fixes, including taking a look at the zsh issue and improving the workaround for non-breaking space).
(In reply to Sean Farley from comment #7) Checking a debugging trace, the failure to load that particular font was done in #327, though the warning didn't show up. It does now, because of the way I split up a rather long function.
If I make a special case for 0xa0, like this diff -u -r1.1488 charproc.c --- charproc.c 2017/06/12 22:00:15 1.1488 +++ charproc.c 2017/06/13 00:43:59 @@ -5085,6 +5085,9 @@ } TRACE(("...will%s display soft-hyphen\n", eat_it ? " not" : "")); + } else if (ch == 0xa0) { + eat_it = False; + last_chomp = 1; } /* * Supposedly we dealt with combining characters and that works around the FreeBSD locale problem. I'll investigate further on that. However, there is a workaround using a resource setting: *mkSamplePass:10 which tells xterm to use its own tables for wcwidth(). You may of course run into problems with that (xterm's tables aren't perfect...), but vim's powerline displays as expected for me using that approach.
The "10" in that resource setting works because xterm finds (by default) 14 differences in the first 1024 characters. My trace shows that a couple of those are problems with FreeBSD's tables, and the others are supposed to be reserved (unused), but not handled as such in xterm's built-in tables. (I made a to-do item to improve that).
(In reply to dickey from comment #21) Oops. I meant 0xa0 as I had coded into the example program. My comment incorrectly said 0xa. :) Thank you for the explanation. I now see that the locale from the environment is not used until the call by setlocale(). I did a bit of searching around and see that a somewhat used hack is to assume the width is 1 if wcwidth() returns -1. However, that is a hack. I was just surprised how common it is. I did not find any FreeBSD-specific calls in esoteric system libraries that could help, but that was a long-shot. Setting mkSamplePass to 10 does help powerline. It does not help zsh, so that is probably a separate issue. Do you see similar problems with xterm on MacOS or Solaris? I have read some old reports that they had similar issues to FreeBSD with wcwidth(). BTW, I ran across this resource for anyone to peruse: http://crashcourse.housegordon.org/coreutils-multibyte-support.html#wcwidth Thanks again.
(In reply to Sean Farley from comment #27) The standard on wcwidth is terse: http://pubs.opengroup.org/onlinepubs/9699919799/functions/wcwidth.html The wcwidth() function shall either return 0 (if wc is a null wide-character code), or return the number of column positions to be occupied by the wide-character code wc, or return -1 (if wc does not correspond to a printable wide-character code). That's a starting point. A -1 says it's not printable. There's more than one thing it could be (only by experimentation do we find the difference between 0 and -1 for combining characters, reserved characters, control characters). But treating a -1 return from wcwidth is treating it as if it were printable anyway. Markus Kuhn's wcwidth uses -1's for control characters (which generally are zero-width except for special cases such as tab, backspace, carriage return and line-feed). That uses some tables generated from the Unicode data files, but since it doesn't deal with reserved characters is incomplete (I'm considering doing that, so I can test FreeBSD better). With a more complete wcwidth, I'd cut down the threshold for this, so FreeBSD's quirk wouldn't be seen in the first place. Solaris locales have their own problems - all of the line-drawing characters are marked as double-width - xterm ends up using its own table there (the Markus Kuhn code). If wcwidth standardization had been thought about, they'd have made most of the ambiguous-width characters locale-dependent rather than making it implementation-dependent.
(In reply to Sean Farley from comment #27) I don't know how to reproduce your problem with zsh: I setup a simple ".zsh" file which just sources the syntax-highlighting script, and that seems to work as expected for line-editing, using the left/right cursor and the backspace character.
(In reply to dickey from comment #28) You are being generous with the "terse" description of wcwidth() documentation. I found a couple of wcwidth implementations on GitHub. The first is an updated(?) version of Markus Kuhn's wcwidth. The second is a port of a Python version of wcwidth. 1. https://github.com/fumiyas/wcwidth-cjk 2. https://github.com/termux/wcwidth The FreeBSD version goes a bit deeper than I followed into the code due to lack of time.
(In reply to dickey from comment #29) For the zsh issue, I experimented and found a simple way to reproduce it. # cat .zshrc . /usr/local/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh ZSH_HIGHLIGHT_HIGHLIGHTERS=(cursor)
(In reply to Sean Farley from comment #30) I wouldn't call it "updated", but "tweaked". I have uniset, and used it to update xterm's copy in 2013 (Unicode 6.2.1) this one is actually older. I've seen a few of these, but since they don't leave a trail, it's hard to say where this one came from. I took a quick look at Unicode 9 a year ago, but didn't notice that it would change the content of wcwidth.c (though I'm checking over that part now, in case something was overlooked). I'll probably adapt uniset to generating the list of intervals for reserved characters.
(In reply to Sean Farley from comment #31) I can see the cursor vanish with these settings (will investigate).
Created attachment 183511 [details] proposed fix for cursor-visibility The problem was that the highlighting scripts use inverse-video. Here is a change which seems to work for me.
(In reply to dickey from comment #34) This appears to fix it for me too. Thank you! I will let you know if I see any thing more about this. I also tested your exception for 0xa0 a bit more. That seems to almost work for powerline except there is this box (probably a font issue for me) that does not show up in the status line. It appears as a black space instead of a red box. By box, I mean that I can see a brighter red foreground outline of a box with a slightly darker red background. With the *mkSamplePass: 10 resource setting, it works as it did in xterm-327. I recall you were going to work on a better change than what you put in comment #25, so I wanted to let you know that small difference.
yes - I am studying uniset to see how to make a complete table. Once I have that, I'll have to check for quirks with 0xa0 that might arise from other locale checks.
I've got a workable update to wcwidth now, though the distribution of differences is changed. In the first 64k, there are about 7,000 differences, but in the first 1000 characters, only 6 differences, as seen in my debugging trace: initialized unicode_font to 0 .. width(U+00A0) = -1, expected 1 .. width(U+00AD) = 1, expected 0 .. width(U+0374) = -1, expected 1 .. width(U+037E) = -1, expected 1 .. width(U+0385) = -1, expected 1 .. width(U+0387) = -1, expected 1 systemWcwidthOk: 6/1024 mismatches, allowed 10
Created attachment 183559 [details] cumulative patch, with updated wcwidth This doesn't change the threshold for choosing the built-in wcwidth vs system. I'll run some tests to see how that should be changed.
There's a lot of data to sort through. I made a quick test-program to show differences between the system- and my updated wcwidth. FreeBSD 11.1 is noticeably different from other systems. Here's some output from my test program (will attach that, though it relies upon a couple of scripts to get the compiler and system name, "path" and "partition" respectively): FreeBSD 11.1 (just the first 64k): 100 61 37.9% a1 1 5f 0 40 1 c0 0 200 161 68.9% 1a1 1 5f 0 40 1 1c0 0 400 358 83.6% 3a1 1 5f 0 49 71 347 0 800 721 89.1% 7a1 1 5f 0 80 118 669 0 1000 cd8 80.3% fa1 1 5f 0 2c9 232 b06 0 2000 1b56 85.4% 1fa1 1 5f 0 44b 3d7 177f 60 4000 3a0e 90.7% 3fa1 1 5f 0 593 43b 24a1 1192 8000 7a04 95.3% 7fa1 1 5f 0 59d 43b 24e1 5148 10000 f024 93.8% ffa1 1 5f 0 f7d 4c4 479b a425 Solaris10: 100 0 0.0% 40 1 c0 0 40 1 c0 0 200 e 2.7% 40 1 1b2 e 40 1 1c0 0 400 41 6.3% 76 6c 30b 14 49 71 347 0 800 107 12.8% 167 ec 59a 14 80 118 669 0 1000 287 15.8% 4d2 1fb 920 14 2c9 232 b06 0 2000 775 23.3% a7b 224 125e 104 44b 3d7 177f 60 4000 e09 21.9% fd6 24e 1967 1476 593 43b 24a1 1192 8000 e09 11.0% fe0 24e 19a7 542c 59d 43b 24e1 5148 10000 2e65 18.1% 211c 266 1d3c bf43 f7d 4c4 479b a425 OSX 100 0 0.0% 40 1 c0 0 40 1 c0 0 200 0 0.0% 40 1 1c0 0 40 1 1c0 0 400 45 6.7% 8e 70 303 0 49 71 347 0 800 18d 19.4% 198 76 5f3 0 80 118 669 0 1000 370 21.5% 51a 87 a60 0 2c9 232 b06 0 2000 95d 29.3% bc5 8b 1356 5b 44b 3d7 177f 60 4000 dac 21.4% 1153 a9 1cb9 114c 593 43b 24a1 1192 8000 dec 10.9% 119d a9 1cb9 5102 59d 43b 24e1 5148 10000 1d39 11.4% 1ab9 ad 415d a33e f7d 4c4 479b a425 Debian/testing 100 1 0.4% 40 2 bf 0 40 1 c0 0 200 1 0.2% 40 2 1bf 0 40 1 1c0 0 400 1 0.1% 49 72 346 0 49 71 347 0 800 1 0.0% 80 119 668 0 80 118 669 0 1000 3 0.1% 2c9 231 b07 0 2c9 232 b06 0 2000 a3 2.0% 44b 336 1820 60 44b 3d7 177f 60 4000 ed 1.4% 593 394 250e 11cc 593 43b 24a1 1192 8000 ed 0.7% 59d 394 254e 5182 59d 43b 24e1 5148 10000 ed 0.4% f7d 41d 4808 a45f f7d 4c4 479b a425 The last appears to have Unicode 9, and basically matches in the first 64K (BMP) except for some places where Markus Kuhn intended differences (a block of characters for instance which others treat as printable and he marked as combining). I might change some of those later, but my updates look okay for now.
Created attachment 183574 [details] simple test-driver for wcwidth.c This executes the system- and Kuhn's wcwidth in xterm's source-directory, showing the amount of difference over successively larger ranges. The two groups of 4-columns after the percentage are the totals for the return-values (limited to -1..2). Basically -1's are returned for controls and missing characters, and 2's may/may not be returned for ambiguous-width CJK's. When I first made the threshold checks, locale-support was much poorer than now.
hmm - I see that I ran the FreeBSD case with POSIX locale - will provide an amended report for that.
Here's the corrected FreeBSD 11.1 report: 100 1 0.4% 41 1 bf 0 40 1 c0 0 200 1 0.2% 41 1 1bf 0 40 1 1c0 0 400 5 0.5% 4e 71 342 0 49 71 347 0 800 18 1.2% 8a 10f 668 0 80 118 669 0 1000 7a 3.0% 2d9 1d0 b58 0 2c9 232 b06 0 2000 129 3.6% 45a 2cc 187b 60 44b 3d7 177f 60 4000 177 2.3% 5bb 300 25ae 1198 593 43b 24a1 1192 8000 1b7 1.3% 5c5 300 25ae 518e 59d 43b 24e1 5148 10000 1b66 10.7% 28c7 31d 2fdb a442 f7d 4c4 479b a425 Essentially it notes the problem with 0xa0, and is mostly okay until the second block of 32k. Given all of that, I'm inclined to make the sample-size 64k, and use a 1% threshold.
A commit references this bug: Author: ehaupt Date: Wed Jun 21 05:57:43 UTC 2017 New revision: 444014 URL: https://svnweb.freebsd.org/changeset/ports/444014 Log: Update to 330 http://invisible-island.net/xterm/xterm.log.html#xterm_330 PR: 219800 (based on) Submitted by: jakub_lach@mailplus.pl Changes: head/x11/xterm/Makefile head/x11/xterm/distinfo head/x11/xterm/files/
I think this can be closed as all the reported issues seem to have been solved. Objections?
Since xterm-330, I have seen no further regressions. "Works for me." :) Thomas, thank you very much for fixing the issues as well as the detailed explanations!
MARKED AS SPAM