I tried using the function length() with UTF-8 strings. And this function produces an incorrect result. The function works with strings not as characters, but as bytes. And the number of characters per string is multiplied by two. Steps to reproduce (for LANG=ru_RU.UTF-8): echo 'Привет' | awk '{print length($1)}' If I use the function length() with lang/gawk, then UTF-8 string length is calculated correctly. Are you planning to update awk in the base system to support UTF-8 strings in the near future?
I'm not sure it makes sense to compute length() on UTF-8 strings as unicode codepoints. POSIX awk is somewhat clear that you're correct: > LC_CTYPE > Determine the locale for the interpretation of sequences of bytes of text > data as characters (for example, single-byte as opposed to multi-byte > characters in arguments and input files), the behavior of character classes > within regular expressions, the identification of characters as letters, and > the mapping of uppercase and lowercase characters for the toupper and > tolower functions. However, the resulting behavior around indexing is nutty: this implies that index(), match(), etc, are measured in *characters*. To do this efficiently one probably has to convert non-ASCII strings to wchar_t and operate on those. As you could imagine, that would immensely slow down awk as a fast stream processing utility. POSIX is more explicit about toupper() and tolower(), where taking locale into consideration is easier. I guess I'm not clear on what value a length() function is that operates on codepoints rather than bytes.
I don't know if this issue is related to that bug report, but the following command prints 'bin': % echo "bin" | LANG=en_US awk '$1 ~ /^[\t -~]/ {print $0}' while this one prints nothing: echo "bin" | LANG=en_US.UTF-8 awk '$1 ~ /^[\t -~]/ {print $0}' The range from ' ' to '~' includes alphabetical characters when the locale is not utf-8 but does not when the locale is utf-8. We can notice that '/^[\t -~]/' matches "bin" with C.UTF-8.
https://reviews.freebsd.org/D31114 discusses the matter of locales.
(In reply to Warner Losh from comment #3) Note that this DR was committed Jul 30 2021.