As discussed in https://lists.freebsd.org/archives/freebsd-stable/2021-August/000193.html > > # uname -a > > FreeBSD 13STABLE 13.0-STABLE FreeBSD 13.0-STABLE #49 stable/13-n246779-64085efb677-dirty: Mon Aug 16 08:42:53 CEST 2021 root@XXX amd64 > > # export LANG=en_US.ISO8859-1 > > # (echo bla; echo Bla) | grep '[A-Z]' > > bla > > Bla > > This one is unexpected, the upper case should be a range of its own > and should not include any lower case letters. > > # export LANG=en_US.UTF-8 > > # (echo bla; echo Bla) | grep '[A-Z]' > > Bla > > Here I had expected the result you got with en_US.ISO8859-1 ... > > For comparison, a Linux RHEL box delivers the expected results: > > > > # uname -a > > Linux rhel.local 3.10.0-1062.9.1.el7.x86_64 #1 SMP Mon Dec 2 08:31:54 EST 2019 x86_64 x86_64 x86_64 GNU/Linux > > # export LANG=en_US.ISO8859-1 > > # (echo bla; echo Bla) | grep '[A-Z]' > > Bla > > # export LANG=en_US.UTF-8 > > # (echo bla; echo Bla) | grep '[A-Z]' > > Bla > > Seems that this version uses a POSIX style collating sequence for UTF-8. > Definitely a bug in the definition of the collating sequences. > > And I have just verified that de_DE.ISO8859-1 wrongly considers "ö" > to be within [a-z], while de_DE.UTF-8 does not (but should). > > Seems that the correct collating sequences for ISO8859-1 and UTF-8 are > each assigned to the other one. Can some knowledgeable person please validate?
I don't think this is a bug. POSIX doesn't define this for anything other than the C locale.
While it is true that POSIX does not define it for ISO8859-1 or UTF-8, it always used to work for ISO8859-1 (as a simple extension of ASCII). The really surprising result is that ISO5589-1 obviously includes lower case letters in the range [A-Z] (it never did before!), while UTF-8 excludes them (and the common practice in Unicode is to have a collating sequence of "aAbBcC..." for latin based character sets. There is obviously code that applies some collating sequence rules, but opposite to what I'd expect. The Linux example shows that they decided to use the traditional collating sequence any locale including ISO8859-1 and UTF-8 (and as said, POSIX does not care at all). We could make ISO8859-1 use the traditional collating sequence and UTF-8 the Unicode convention of lower case just before upper case letter, or we could always apply the traditional collating sequence, but we should definitely use traditional for UTF-8 and Unicode style for ISO8859-1.
(In reply to Warner Losh from comment #1) True, but kind of surprising behaviour - in particular as Linux delivers the naïvely expected results different from ours.
Tagging in Baptiste (locales) and Yuri (regex +locales)
More fun with ISO8859 variants: # (echo Bla; echo bla) | LANG=en_US.ISO8859-1 grep '[A-Z]' Bla bla # (echo Bla; echo bla) | LANG=en_US.ISO8859-15 grep '[A-Z]' Bla bla # (echo Bla; echo bla) | LANG=en_US.ISO8859-11 grep '[A-Z]' Bla # Even better: it seems only 1 and 15 are, err, "special": # seq 1 16 | while read f; do echo $f; (echo Bla; echo bla) | LANG=en_US.ISO8859-$f grep '[A-Z]'; done 1 Bla bla 2 Bla 3 Bla 4 Bla 5 Bla 6 Bla 7 Bla 8 Bla 9 Bla 10 Bla 11 Bla 12 Bla 13 Bla 14 Bla 15 Bla bla 16 Bla #
Ah, sorry, my fault. These locales don't exist so we are silently reverting to C.
All I can say here is quote the following from https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html: --- In the POSIX locale, a range expression represents the set of collating elements that fall between two elements in the collation sequence, inclusive. In other locales, a range expression has unspecified behavior: strictly conforming applications shall not rely on whether the range expression is valid, or on the set of collating elements matched. --- So my take on this is "don't use range expressions".