Re: UTF-8 locales on BSDs do not support collation correctly

["Jun T." <takimoto-j@xxxxxxxxxxxxxxxxx>]

On 2017/01/30, at 12:59, Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx> wrote:

> Is this
> just an issue with the test or is there a real problem here?

I believe there is no problem in zsh.
The problem is that macOS does not support UTF-8 collation at all.
For example, on macOS,
  /usr/share/locale/pl_PL.UTF-8/LC_COLLATE
is a symlink to
  /usr/share/locale/la_LN.US-ASCII/LC_COLLATE
and the strcoll(3) always uses ASCII collation.

The commit 72e5fe7 modifies glob.c so that unmetafied file names are
(correctly) used for glob sorting. In order to test this on both Linux
and macOS, we need two characters (or strings) c1 and c2 which satisfy

  c1 < c2    and    metafy(c1) > metafy(c2)

in both UTF-8 and ASCII collations. It seems the following two
characters can be used:

       Unicode      UTF-8      metafied
---------------------------------------
c1  Ą   U+0104      c4 84      c4 83 a4
c2  Ġ   U+0120      c4 a0      c4 83 80

So how about the following patch? With this patch, the test fails
without the commit 72e5fe7 but succeeds with it, on both Linux and macOS.


diff --git a/Test/D07multibyte.ztst b/Test/D07multibyte.ztst
index 0ff65c7..e203153 100644
--- a/Test/D07multibyte.ztst
+++ b/Test/D07multibyte.ztst
@@ -551,22 +551,20 @@
   : $functions)
 0:Multibyte handling of functions parameter
 
-  if [[ -n ${$(locale -a 2>/dev/null)[(R)pl_PL.(utf8|UTF-8)]} ]]; then
-  (
-    export LC_ALL=pl_PL.UTF-8
-    local -a names=(a b c d e f $'\u0105' $'\u0107' $'\u0119')
-    print -o $names
-    mkdir -p plchars
-    cd plchars
-    touch $names
-    print ?
-  )
-  else
-    ZTST_skip="No Polish UTF-8 locale found, skipping sort test"
-  fi
-0:Sorting of metafied Polish characters
->a ą b c ć d e ę f
->a ą b c ć d e ę f
+# c1=U+0104 (Ą) and c2=U+0120 (Ġ) are chosen so that
+#   u1 = utf8(c1) = c4 84  <  u2 = utf8(c2) = c4 a0
+#   metafy(u1) = c4 83 a4  >  metafy(u2) = c4 83 80
+# in both UTF-8 and ASCII collations (the latter is used in macOS
+# and some versions of BSDs).
+  local -a names=( $'\u0104' $'\u0120' )
+  print -o $names
+  mkdir -p colltest
+  cd colltest
+  touch $names
+  print ?
+0:Sorting of metafied characters
+>Ą Ġ
+>Ą Ġ
 
   printf '%q%q\n' 你你
 0:printf %q and quotestring and general metafy / token madness