Re: Unicode allowables in Environment variables

[Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx>]

On Mon, Nov 4, 2024 at 4:11 PM William DeShazer <earl.deshazer@xxxxxxxxx> wrote:
>
> Please educate me if I am wrong about Valgrind, but I believe the reason that Valgrind didn’t catch it is ...

Valgrind looks for pointers running into memory outside the bounds of
a string.  Mark suggested that the reason unmetafy() was having an
effect is because it was failing to stop at the end of the string.  My
only point was that valgrind did not find THAT to be the problem.

Subsequently ...

I put this through gdb and found that getsparam("ZDOTDIR") appears to
be returning a consistent value all the way through the point of
failure, and further that getsparam_u() also appears to return that
same value.  So I went looking for the actual error occurrence in
paramsubst().

What I found is that in the erroring case, when we get to subst.c:169
in prefork(), the value returned from paramsubst() contains

App\360\224\256Bundles/Configurations/Zsh/•Z

However, if I then "fix" ZDOTDIR by assigning from $(printenv), upon
getting back to that line in prefork() the value instead contains

App\360\203\277\203\264\256Bundles/Configurations/Zsh/ \202Z

I haven't found where the internal representation is getting changed,
but this is looking a lot like the same oddity discussed in "ZLE
character width with emoji presentation variation selectors in
Unicode"
  https://www.zsh.org/mla/workers/2024/msg00481.html (and followups)
wherein we seem to have concluded that MacOS has a broken wcwidth()
implementation but never arrived at a resolution for it.

I seem to recall that the MacOS file system automatically converts
between the two formats in order to standardize on the one used for
file names, so you'll never see this in context of finding files in
the ZDOTDIR directory.