Re: Odd behavior with various (q) array modifiers and non-printable characters (backspace, newline)

[Zach Riggle <zachriggle@xxxxxxxxx>]

TLDR 1: I learned/realized that zsh builtin echo automatically
interprets escape codes, unlike bash echo and /bin/echo.
TLDR 2: I also learned that (q+) is my new favorite, and my code
should use "command echo" to avoid the above behavior.
TLDR 3: Thank you all for taking the time to write back.

> You need to stop testing things with "echo".  The "echo" builtin interprets some backslash escapes itself...

Ah, the real culprit.

For some reason Bart's comments didn't land in my inbox, but I am a
mixture of disappointed in myself (for forgetting the builtin
evaluates escape codes) and surprised (bash's builtin echo requires
"-e" to do this, /bin/echo does not support this at all on BSD).

Indeed, avoiding the builtin echo's interpretation of escape codes by
default (not sure how I feel about that, but it is what it is) shows
that the expansion modifiers work as they say on the tin.

    $ command echo ${(qq)mysed}
    'gsed' 's/\ba\b/x/'

    $ command echo ${(q+)mysed}
    gsed 's/\ba\b/x/'

    $ command echo ${(q-)mysed}
    gsed 's/\ba\b/x/'

> Presumably you thought "..." works like $'...'.

Nope, that one I knew pretty well, but I was indeed spoiled by the
shell interpretation of the string "\n".

Leaning on Python a bit to make it more obvious, but yeah:

    $ python -c 'import sys; print(sys.argv)' a b "c\n" d
    ['-c', 'a', 'b', 'c\\n', 'd']

    $ python -c 'import sys; print(sys.argv)' a b $'c\n' d
    ['-c', 'a', 'b', 'c\n', 'd']

Interestingly, the handling of ACTUAL newlines ($'\n') by (q-) is less
than ideal, (q) works but is clunky, and (q+) seems to work the best.

    $ newline=( aa $'b\nb' cc )

    $ command echo ${(q)newline}
    aa b$'\n'b cc

    $ command echo ${(q-)newline}
    aa 'b
    b' cc

    $ command echo ${(q+)newline}
    aa $'b\nb' cc

Thanks for all of the help fielding my silly questions!

Zach Riggle

On Wed, Aug 11, 2021 at 4:45 PM Lawrence Velázquez <larryv@xxxxxxx> wrote:
>
> On Wed, Aug 11, 2021, at 5:16 PM, Bart Schaefer wrote:
> > You need to stop testing things with "echo".  The "echo" builtin
> > interprets some backslash escapes itself, which will confuse you about
> > what the quoting options have done.
> >
> > Repeat all your tests instead with
> >   printf "%s\n" ${(q)...}
> > and so on, and come back if you still have questions.
>
> Additionally, \b and \n are not interpreted in double quotes, so
> your initial data does not actually contain BS or NL characters.
> Presumably you thought "..." works like $'...'.
>
> % mysed_orig=( gsed "s/\ba\b/x/" )
> % typeset -p mysed_orig
> typeset -a mysed_orig=( gsed 's/\ba\b/x/' )
>
> % mysed_fixed=(gsed $'s/\ba\b/x/')
> % typeset -p mysed_fixed
> typeset -a mysed_fixed=( gsed $'s/\C-Ha\C-H/x/' )
>
> % newline_orig=( aa "b\nb" cc )
> % typeset -p newline_orig
> typeset -a newline_orig=( aa 'b\nb' cc )
>
> % newline_fixed=( aa $'b\nb' cc )
> % typeset -p newline_fixed
> typeset -a newline_fixed=( aa $'b\nb' cc )
>
> As per the QUOTING section of zshmisc(1):
>
>         Inside double quotes (""), parameter and command substitution
>         occur, and `\' quotes the characters `\', ``', `"', `$',
>         and the first character of $histchars (default `!').
>
> --
> vq