Re: more splitting

[Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx>]

On Tue, Apr 14, 2026 at 9:44 PM Ray Andrews <rayandrews@xxxxxxxxxxx> wrote:
>
> On 2026-04-14 20:26, Bart Schaefer wrote:
> > Arguments ($@) are a list of words that the shell has already divided
> > up.  Pipes (and other sorts of file input) are a stream of bytes.
> Yeah, that gets to the heart of my issue.  It's this 'shell has already'
> ... how?

By parsing.

> Where is the information stored?

This is almost literally the same as the distinction between a
(char[]) and a (char[][]) in C.  The information is stored in a data
structure in the shell.  When you write
   var=("a b" c$'\n''d e f'' ''g h')
the shell parses the quoted sections and builds a data structure,
which zsh calls an array.  The quotes themselves are gone, they were
only needed to tell how to build the array.

> When you say 'stream of
> bytes' in my mind that stream must include whatever information is
> needed to determine the split.

No, really, it's just a stream of bytes.  It doesn't have any inherent
semantics at all.  You can add things to the stream that can later be
interpreted as semantics, but then you also have to provide the
interpreter.

When you write
   print -rn $var
you are instructing the shell to dump to stdout, as string, the
contents of the data structure.  You haven't told it to restore the
original tokens from before the parse.  In fact it can't restore the
original: as I said, those quotes are gone.

> Where else could the information reside?

In the case of $var, it's still in that array structure named "var".
But the rules for what happens when you use $var to "output" that
array depend on context.  For a simple usage like
   print -rn $var
the rule is to combine all the elements into a single string with
spaces between them.

If you don't want that structure information to be lost, you have to
tell zsh to re-create it.  It can't promise to restore it exactly as
it was, because that's gone, but if you write
   print -rn ${(q+)var}
the (q+) tells zsh to rebuild something that has the same semantics as
the original quoting.  Notice that Stephane used (q+) in the original
"hex" function you posted.

If you want to see a representation of the whole internal structure
including it's name, instead of just a representation of its value,
you can write
   typeset -p1 var
and then you get a similar reconstruction of the semantics (but still
not the original source).