Tutorial on zsh substitutions (Re: OPTARG not being set?)

["Bart Schaefer" <schaefer@xxxxxxxxxxxxxxxx>]

On Feb 3,  1:35pm, Sweth Chandramouli wrote:
> Subject: Re: OPTARG not being set?
> On Wed, Feb 03, 1999 at 12:03:02AM -0800, Bart Schaefer wrote:
> > Nope.  Nested ${...} constructs don't act like that in zsh; the doc says
> > 
> > ${NAME}
> >     The value, if any, of the parameter NAME is substituted. [...]
> > 
> >     If a `${'...`}' type parameter expression or a `$('...`)' type command
> >     substitution is used in place of NAME above, it is expanded first and
> >     the result is used as if it were the value of NAME.
> >     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 	
> 	see below for my comments on the syntactic definition of a "name".

It's true that "the value of NAME" is a bad turn of phrase there.  A better
way to describe it would be to say

    ${THING}
	When THING is a parameter name, the value of that parameter
	replaces THING.  When THING is another substitution expression,
	introduced by a leading '$', the value of that expression
	replaces THING.  All other processing specified by parameter
	substitution flags and modifiers is applied to the replacment,
	and the result is substituted for the entire expression.

The reason it's not like that is because (a) the manual was incrementally
built up from documentation that predates nested substitutions, and (b)
there was a desire not to "forward reference" flags, modifiers, and nested
expression substitution until the simpler parameter-name-only cases were
fully described.

> > All the rest of the processing
> > specified by flags or modifiers in the outer braces is applied to the
> > resulting value string or array.  That's why ${(t)${bob}} doesn't work,
> > because a value doesn't have a type; only names have types (which are
> > instructions for interpreting the value).
> 	is the difference between a name and a value semantic instead of
> syntactic, then?

Not exactly.  The difference is essentially that a name can never begin
with a '$' character, so anything not beginning with '$' is a name that
is used to look up a value, and anything that does begin with '$' is an
expression that must be recursively interpreted to produce a value.

Some of the flags then mean, "iff you have a name, change the way you
look it up so that instead of its value you get the (type,length,etc.)
of its value."

> i would assume that the shell decides what is a value
> and what is a name by the context in which it is placed; based on the
> definitions given in the manpage, it seems that ${...} constructs 
> definitionally interpret the first token inside the braces to be a name.

That's because the man page is referring only to parameter substitution at
that point, and parameters have names.  It's explaining the simple case
very simply, at the expense of making the complex case appear even more
complex than it is.

> after reading over the paragraph you quote above a few times, i can see
> how it can be interpreted to mean what you say it means, which is that
> nested uses of ${...} override the syntactic identification of the
> first token of all but the innermost ${...} as a name

The problem with thinking of it that way is that the innermost thing may
not be a name at all.  It could be that a $(command) is innermost.  Such
an inner expression doesn't have a "name" to identify.

> 	2) after i sent in my original question, i did a search on the
> zsh-workers archive, and noticed that you had done some work on this
> same topic, at least in standardizing use of terms like "expansion"
> and "substitution".  i think that it would help a lot to do the same
> thing for terms like "name", "word", and value"

Yes, and "parameter" vs. "variable" too, and all sorts of other things.
The problem with doing so is that it makes various sections of the manual
incomprehensible without having previously understood the "shell grammar"
or some other definition section.

> then i think that this could just be explained in the definitions of
> the various ${...} constructs themselves, rather than in a footnote
> at the end.

As I noted, that organization is intentional, to avoid confusing those
who only care about the simplest case.

> 	3) i'm not a huge fan of "magic" values and exceptions (which is
> one of the reasons that i've never really gotten that into perl); the
> third solution, then, would be to make the behaviour of ${...} standard

It is standard; what you're proposing amounts to an order-of-evaluation
change.  That is, presently nested ${...} expressions are evaluated from
the outside in, whereas you want them evaluated from the inside out.

> > What Sven's new (P) flag does is force the value to be interpreted as a
> > new name, the way the syntax misled you expect all along.  (Maybe the
> > (P) flag ought to be ($), as in ${($)${bob}}.  Or more likely not.)
> 	is this added in a dev version?  i can't find reference to it
> in the 3.1.5 docs.

It was a patch just posted yesterday.  I didn't notice that this message
was going to zsh-users, where some people wouldn't recognize the context.

The HTML manual available at www.zsh.org is now for 3.1.5-pws-5, which is
an unofficial patched version that Peter Stevenson has been releasing.
It has most of the details (including many man page clarifications) up
to but not including the (P) flag.

> > First, note that ${bob} substitutes to "joe".  ${(e)bob} is similar (but
> > not identical) to $(echo ${bob}).  
> 	how so?  according to the docs, (e) says to "[p]erform parameter
> expansion, command substitution, and arithmetic expansion on the result."

Yes.  What that means is:
    parameter expansion -- if the result looks like $param or ${...},
			   then substitute the value of that expression
    command substitution -- if the result looks like $(command ...),
			    then evaluate command and substitute its output
    arithmetic expansion -- if the result looks like $[expr], then
			    compute expr and substitute that result

That's "similar (but not identical) to" $(echo) because there really isn't
any other syntax that expresses the same semantics -- the $(echo) is just
as close as I could get to another way to explain it.

> i originally thought that this meant that ${(e)${bob}} would parse to
> ${(e)joe}, and then perform parameter expansion on ${joe}

No.  "The result" is just "joe".  Outside in, not inside out -- the outer
${...} is already "used up" by the time the (e) is applied.  Get it?

> > 	bob='$joe'		# Note ${bob} now substitutes $joe
> > 	${(e)${bob}}		# Like $(echo $joe) --> hello world
> 
> 	to do this dynamically (i.e. to get this effect when you don't
> know ahead of time that joe is name of the parameter to be expanded),
> however, you need to do something like bob="${${param_whose_value_is_joe}}",
> which is the original situation that caused me to start trying to figure
> all of this out.

Right.  This is what the new (P) flag is for.  What you need without (P) is

	eval bob=\$${param_whose_value_is_joe}

Note that I *didn't* write \${${param_whose_value_is_joe}}, to make it clear
that the intent is that ${param_whose_value_is_joe} produce a string that
can be interpreted as a name (and NOT produce another '$'-expression).

> 	what is the difference between grave-accent command substitution,
> $(...) command substitution, and eval's pseudo-command substitution?  it
> looks like grave accents strip meta-char escapes, do param expansion, and 
> then evaluate, while $(...) just does param expansion and then evaluates, 
> and eval does param expansion, strips meta-char escapes, and then evaluates.

The main difference between "eval" and either of $(...) or `...` is that
eval does not capture the output of the command that gets executed.

The main difference between `...` and either of $(...) or eval is that
`...` is a quoting mechansism and gets parsed as one.  This causes the
stripping of the backslash-escapes, and delays interpretation of other
quoting mechanisms until the time of the evaluation.

You can see this most obviosly by using an unbalanced double-quote in
the input somewhere and looking at the secondary prompt:

zsh% eval echo This " is unbalanced
dquote> 

zsh% $(echo This " is unbalanced)
cmdsubst> 

zsh% `echo This " is unbalanced`
zsh: unmatched "
zsh: parse error in command substitution