Re: ${(z)} split of unmatched, doubled ((

[Daniel Shahaf <d.s@xxxxxxxxxxxxxxxxxx>]

[ replying out of order ]

Bart Schaefer wrote on Sun, Sep 27, 2015 at 09:00:48 -0700:
> On Sep 27,  1:23am, Daniel Shahaf wrote:
> }
> } % print -rl - ${(z):-'(( e'}
> }  e
> } %
> } 
> } Shouldn't it output the parentheses as well?
> 
> There's also an issue of how to treat "e" in this example.  If the
> double parens are taken as math context, then "e" is a single double-
> quoted token, otherwise it has to be decomposed into shell words.  As
> you can see, 4.2.0 parses it BOTH ways (eek).
> 

The "e x y" should be treated consistently with the "((": the latter
should be reported as two tokens iff the "e x y" is split into words.

Since the "two subshells" case can be disambiguated by adding a space,
but arithmetic evluations cannot be disambiguated, I assume ambiguous
cases should be resolved in favour of the latter.

> This has always been broken.  The '((' parse doesn't have a tokstr
> value (cf. comments about parsing "for (( ... ))" in bufferwords()
> [hist.c]).  Prior to recent fixes to backtrack this properly, the
> error was even worse:
> 
> torch% print $ZSH_VERSION
> 4.2.0
> torch% print -rl - ${(z):-'(( e x y'}
>  e x y 
> (
> e
> x
> y
> torch% 
> 

I see the problem: ${(z)} is bufferwords(), which calls ctxtlex(), which
ultimately calls cmd_or_math(), which classifies the unbalanced opening
parentheses as a syntax error, because they have no matching ')' before
the end of the input.  Consequently, cmd_or_math() returns CMD_OR_MATH_ERR
on line 512 (in the 'if (lexstop)' block), which causes ctxtlex() to
return LEXERR.

I guess cmd_or_math() is actually doing the right thing, insofar as the
"interpret and execute code" use-case of the lexer is concerned.  But
the bufferwords() caller shouldn't simply skip over the "((" characters
in the input buffer.  (More on this below.)

> I don't have an answer for where the right place to "output" the parens
> would be; the backtracking makes this ugly.

Looking at bufferwords(), the "e" is added to the output by the
addlinknode() in this block:

  3350	    if (buf && tok == LEXERR && tokstr && *tokstr) {
  3351		int plen;
  3352		untokenize((p = dupstring(tokstr)));
  3353		plen = strlen(p);
  3354		/*
  3355		 * Strip the space we added for lexing but which won't have
  3356		 * been swallowed by the lexer because we aborted early.
  3357		 * The test is paranoia.
  3358		 */
  3359		if (plen && p[plen-1] == ' ' && (plen == 1 || p[plen-2] != Meta))
  3360		    p[plen - 1] = '\0';
  3361		addlinknode(list, p);
  3362		num++;
  3363	    }

If this addlinknode() were skipped, output would stop immediately before
the '((':

    [with line 3361 commented out]
    % Src/zsh -fc 'print -rl - ${(z):-":; (( e"}' 
    :
    ;

However, removing the addlinknode() call makes a test fail:

    ./D04parameter.ztst: starting.
    *** /tmp/zsh.ztst.out.13138	2015-09-27 20:16:41.812154669 +0000
    --- /tmp/zsh.ztst.tout.13138	2015-09-27 20:16:41.816154673 +0000
    ***************
    *** 3,11 ****
      line
      with
      #
    - someone's comment
    - another line # (1 more
    - another one
      *** Kept ***
    --- 3,8 ----
    Test ./D04parameter.ztst failed: output differs from expected as shown above for:
      line=$'A line with # someone\'s comment\nanother line # (1 more\nanother one'
      print "*** Normal ***"
      print -l ${(z)line}
      print "*** Kept ***"
    Was testing: Comments with (z)

So I'm not sure what's right here.  Perhaps the addlinknode() should be
skipped for the "(( e" case?  (i.e., parse as far as possible, and stop
before the ambiguity)  And why is tokstr " e " when tok is LEXERR?  It's
not the " e" that caused the error...

Cheers,

Daniel