Re: PATCH: parse from even deeper in hell

[Mikael Magnusson <mikachu@xxxxxxxxx>]

On Thu, Feb 19, 2015 at 11:13 AM, Peter Stephenson
<p.stephenson@xxxxxxxxxxx> wrote:
>   And Tomlinson looked down and down, and saw beneath his feet
>   The frontlet of a tortured star milk-white in Hell-Mouth heat.
>
> % print $((echo one); (echo two))
> zsh: bad math expression: operator expected at `one); (ech...'
>
> At the point this goes wrong, we've actually already established this is
> a command substitution, not a math expression.  However, we're now in
> the substitution code and it doesn't have any marker that that's
> happened.  Instead, it just looks to see if there are two parentheses at
> the end, which there are.
>
> Note that it's not a fix to count active parentheses in the middle at
> that point: those aren't tokenized because we're parsing this as a
> string for later expansion.  So the ones at the end are the first that
> skipparens() picks up.  In any case re-counting when we've already
> established what's supposed to happen is a pretty kludgy fix.
>
> The fix here is to use different tokens for the first and last
> parenthesis for math.  We then just look for the matching close marker
> when we find the open marker.  We can't have nested math expansions so I
> think this ought to be robust.
>
> I've incremented the version as this changes the way strings are
> tokenized.
>
> The tests might more logically be with command substitution rather than
> arithmetic, but I've left them in order to keep the tests for is / isn't
> arithmetic in one place for easy comparison.

I get a crapton of "bad(2) wordsplit reading history:" with this
patch. It seems like all the failed lines have metafied characters in
them, if that's a hint. Most don't contain any syntax characters at
all, for example:
 hist.c:3499: bad(2) wordsplit reading history: mp3info 好きになり\M-c\M-^Aい.mp3
at: 好きになり\M-c\M-^Aい.mp3
word: 好きになり\M-c\M-^Aい.mp3
The (2) means it's the second of the two bad=1; assignments
triggering. (Left over from last time I had a problem in that area).
I'm also not sure why the utf8 is slightly mishandled in the output
there. It has at least been unmetafied, the raw string in the history
file is more or less:
mp3info 好ぃ�になゃ�たぃ�.mp3

-- 
Mikael Magnusson