Re: PATCH: parse from even deeper in hell

[Mikael Magnusson <mikachu@xxxxxxxxx>]

On Fri, Feb 20, 2015 at 11:00 AM, Peter Stephenson
<p.stephenson@xxxxxxxxxxx> wrote:
> On Fri, 20 Feb 2015 04:43:49 +0100
> Mikael Magnusson <mikachu@xxxxxxxxx> wrote:
>> Oh I see, you renumbered a bunch of stuff in zsh.h, so text would be
>> metafied differently now. But only metafy uses the table, unmetafy
>> doesn't. That explains why the strings are different, but not why the
>> old string causes an error. Unless we are parsing it before
>> unmetafying it, which means any random bytes that weren't special
>> before but are now would be interpreted specially.
>
> I haven't confirmed the behaviour in detail, but I think what's
> happening is the old string read in from history now looks like
> an already metafied byte and when we scan the line gets unmetafied into
> an invalid 8-bit character which is where things are falling over.
>
>> Can we
>> unmetafy+metafy the string before lexing? I guess that might be slower
>> though. (Sorry for the 500 mails).
>
> I believe any character read in from the history file that appears as
> "needs metafication" is illegal, because we don't output tokenised
> text.  So we simply metafy it at that point.  We can pre-scan for this
> so we don't treat lines we don't need to it.
>
> The question is where to put this in on history read.  I think it's
> going to affect non-lexical history, too, but the error on reading won't
> be flagged up.

I don't think so, unmetafy() doesn't care about the table. And as I
checked earlier, both the old and new version of the string in my
history file is unmetafied to the correct UTF-8 string. The 'only'
problem is that the lexer is looking at some bytes before it's
unmetafied and some stuff that should have been metafied to avoid
being parsed as tokens, isn't, because they weren't special in the old
version. That's why I think running unmetafy before lexing is
needed... And if the lexer wants metafied text then we'd just have to
metafy it again right away.

My theory doesn't fully explain the weird output from the error
message, but maybe that's because of dingbats in the wobblifier.

-- 
Mikael Magnusson