Re: PATCH: parse from even deeper in hell

[Mikael Magnusson <mikachu@xxxxxxxxx>]

On Fri, Feb 20, 2015 at 4:16 AM, Mikael Magnusson <mikachu@xxxxxxxxx> wrote:
> On Thu, Feb 19, 2015 at 11:03 PM, Peter Stephenson
> <p.w.stephenson@xxxxxxxxxxxx> wrote:
>> On Thu, 19 Feb 2015 22:47:12 +0100
>> Mikael Magnusson <mikachu@xxxxxxxxx> wrote:
>>> I get a crapton of "bad(2) wordsplit reading history:" with this
>>> patch. It seems like all the failed lines have metafied characters in
>>> them, if that's a hint. Most don't contain any syntax characters at
>>> all, for example:
>>>  hist.c:3499: bad(2) wordsplit reading history: mp3info 好きになり\M-c\M-^Aい.mp3
>>> at: 好きになり\M-c\M-^Aい.mp3s
>>> word: 好きになり\M-c\M-^Aい.mp3
>>
>> Unless I'm missing something, I don't think you've said what the real
>> characters you're expecting are.  The broken ones aren't much use for
>> testing.
>>
>>> The (2) means it's the second of the two bad=1; assignments
>>> triggering.
>>
>> At line 3490?
>
> Yes.
>
>>> I'm also not sure why the utf8 is slightly mishandled in the output
>>> there. It has at least been unmetafied, the raw string in the history
>>> file is more or less:
>>> mp3info 好ぃ�になゃ�たぃ�.mp3
>>
>> So those aren't actually valid characters?  Does that mean metafied
>> characters are getting into the history?  I've made it necessary for two
>> more bytes to be metafied, so if the shell was expecting them to be
>> metafied in the history file they won't be.  The bytes are 0x9e and
>> 0x9f.  I guess we could special case those, but do we really output
>> metafied characters to the history file?
>
> The actual line in the history is
> mp3info 好きになりたい.mp3
> but in the history _file_, it's stored metafied, which is hard to
> paste into an email. I'm not sure why pasting the original string
> didn't occur to me. AFAIK, history files have always been metafied.
> I'm not sure why the た is mangled in the error message is what I tried
> to say originally. The final byte is 9f which I suppose is an esc with
> the 8th bit set. Maybe something is trying to double unmetafy? Running
> it through unmetafy() twice doesn't cause any problems though...

Just looked at the debug code and found out about ZSH_DEBUG_LOG, turns
out there's also a 0x8A just before the \M-c\M-^

-- 
Mikael Magnusson