Re: Bug in alias expansion

[Peter Stephenson <p.stephenson@xxxxxxxxxxx>]

On Tue, 17 Nov 2015 20:55:08 -0800
Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx> wrote:
> (Of course 37128 and 37122 aren't mutually exclusive, we can apply
> both; there's probably some other case where subscript parsing may
> go wrong that's fixed by 37128.)

37128 is fixing a particular problem to do with assumptions about
parsing that are no longer valid; for this, it is entirely irrelevant
where the alias actually gets expanded.  I haven't followed whether
there is a visible bug as a consequence of the latter.

> } It's not very efficient, and it still has the problem that if something
> } funny happens to the length --- and I don't think there's any guarantee
> } of length preservation built into the lexer even though it happens to
> } work in the cases we've looked at so far --- then somebody's going to
> } get hurt.
> 
> I'm not following this stuff about the length -- it can't mean that the
> length of the alias expansion is the same as the length of the aliased
> token -- so to what does it refer?

The code assumes that if you pass a string through parse_subscript(), or
more accurately through dquote_parse(), then the sequence of characters
that comes out maps one-to-one onto the sequence of characters that went
in. For example, a ( turns into an Inpar, a " turns into a Dnull, and
so on --- similarly to if you simply tokenised it character by
character but with more context sensitivity.  The change makes the
alias expansion irrelevant to this --- we backtrack over that -- but
the code still relies on this assumption.  Given how much can now
happen inside dquote_parse() (as we see here, if only as a side issue
to the bug being fixed), this doesn't seem ideal.  However, I don't know
of a case where it fails.

pws