Re: compset -q oddities

[Peter Stephenson <p.stephenson@xxxxxxxxxxx>]

On Mon, 12 Sep 2016 23:28:53 -0700
Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx> wrote:
> This one --
>    % _g() { compset -q } 
>    % compdef _g g
>    % g $'\'<TAB>
>    compcore.c:1657: expecting 'x' at offset 2 of "'x"> 
> 
> -- seems to be a legit problem with counting bytes when looking for an
> unbalanced $'...'.  I don't think fixing that would change the outcome,
> that is $'\' --> \' (see quoting converted, above).  I'm not confident
> of how to fix it; PWS was last here in workers/22026 (git 34381548).

The missing offset seems to come because we have an input string
(omitting double quotes) " \\'x" even before we do the double quote
processing.  The \\ goes aways with the double quote processing leaving
the ', which seems to be fine.  It looks like it ought to modify tl to
give the length we set to zlemetall later on, since that comes from the
modified tmp (great names), but it doesn't; however, that doesn't seem
to be the problem.

When we run the lexer over this and get an error, it then ignores that
initial " ", and I think that's what's giving the bad offset.  tokstr is
now "'x " except with ' tokenised (that's not a problem here); we make
efforts to deal with the added final " " but not the fact the previous
initial space has (in technical language) gorn.

I don't know where that initial space is coming from.

I don't know if this means the lexerr stuff is therefore too brittle
about characters it found on an unterminated string to be useful and we
should simply give up in some more well-defined way.

It might be sensible instead of using tokstr to take the input string
when we encounter an error, but I presume we'd need to know at least the
start of input for the current iteration of the loop containing
ctxtlex() (we do have mechanisms for copying literal input during lexing
for use with the new-improved $(...) parsing but that's largely
orthogonal to what's happening here, where "largely" means "actually I
don't really know what's going on in terms of hierarchy because it's way
too complicated").

Anyway, I suspect it ought to be possible to do better.  I've seen this
case crash intermittently as we're accessing bad memory --- we could
probably at least fix up accesses off the end of the string after the
DPUTS for safety.

However, I'm not sure we want to fiddle with this before a release which
we really ought to be making imminently as this is just one of a series
of incrementally partially fixed problems in this area

(This whole interface is horribly brittle anyway, of course, but it's
unlikely anyone's going to dare to rewrite it any time soon.)

pws