Re: vared/zle silently discards non-utf8 bytes

[Peter Stephenson <pws@xxxxxxx>]

On Wed, 23 Dec 2009 11:44:51 +0100
Mikael Magnusson <mikachu@xxxxxxxxx> wrote:
> Ufortunately it seems vared discards
> anything after an invalid byte. To reproduce, just do
> 
> % a=hi$'\374'nothing
> % vared a

This is currently the designed behaviour if multibyte support is compiled
in.  In this case the editing line is a set of wide characters.  If it
can't convert the input into wide characters it's stuck.

Internally, there are two options

(i) I could simply make it ignore invalid characters, which gets you some
of the line, but is probably even more dangerous

(ii) you could have a go at rewriting the way characters are stored for
editing to use a marker that a character isn't a valid wide character but
is being stored to represent an octet.  This is a big job to get consistent
all the way through (display including width, character tests, conversion
back and forth).

Note that a simpler wrapper

varedquote() {
  # ignoring vared options for now....
  local var=${argv[-1]}
  local val=${(q)${(P)var}}
  # hmmm... if the user stripped some quoting the following is
  # a bit fraught...
  vared val && eval ${var}=${val}
}

should work because the (q) flag is already smart about unprintable
characters (except it does rely on the user not removing backslashes in the
variable).  This could be made a vared option.  It's a little bit hairy
making it default behaviour because it changes the meaning of special
characters in the string you're editing---it's no longer "raw" in other
ways than just $'...' quoting.

-- 
Peter Stephenson <pws@xxxxxxx>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom