Re: UTF-8 support

[Peter Stephenson <pws@xxxxxxx>]

Oliver Kiddle wrote:
> In my opinion it would be sensible to support multibyte encodings in
> general and not just UTF-8. Doing this isn't much effort beyond handling
> UTF-8 if we assume basic ASCII compatibility and don't worry about
> stateful encodings.

I came to the conclusion that was going to be very time consuming --- it
means unmetafying potentially a long string (we don't know where the
characters end) and calling a function every time we want to compare multibyte
characters.  Doing it only for UTF-8 can be optimised to work with
extensions to the current tests; it's simple to test for the length of a
UTF-8 character (although some error checking is also necessary).

Given that the whole point of Unicode is to replace all other schemes,
I'm not so keen about supporting other schemes if it's that much less
efficient.

-- 
Peter Stephenson <pws@xxxxxxx>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************