From: David Flanagan <david@...>
Date: 2007-10-23T04:19:01+09:00
Subject: Re: \u escapes in string literals: proof of concept implementation

Yukihiro Matsumoto wrote:
> Hi,
> 
> In message "Re: \u escapes in string literals: proof of concept implementation"
>     on Sat, 20 Oct 2007 09:50:58 +0900, David Flanagan <david@davidflanagan.com> writes:
> 
> |> If the current encoding is not based on the Unicode character
> |> set, I think \u should:
> |> 
> |> a. raise compile error,
> |> b. be converted from Unicode to the encoding, or
> |> c. make the whole string UTF-8 encoding (and raise compile error
> |>    if other non-ascii characters are there).
> |
> |The new patch attached here does c.  Any string with a \u in it is 
> |forced to utf-8 encoding.  Unless the \u encodes a character in the 
> |ASCII range, in which case it leaves the encoding alone.
> 
> I'd happy to merge the third patch, once I've been convinced by option
> c.  Originally I was thinking option b, but option c is much simpler.

Are you happy with \u escapes in Regexp literals?  Note that the \u is 
interpreted by the lexer rather than being passed on to the regexp engine.

I don't have an argument against option b.  I didn't even consider it 
because I don't know how to that kind of conversion.  Are you still 
planning to add a method for transcoding of strings in 1.9?  If we can't 
to transcoding explicitly, then I'd argue that making \u do transcoding 
doesn't make sense.  (Is "transcoding" the right word to use here?)

Option c was relatively simple. Note, though, that I don't currently 
implement the part in parentheses. That is, there is no check to see if 
a string has other multi-byte characters in it.  So with -Ke or -Ks you 
could start a Kanji string in one of those encodings and then throw in a 
\u escape. You'd probably end up with an string that was not legal utf-8 
nor legal euc or sjis.  The \u escape could raise an error.  But what 
about the reverse case.  Say I start writing a string with a \u escape, 
then then add a Kanji character at the end of the string.  I don't 
believe that the lexer can currently detect that case and complain.  As 
I understand it, we can use \x to mess up any string, and I don't 
believe that the lexer applies any sanity checks to string literals.  So 
if we're not doing that, I'm not sure we need to apply those sanity 
checks to \u.

On a related note, Martin submitted test cases that expected errors to 
be raised for string like "\uD800" because (I think) that codepoint is 
supposed to be part of a surrogate pair and it should be illegal to have 
it by itself in a string.  He's got a point, but I'm not sure that is 
something we want to enforce at the \u level.  If the lexer is going to 
complain about unpaired surrogates, I think it needs to catch them when 
encoded with \x or when they're just in the byte stream of the source 
file as well.

Anyway, that is my "argument" for option c.

> Anyway, I'd like to give you CVS commit permission (if you want) to
> check in by yourself after the final decision is made.  Please send
> your SSH2 public key signed with PGP to <cvs-admin@ruby-lang.org>.

Thank you for the honor.  I've sent my key.

	David

> 							matz.
>