Re: \u escapes in string literals: proof of concept implementation

Martin Duerst wrote:
> It should also complain about paired surrogates. There are no surrogate
> pairs in UTF-8. \u designates a codepoint/character, but there is no
> codepoint or character \uD800, so we should complain.

I believe that Java's version of UTF-8 encodes surrogate pairs as two 
3-byte sequences.  And Java's \u escape only supports 16 bits.  So 
anyone porting Java software to Ruby will have to deal with strings 
containing surrogate pairs.  Even if it is not technically correct, I 
suspect we ought to allow this usage, following Java's lead.

> That things are simpler is quite clear. But option a wouldn't be
> difficult to implement, either, I guess. My suggestion is to stay
> with option a until we have a better idea of which of b and c is
> really needed/implementable/... 

The appeal, to me, of my current version is that it is independent of 
the primary encoding.  \u escapes work no matter what -K option you 
specify, and they always translate to a specific byte sequence. On the 
other hand, I really don't know how to handle strings that mix 
sjis-encoded Kanji characters with \u escapes.  What should the encoding 
of the resulting string be?

So chosing option a might be the best bet: \u escapes cause an error 
with -Ks or -Ke.  They are only allowed when the primary encoding is 
ascii or utf-8.  I think that would mean that no transcoding would be 
necessary.  I also think that my current patch doesn't need much 
modification: just the addition of errors when the encoding does not 
allow \u.

I'll make this change if others agree that it is the right thing to do.

	David