Re: \u escapes in string literals: proof of concept implementation

At 13:10 07/10/23, David Flanagan wrote:
>Martin Duerst wrote:
>> It should also complain about paired surrogates. There are no surrogate
>> pairs in UTF-8. \u designates a codepoint/character, but there is no
>> codepoint or character \uD800, so we should complain.
>
>I believe that Java's version of UTF-8 encodes surrogate pairs as two 3-byte sequences.

This may have been the case for some older (read: by now, really old)
versions of Java, and would clearly not conform to the Unicode standard.
It may also have been the case for some Java-internal, UTF-8-like
encoding, but that's not UTF-8.

>And Java's \u escape only supports 16 bits.

Yes. But Java has \Uxxxxxx for characters outside the BMP. For
details, please see http://java.sun.com/developer/technicalArticles/Intl/Supplementary/.
Our syntax (\u{xxxxxx}, variable length) and Java's syntax are
different, but then Ruby and Java are different, anyway.

>So anyone porting Java software to Ruby will have to deal with strings containing surrogate pairs.

Yes, they will have to convert \Uxxxxxx to \u{xxxxxx}. Too bad,
or not a big deal, your milage may vary. If they still have
\u with surrogate code units, they should clean up their act.

>Even if it is not technically correct, I suspect we ought to allow this usage, following Java's lead.

Java is moving in the right direction, away from directly using
surrogates in character escapes. We shouldn't go in the opposite
direction.

>> That things are simpler is quite clear. But option a wouldn't be
>> difficult to implement, either, I guess. My suggestion is to stay
>> with option a until we have a better idea of which of b and c is
>> really needed/implementable/... 
>
>The appeal, to me, of my current version is that it is independent of the primary encoding.  \u escapes work no matter what -K option you specify, and they always translate to a specific byte sequence. On the other hand, I really don't know how to handle strings that mix sjis-encoded Kanji characters with \u escapes.  What should the encoding of the resulting string be?
>
>So chosing option a might be the best bet: \u escapes cause an error with -Ks or -Ke.  They are only allowed when the primary encoding is ascii or utf-8.  I think that would mean that no transcoding would be necessary.  I also think that my current patch doesn't need much modification: just the addition of errors when the encoding does not allow \u.
>
>I'll make this change if others agree that it is the right thing to do.

I very much agree that this is currently the best way to proceed.
We may come to another conclusion once other pieces of the puzzle
fall into place, but it's easier to make something work that once
was an error than to change functionality.

Regards,    Martin.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[e-mail]