Re: \u escapes in string literals: proof of concept implementation

Martin Duerst wrote:
> At 13:10 07/10/23, David Flanagan wrote:
>> Martin Duerst wrote:
>>> It should also complain about paired surrogates. There are no surrogate
>>> pairs in UTF-8. \u designates a codepoint/character, but there is no
>>> codepoint or character \uD800, so we should complain.
>> I believe that Java's version of UTF-8 encodes surrogate pairs as two 3-byte sequences.
> 
> This may have been the case for some older (read: by now, really old)
> versions of Java, and would clearly not conform to the Unicode standard.
> It may also have been the case for some Java-internal, UTF-8-like
> encoding, but that's not UTF-8.

Its not UTF-8.  Sun calls it "Modified UTF-8" and it is used internally 
by the Java implementation.  Class files, for example, are encoded in 
this form.  Probably not relevant to the discussion here.

> 
>> And Java's \u escape only supports 16 bits.
> 
> Yes. But Java has \Uxxxxxx for characters outside the BMP. For

According to the document you link to, there is some kind of demo input 
method that supports the \U escape.  But Java source code itself still 
uses \u and uses surrogate pairs.  I tested this with Java 6:

$javac /tmp/U.java
/tmp/U.java:3: illegal escape character
         System.out.println("\U10FFFF");
                              ^
 > details, please see 
http://java.sun.com/developer/technicalArticles/Intl/Supplementary/.

I like the \Uxxxxxx escape instead of \u{}.  Would you consider this, Matz?

Having stated these facts about Java, I have to confess that I don't 
understand all this at a deep enough level to really know the pros and 
cons of supporting or forbidding codepoints from the surrogates block.

Right now, my patch delegates the codepoint->utf translation to the 
encoding object defined in enc/utf8.c. So it is possible to enable \u 
support without committing to the exact details of the translation.

	David