From: Tim Bray <tbray@...>
Date: 2006-06-18T13:17:35+09:00
Subject: Re: Unicode roadmap?

On Jun 17, 2006, at 4:08 AM, Juergen Strobel wrote:

> 1. Strings should deal in characters (code points in Unicode) and not
> in bytes, and the public interface should reflect this.

Be careful.  People who care about this stuff might want to read  
http://www.w3.org/TR/2005/REC-charmod-20050215/ It turns out that  
characters do not correspond one-to-one with units of sound, or units  
of input, or units of display.  Except for low-level stuff like  
regexps, it's very difficult to write any code that goes character-at- 
a-time that doesn't contain horrible i18n bugs. For practical  
purposes, a String is a more useful basic tool than a character.

> 5. Since the String class is quite smart already, it can implement
> generally useful and hard (in the domain of Unicode) operations like
> case folding, sorting, comparing etc.

Be careful.  Case folding is a horrible can of worms, is rarely  
implemented correctly, and when it is (the Java library tries really  
hard) is insanely expensive.  The reason is that case conversion is  
not only language-sensitive but jurisdiction sensitive (in some  
respects different in France & Qu�bec).  Trying to do case-folding on  
text that is not known to be ASCII is likely a symptom of a bug.

> - This ties Ruby's String to Unicode. A safe choice IMHO, or would we
> really consider something else? Note that we don't commit to a
> particular encoding of Unicode strongly.

For information: The XML view is that Shift-JIS, KOI8-R, EBCDIC, and  
many others are all encodings of Unicode and a best effort should be  
made to accept and emit all sane encodings on demand.  Most XML  
software sticks to a single encoding, internally.

  -Tim