From: Tim Bray <tbray@...>
Date: 2006-06-19T05:52:02+09:00
Subject: Re: Unicode roadmap?

On Jun 18, 2006, at 12:17 PM, Christian Neukirchen wrote:

>> This is another thing you need your String class to be smart about.
>> You want an equality test between "m�s" and "m�s" to always be true
>> even their "�" characters are encoded differently.  The right way to
>> solve this is called "Early Uniform Normalization" (see http://
>> www.w3.org/TR/2003/WD-charmod-20030822/#sec-Normalization); the idea
>> is you normalize the composed characters at the time you create the
>> string, then the internal equality test can be done with strcmp() or
>> equivalent.
>
> Does that mean that  binary.to_unicode.to_binary != binary  is  
> possible?
> That could turn out pretty bad, no?

Yes, but having "m�s" != "m�s" is pretty bad too; the alternative is  
normalizing at comparison time, which would really hurt for example  
in a big sort, so you'd need to cache the normalized form, which  
would be a lot more code.

binary.to_unicode looks a little weird to me... can you do that  
without knowing what the binary is?  If it's text in a known  
encoding, no breakage should occur.  If it's unknown bit patterns,  
you can't really expect anything sensible to happen... or am I  
missing an obvious scenario?  -Tim