From: Christian Neukirchen Date: 2006-06-19T20:16:42+09:00 Subject: Re: Unicode roadmap? Tim Bray writes: > On Jun 18, 2006, at 12:17 PM, Christian Neukirchen wrote: > >>> This is another thing you need your String class to be smart about. >>> You want an equality test between "m�s" and "m�s" to always be true >>> even their "�" characters are encoded differently. The right way to >>> solve this is called "Early Uniform Normalization" (see http:// >>> www.w3.org/TR/2003/WD-charmod-20030822/#sec-Normalization); the idea >>> is you normalize the composed characters at the time you create the >>> string, then the internal equality test can be done with strcmp() or >>> equivalent. >> >> Does that mean that binary.to_unicode.to_binary != binary is >> possible? >> That could turn out pretty bad, no? > > Yes, but having "m�s" != "m�s" is pretty bad too; the alternative is > normalizing at comparison time, which would really hurt for example > in a big sort, so you'd need to cache the normalized form, which > would be a lot more code. > > binary.to_unicode looks a little weird to me... can you do that > without knowing what the binary is? If it's text in a known > encoding, no breakage should occur. If it's unknown bit patterns, > you can't really expect anything sensible to happen... or am I > missing an obvious scenario? -Tim Those were just fictive method calls. But let's say I read from a pipe and I know it contains UTF-16 with BOM, then .to_unicode would make perfect sense, no? In case of binary bit patterns, I sooner or later would expect some kind of EncodingError, given this API. (I haven't seen yet drafts of how the API really will be.) -- Christian Neukirchen http://chneukirchen.org