From: Curt Sampson Date: 2002-08-06T09:52:57+09:00 Subject: Re: Unicode in Ruby now? On Tue, 6 Aug 2002, Clifford Heath wrote: > I see you have built a new 21-bit computer which is going to conquer > the world, have you? If not, what size memory word *do* you store that > 21-bit value in? If I want to store 21-bit values, I store them in a 32-bit word, wasting 11 bits. That does not change the value to be 32 bits in range. But as far as Unicode goes, I store that in 16-bit words, for reasons I have gone into in detail before. > > Also note that UTF-32 is still "variable length" in some senses, > > in that it can still have combining characters that you need to > > interpret. > > Yes, and aren't they a bugger! But unavoidable, since rooted in real > languages. The only saving grace is that many applications don't have > to worry about them. Many applications need not worry about surrogate pairs, either. Also, surrogate pairs are much, much easier to deal with than combining characters. So removing surrogate pairs does little to solve this problem. > I *do* have an > issue with the way the standard shuts out a large number of characters > in order to protect a broken encoding. It doesn't shut them out in the slightest; surrogate pairs work just fine. Please re-read my previous posts. > Unicode 3 doesn't go far enough > not because it panders to the majority but because it shuts out the > minority who need those characters. Hello! It does not shut them out! What part of "surrogate pairs work fine without special processing" don't you understand? Why don't you point out how they break, if you think that they do, rather than making unsubstantiated claims? Also, I would like to know just who is using these characters, and for what. In Japan, I have *never* met anyone outside of someone testing Unicode or Unicode-based applications that has ever used a surrogate character. Aside from a few kanji specialists, Japanese people would be hard pressed even to identify the readings and meanings of more than a handful of these surrogate characters if you showed them the whole lot. The Japanese have demonstrated that they have no pressing need for these characters in their computer character sets by sticking for years to standards (EUC-JP, Shift-JIS, ISO-2022-JP) that do not contain any of these characters. (Unicode handles the characters from all of these character sets without using surrogate pairs.) > I have. Optimising for typical string use shouldn't come at the expense > of representational completeness. Of course not. But UTF-16 is representationally complete. There is no Unicode character you can represent in UTF-32 that you cannot represent in UTF-16. > How would you feel if some European > body decided that the letters J and Y were to be dropped from English > because they didn't fit in some six-bit encoding? (I once worked on a > CDC computer which used six-bit characters - no lower case!) NOTHING IS BEING DROPPED! ALL CHARACTERS ARE REPRESENTABLE IN UTF-16! HELLO! The proper parallel here is, "how would you feel if some European body decided that the thorn character was, only under certain circumstances, going to be marginally harder to deal with." BTW, when was the last time you used thorn? That's the equivalant of the kanji that are in the surogate range. cjs -- Curt Sampson +81 90 7737 2974 http://www.netbsd.org Don't you know, in this new Dark Age, we're all light. --XTC