From: Curt Sampson Date: 2002-08-07T16:38:25+09:00 Subject: Re: Unicode in Ruby now? On Tue, 6 Aug 2002, Tom Sawyer wrote: > but instead to ending it this way, can anyone sum up the overall > conclusions of this lengthy discussion? that's what i'd like to hear. Unicode covers a lot of what people want to do, but not everything. Therefore there will be specialized situations in which you cannot use Unicode. Unicode has some variable length issues with surrogates, combining pairs, and various encodings. In all encodings, combining pairs are multiple Unicode code points representing a single glyph on screen. UTF-32: 4-byte chars, except for combining pairs. UTF-16: 2-byte chars, except for combining pairs and surrogates. UTF-8: 1-byte chars, except for non-ASCII, combining pairs and surrogates. Note well that *all* Unicode encodings have variable length issues, due to the combining chars. Surrogate pairs are dealt with very, very simply if you are not actually interpreting them (you can easily tell from looking at a single character if it's a high byte or low-byte surrogate, and the only rule is not to split them). Combining chars are rather more difficult. The processing you do on strings with surrogate pairs can be divided into three categories. 1. No special processing required, because your processing cannot break anything. (E.g., web page form information to and from a database). 2. Little special processing required to avoid any breakage (i.e., don't split pairs); minimal damage if you do split pairs. 3. Extensive processing required because you're interpreting the pairs. 3 is necessary only for things interacting with a user that have to display proper glyphs and accept input. Most such things are language-specific, because it's impossibly complex to have one way of doing things that covers everything. (For a start, just try to think up an input method that works for Chinese, Japanese and Korean simultaneously. And then also handles Hebrew.) And most people don't need that anyway. 2 Is pretty trivial to implement above the string layer, where necessary, and often needs to be combined with other stuff anyway. (E.g., line wrapping algorithms, which are language-specific.) 1 Is suprisingly common situation. There is no equivalant for combining pairs; you have to do some real processing there. Pretty much everybody has just ignored the problem from the beginning, and it's not been that big a deal. And one more thing a lot of people have missed: UTF-8 is the most efficient storage format only for text the majority of which is ASCII; if it's not UTF-16 is about as efficient or, in the case of Asian languages, more efficient. A rather more disputed fact is that surrogate pairs and combining characters are pretty darn rare. But one thing to remember is that a) people have been mostly ignoring combining pairs all along, with very little fuss from anyone, and b) only within the last year or two have there even been any characters assigned to the surrogate pair area. > i.e. does any encoding scheme out there do the job, the whole job, and > nothing but the job? or are they all flawed and somebody someday needs > to sit down and figure the problem out and fix it for good? They are all flawed in one way or another. Figuring out something that covers everything would probably not be much more difficult than Regan's Star Wars project, however. One should also keep in mind that the general public has been dealing extensively with systems using flawed encodings for the past thirty years, in the case of ASCII and the English speaking world, and ten years or so in the case of Asian standards and the Asian world, and there have not been major complaints regarding single-language support. (Unicode basically just combines all the popular character sets of the world into one big one to solve the multiple-language support problem. It does not attempt--much, at any rate--to solve other problems that have been present all along that people have brought up here. I opine that that's because had those problems really needed a solution, the solution would have been created and standardized.) cjs -- Curt Sampson +81 90 7737 2974 http://www.netbsd.org Don't you know, in this new Dark Age, we're all light. --XTC