From: Michal Suchanek Date: 2006-03-14T01:44:58+09:00 Subject: Re: unicode in ruby On 3/12/06, Austin Ziegler wrote:> On 3/11/06, Michal Suchanek wrote:> > On 3/11/06, Austin Ziegler wrote:> >> UTF-16 is actually pretty performant and the implementation of> >> wchar_t on MacOS X and Windows is (you guessed it!) UTF-16. The> >> filesystems for both of these operating systems (which have *far*> >> superior Unicode support than anything else) both use UTF-16 as the> >> native filename encoding (this is true for HFS+, NTFS4, and NTFS5).> >> The only difference between what MacOS X does and Windows does for> >> this is that Apple chose to use decomposed characters instead of> >> composed characters (e.g., LOWERCASE E + COMBINING ACUTE ACCENT> >> instead of LOWERCASE E ACUTE ACCENT).> >>> >> Look at the performance numbers for ICU4C: it's pretty damn good.> >> UTF-32 isn't exactly space conservative (since with UTF-16 *most* of> >> the BMP can be represented with a single wchar_t, and only a few need> >> surrogates taking up exactly *two* wchar_ts, whereas *all* characters> >> would take up four uint32_t under UTF-32). ICU4C uses UTF-16> >> internally. Exclusively.> > I do not care what Windows, OS X, or ICU uses. I care what I want to> > use. Even if most characters are encoded with single word you have to> > cope with multiword characters. That means that a character is not a> > simple type. You cannot have character arrays. And no library can> > completely wrap this inconsistency and isolate you from dealing with> > it.>> If you're simply dealing with text, you don't need arrays of characters.> Frankly, if you don't care what Windows, OS X, and ICU use, then you're> completely ignorant of the real world and what is useful and necessary> for Unicode. The native encoding is bound to be different between platforms. I wantto use an encoding that I like on all platforms, and convert thestrings for filenames or whatever to fit the current platform. That iswhy I do not care what a particular platform you name uses. >> By the way, you are wrong -- you *can* have arrays of characters. It's> just that those characters are not guaranteed to be a fixed length. It> will be the same with Ruby moving forward. Yes, you can have arrays of strings. Nice. But to turn a text stringinto a string of characters you have to turn it into an array ofstrings. Instead of just indexing an array of basic types thatrepresent the characters. And there is a need to look at the actual characters at times.Thereare programs that actually process the text, not only save what theuser entered in a web form. I can think of text editors, terminalemulators, and linguistic tools. I am sure there are others. >> > Even if the library is performant with multiword characters it is> > complex. That means more prone to errors. Both in itself and in the> > software that interfaces it.>> Nice theory. What reduces the number of errors is no longer thinking in> terms of arrays of characters, but in terms of text strings. Or strings of strings of 16-bit words, packed? No, thanks. I want to avoid that. >> > You say that utf-16 is more space-conserving for languages like> > Japanese. Nice. But I do not care. I guess text consumes very small> > portion of memory on my system. Both ram and hardrive. I do not care> > if that doubles or quadruples. In the very few cases when I want to> > save space (ie when sending email attachments) I can use gzip. It can> > even compress repetitive text which no encoding can.>> If you don't care, then why are you arguing here? The Japanese -- which> would include Matz -- *do* care. I do not care about the space inefficiency. Be it inefficiency instoring Czech text, Japanese text, English text, or any other. It hasnothing to do with the fact I do not speak Japanese. I think that most of my ram and hardrive space is consumed by otherstuff than text. For that reason I do not care about the relativeefficiency of text encoding. It will have minimal impact on theperformance or amounts of memory consumed on the system. And there isalways the possibility to compress the text. >> >> On 3/10/06, Anthony DeRobertis wrote:> >>> Austin Ziegler wrote:> >>> Personally, my file names have been in UTF-8 for quite some time> >>> now, and it works well: What exactly is this 'stone age' you refer> >>> to?> >> Change and environment variable and watch your programs break that> >> had worked so well with Unicode. *That* is the stone age that I refer> >> to. I'm also guessing that you don't do much with long Japanese> >> filenames or deep paths that involve *anything* except US-ASCII (a> >> subset of UTF-8).> > Hmm, so you call the possibility to choose your encoding living in> > stone age. I would call it living in reality. There are various> > encodings out there.>> Yes, it's the stone age. The filesystem should allow you to see things> in UTF-8 or SJIS or EUC-JP if you want, but internally it should be> using something a hell of a lot smarter than those encodings. This is> what HFS+ and Windows allow.> Well, the libc could store the strings in some utf-* encoding on thedisk, and translate that based on the current locale. I wonder if thatis against POSIX or not.But it is not done, and it is wrong. There are problems of this kindon Windows as well. It is still not recommended to use non-asciicharacters in filenames around here.. Thanks Michal