From: Michal Suchanek Date: 2006-03-15T00:36:22+09:00 Subject: Re: unicode in ruby On 3/13/06, Austin Ziegler wrote:> On 3/13/06, Michal Suchanek wrote:> > On 3/12/06, Austin Ziegler wrote:> >> On 3/11/06, Michal Suchanek wrote:> >>> I do not care what Windows, OS X, or ICU uses. I care what I want to> >>> use. Even if most characters are encoded with single word you have> >>> to cope with multiword characters. That means that a character is> >>> not a simple type. You cannot have character arrays. And no library> >>> can completely wrap this inconsistency and isolate you from dealing> >>> with it.> >> If you're simply dealing with text, you don't need arrays of> >> characters. Frankly, if you don't care what Windows, OS X, and ICU> >> use, then you're completely ignorant of the real world and what is> >> useful and necessary for Unicode.> > The native encoding is bound to be different between platforms. I want> > to use an encoding that I like on all platforms, and convert the> > strings for filenames or whatever to fit the current platform. That is> > why I do not care what a particular platform you name uses.>> I think you're just confused here, Michal.In what way?>> >> By the way, you are wrong -- you *can* have arrays of characters.> >> It's just that those characters are not guaranteed to be a fixed> >> length. It will be the same with Ruby moving forward.> > Yes, you can have arrays of strings. Nice. But to turn a text string> > into a string of characters you have to turn it into an array of> > strings. Instead of just indexing an array of basic types that> > represent the characters.>> > And there is a need to look at the actual characters at times.There> > are programs that actually process the text, not only save what the> > user entered in a web form. I can think of text editors, terminal> > emulators, and linguistic tools. I am sure there are others.>> NO! This is where you're 100% wrong. Text editors, terminal emulators,> and linguistic tools *especially* should never be looking at the raw> bytes underneath the character strings. They should be dealing with the> characters as discrete entities. I am saying I want to look at characters, not that I want to look at bytes.And I am saying that looking at entities that happen to be all thesame size makes things much simpler than looking at strings packedinto another string without separators. And multiword characters areword strings, nothing else. >> This is what I'm talking about. Byte arrays as characters are nonsense> in today's world. If you don't have an encoding attached to something,> then you can't *possibly* know what it means. No, it is not.Sure, I am not for byte arrays or chunks of data of unknown encoding. >> >>> Even if the library is performant with multiword characters it is> >>> complex. That means more prone to errors. Both in itself and in the> >>> software that interfaces it.> >> Nice theory. What reduces the number of errors is no longer thinking in> >> terms of arrays of characters, but in terms of text strings.> > Or strings of strings of 16-bit words, packed? No, thanks. I want to> > avoid that.>> Um. You're confused here. It's a text string with a UTF-16 encoding. Yes, It is a text string, which is a sring of characters packed oneafter another, which happen to be themself strings of 16-bit words.Give me the 100th character. >> >>> You say that utf-16 is more space-conserving for languages like> >>> Japanese. Nice. But I do not care. I guess text consumes very small> >>> portion of memory on my system. Both ram and hardrive. I do not> >>> care if that doubles or quadruples. In the very few cases when I> >>> want to save space (ie when sending email attachments) I can use> >>> gzip. It can even compress repetitive text which no encoding can.> >> If you don't care, then why are you arguing here? The Japanese --> >> which would include Matz -- *do* care.> > I do not care about the space inefficiency. Be it inefficiency in> > storing Czech text, Japanese text, English text, or any other. It has> > nothing to do with the fact I do not speak Japanese.>> > I think that most of my ram and hardrive space is consumed by other> > stuff than text. For that reason I do not care about the relative> > efficiency of text encoding. It will have minimal impact on the> > performance or amounts of memory consumed on the system. And there is> > always the possibility to compress the text.>> Then you are willfully ignorant of the concerns of a lot of people. OK, you are concerned about the space consumed by text. I wonder howlarge portion of your ram is used for text. Or how large portion ofyour hardrive is used by text for which you can choose the encoding.I got lots of C sources but I suspect that C compilers won't acceptwide characters anytime soon. And anything but byte encoding is quitepointless for C sources. Most of the stuff is identifiers that can beonly 7-bit ascii anyway. Thanks Michal