From: Brian Candler <b.candler@...>
Date: 2009-12-30T03:06:27+09:00
Subject: Re: ruby 1.9 hates you and me and the encodings we rode in on so just get  used to it.

Bill Kelly wrote:
>> IMO, both python 3 and erlang have got the right idea when it comes to 
>> handling UTF8.
> 
> Could you summarize what you feel the key difference of
> the python 3 / erlang approach is, compared to ruby19 ?

As far as I can tell, both have two distinct data structures. One 
represents a binary object: a string of bytes. The other represents a 
textual string, a string of UTF-8 codepoints. (In the case of erlang, 
these are "binaries" and "lists" respectively).

ruby 1.9 has one String which tries to do both jobs. I commonly deal 
with binary data: ASN1 encodings, PDFs, JPGs, firmware images, ZIP 
files, and so on. And yet ruby 1.9 has it now deeply embedded that all 
data is text (which is not clearly true: rather the converse, all text 
is data). At best you can get ruby 1.9 to tell you that your data is 
"ASCII-8BIT", even when it has nothing to do with ASCII whatsoever.

I really miss having an object which simply represents a "sequence of 
bytes". Of course ruby 1.9 can do it, if you jump through the right 
hoops.

I really miss being able to look at a simple expression such as
    a = b + c
when I know that both b and c are String objects, and being able to say 
for definite whether or not it will raise an exception.

> However, my understanding is that one is supposed to be
> able to effectively make ruby behave as a "UTF-8 only"
> language if one makes sure external data is transcoded to
> UTF-8 at I/O boundaries.

If you jump through the right hoops, you can do this. If you omit any of 
the hoops, your program may work on some systems but not on others. ruby 
1.9's behaviour is environment-sensitive.

But the worst part of all this is that it's totally undocumented. Look 
into the 'ri' pages for most of ruby core, for any method which either 
takes a string, returns a string, or acts on a string, and you are 
unlikely to find any definition of its encoding-related behaviour, 
including under what circumstances it may raise an exception.

By tagging every string with its own encoding, ruby 1.9 is solving a 
problem that does not exist: that is, how do you write a program which 
juggles multiple strings in different encodings all at the same time?

And as the OP has discovered, the built-in support is often incomplete 
so that you have to use libraries like Iconv anyway.
-- 
Posted via http://www.ruby-forum.com/.