From: Florian Gilcher <flo@...>
Date: 2012-04-18T17:51:37+09:00
Subject: Re: Upgrading to Ruby 1.9 - Should I? Is it painful?


On Apr 18, 2012, at 12:46 AM, Brian Candler wrote:

> Bartosz Dziewoński wrote in post #1056782:
>> 2. Always immediately convert any incoming data to UTF-8. Sensible
>> frameworks should already do it for you, but mind it if rolling your
>> own. Use code like File.open('asd.txt', 'r:utf-8') if necessary. (E.g.
>> if it doesn't work without the utf-8 part.)
> 
> Note: that doesn't actually "convert incoming data to UTF-8". It just 
> *labels* incoming data as being UTF-8, whether it is or not. If you want 
> to convert data, you have to provide both the target and original 
> encodings, e.g.
> 
> File.open('asd.txt', 'r:utf-8:iso-8859-1')
> 
> The fundamental point with ruby 1.9 is that every String is a 
> two-dimensional vector of bytes + encoding. There are some circumstances 
> in which Strings which contain invalid characters (according to their 
> declared encoding) work, and others in which they don't. There are some 
> circumstances in which Strings with different encodings can be combined, 
> and others when they can't.
> 
> Have fun trying to understand it.

The fundamental problem is that you can only guess what the external world
looks like as input streams have no encoding. The problem exists in Ruby 1.8 
as well - its just not as visible.

In practice though, I rarely have any problems, only when dealing
with systems that are messy with respect to encoding in the first place.

Best regards,
Florian