From: Stefano Crocco <stefano.crocco@...>
Date: 2010-02-02T21:54:43+09:00
Subject: Re: [ENCODING] UTF8 hell

On Tuesday 02 February 2010, Xavier Noëlle wrote:
> |Hello,
> |I'm trying to deal with Ruby flaws with encoding, which I thought
> |would be almost past with Ruby 1.9. I managed to find a solution for
> |Ruby 1.8 and thought I did for Ruby 1.9...but in fact, no !
> |
> |I fetch rows from an UTF8 database and try to work with the string. To
> |do so, I would like it to be UTF8 encoded.
> |
> |"str.encoding()" gives me "ASCII-8BIT"...so, I thought one of these
> |lines would solve the problem
> |str.replace(Iconv.iconv("UTF8", "ascii", self).join())
> |OR
> |self.encode!('UTF-8')
> |
> |But they don't !
> |First one: in `iconv': "\xE8te pour luth" (Iconv::IllegalSequence)
> |Second one: in `encode!': "\xE8" from ASCII-8BIT to UTF-8
> |(Encoding::UndefinedConversionError)
> |
> |The base string is "Oeuvre complète pour luth" and displays well in
> |PHPMyAdmin.
> |
> |Any idea ?
> |TIA,

I'm not sure, but basing on my experience, it may be that the string are 
indeed stored as UTF-8, but the library you use to read from the database 
doesn't take care of informing ruby of the fact, so ruby assumes it is a 
generic array of bytes (which means, ruby thinks the string has encoding 
ASCII-8BIT, which is the same as BINARY).

If this is the case, you don't need to transcode the string (which is what 
encode does), but simply tell ruby which is the correct encoding, using the 
force_encoding method.

I hope this helps

Stefano