From: James Gray <james@...>
Date: 2009-08-11T10:24:54+09:00
Subject: Re: ActiveRecord to_json encoding

On Aug 10, 2009, at 1:36 PM, Calvin Nguyen wrote:

> Question:
> Hi, our company is using Ruby 1.8.6 with Rails 2.2.2.  Does anyone
> know we can explicitly specify what encoding to use when calling

Rails pretty much assumes UTF-8 data everywhere.  The path of least  
pain is definitely to try to work exclusively with UTF-8, since that's  
mainly what Ruby 1.8.x can handle.

However, I believe the default encoding of a web page is ISO-8859-1,  
unless you specify otherwise.  If you served up a form, a browser sent  
you some data from that form, you saved it into the database, and you  
never specified an encoding or tried to transcode the content, your  
data is probably in ISO-8859-1.  Indeed, that seems to be the case,  
from what you are showing:

> If we use the browser to submit HTTP Get requesting JSON format, save
> the file and view it in binary mode in Hexadecimal representation,
> this is what we get.  It looks like this is using extended ASCII.
>
> Bytes                          Text
> 43 61 66 E9                    C a f (should be accented e but get  
> weird
> block unprintable character)

That's ISO-8859-1 data:

$ ruby -KU -r iconv -e 'puts Iconv.conv("UTF-8", "ISO-8859-1", [0x43,  
0x61, 0x66, 0xE9].pack("C*"))'
Café

Thus, you need to transcode it to UTF-8 before using operations like  
to_json() that assume UTF-8, using the reverse of the transform I just  
showed.  Even better, you could transcode the existing data in your  
database to UTF-8 and then mark all pages on your site as UTF-8  
encoded, possibly by adding this line to the head of your HTML:

<meta http-equiv="Content-type" content="text/html; charset=utf-8">

You may also want to instruct your web server to return a proper  
Content-Type encoding header.

I hope that helps.

James Edward Gray II