From: Brian Candler <B.Candler@...>
Date: 2007-02-27T17:40:20+09:00
Subject: Re: open-uri and utf8

On Tue, Feb 27, 2007 at 10:50:11AM +0900, Kjell Olsen wrote:
> I'm trying to make requests to google translate [http://google.com/ 
> translate_t] to translate words through a little ruby script. I can't  
> get open-uri to work on URL's with accented characters (����� etc).  

URLs cannot contain accented characters directly - they must be escaped into
%xx hex form. See RFC 2396:

   Data must be escaped if it does not have a representation using an
   unreserved character; this includes data that does not correspond to
   a printable character of the US-ASCII coded character set, or that
   corresponds to any US-ASCII character that is disallowed, as
   explained below.

> URI.encode()ing the url breaks the characters down into gunk (� => % 
> C3%A9) which the translator doesn't understand.

Looks like you have a disagreement between yourself and Google about what
character set to use. %C3%A9 looks like a two-byte UTF8 character to me.
Perhaps Google expects ISO-8859-1; have a look at their API documentation if
they provide any. You should be able to use iconv to convert from one to the
other.

In ISO-8859-1, your accented character will almost certainly appear as a
single byte, which will URI.encode to a single %xx

HTH,

Brian.