From: Patrick Lajeunesse <patlaj@...>
Date: 2009-07-07T22:18:50+09:00
Subject: Mechanize: How to scrape accented chars that weren't entered as 	entities?

--0016e6464d8c7ce18c046e1d7676
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi,
I'm trying to scrape links using Mechanize. Sometimes accented characters
(on French pages) are corrupt once Ruby gets them. To see what I mean, check
this:

require 'mechanize'
a = WWW::Mechanize.new
page = a.get('http://www.agr.gc.ca/cb/index_f.php?s1=n&s2=index&page=2009_07
')
page.links.each do |a_link|
  puts a_link
end

Of course, it's only the accents that are entered in plain text (i.e.,
without entities) that have this problem. But in an imperfect world, I can't
always count on accents being entered properly.

Is there anything I can do about this? I've tried using Iconv to convert the
strings to UTF-8, but that just resulted in a different (but still wrong)
character in place of the broken ones.

Thanks for any help,

Patrick

--0016e6464d8c7ce18c046e1d7676--