From: Sajal Kayan <sajal@...>
Date: 2008-08-04T17:21:28+09:00
Subject: handling special characters

Hi all.

I am very new to Ruby (5 days old) so my question might sound very
noobish. I am posting it only cause I couldn't find a solution.

I am using ruby to scrape content of a site.

To be precise I am having problems with the ’ character.

Sample source page :
http://thainews.prd.go.th/newsenglish/previewnews.php?news_id=255108040023

The encoding is in tis-620 and I use Iconv to convert it to utf8,
however the special quote character gives the following error on iconv

/home/....../main.rb:37:in `iconv': "\222s announcement "...
(Iconv::IllegalSequence)

the affected code area

body =  story.search("//font[@color=\"#333333\"]").inner_html
body = body.gsub(/<(.|\n)+?>/, "")
body = body.gsub(/�/, "\'")
puts body
body =   Iconv.iconv("utf8", "tis-620", body) #<-- this is line 37
puts body

Or try the following on irb

 require 'rubygems'
require 'net/http'
 require 'open-uri'
require 'iconv'
story =
Hpricot(open('http://thainews.prd.go.th/newsenglish/previewnews.php?news_id=255108040023'
))
body =  story.search("//font[@color=\"#333333\"]").inner_html
body = body.gsub(/<(.|\n)+?>/, "")
body = body.gsub(/’/, "\'")
puts body

no matter whatever i put in the "’" it doesn't replace anything and the
iconv still gives errors.

I am looking for pointers on one of the following.

1) how do i replace "’" to "'" ?
or 2) How can I make iconv ignore the "’" ?

At first I thought this to be a I18n issue, but i guess getting rid of
the special character would be a simple string manipulation which i dont
get.
-- 
Posted via http://www.ruby-forum.com/.