From: Heesob Park <phasis@...>
Date: 2008-08-04T18:15:26+09:00
Subject: Re: handling special characters

Hi,

2008/8/4 Sajal Kayan <sajal@thaindian.com>:
> Hi all.
>
> I am very new to Ruby (5 days old) so my question might sound very
> noobish. I am posting it only cause I couldn't find a solution.
>
> I am using ruby to scrape content of a site.
>
> To be precise I am having problems with the ' character.
>
> Sample source page :
> http://thainews.prd.go.th/newsenglish/previewnews.php?news_id=255108040023
>
> The encoding is in tis-620 and I use Iconv to convert it to utf8,
> however the special quote character gives the following error on iconv
>
> /home/....../main.rb:37:in `iconv': "\222s announcement "...
> (Iconv::IllegalSequence)
>
> the affected code area
>
> body =  story.search("//font[@color=\"#333333\"]").inner_html
> body = body.gsub(/<(.|\n)+?>/, "")
> body = body.gsub(/�/, "\'")
> puts body
> body =   Iconv.iconv("utf8", "tis-620", body) #<-- this is line 37
> puts body
>
> Or try the following on irb
>
>  require 'rubygems'
> require 'net/http'
>  require 'open-uri'
> require 'iconv'
> story =
> Hpricot(open('http://thainews.prd.go.th/newsenglish/previewnews.php?news_id=255108040023'
> ))
> body =  story.search("//font[@color=\"#333333\"]").inner_html
> body = body.gsub(/<(.|\n)+?>/, "")
> body = body.gsub(/'/, "\'")
> puts body
>
> no matter whatever i put in the "'" it doesn't replace anything and the
> iconv still gives errors.
>
> I am looking for pointers on one of the following.
>
> 1) how do i replace "'" to "'" ?
body = body.gsub(/\222/,"\'")

> or 2) How can I make iconv ignore the "'" ?
>
The ' character (0x92) is not in tis-620 but in windows-874 character set.

Refer to
http://www.langbox.com/codeset/tis620.html
http://www.microsoft.com/globaldev/reference/sbcs/874.mspx

Try
body =   Iconv.iconv("utf-8", "windows-874", body).join

Regards,

Park Heesob