From: James Edward Gray II <james@...>
Date: 2009-09-10T22:44:36+09:00
Subject: Re: Ruby1.9 Encoding

On Sep 10, 2009, at 4:20 AM, Juliano 준호 wrote:

> Hey, guys!
>
> I've just started learning Ruby from Python and recently posted a
> question here that was promptly and effectively answered (thanks,
> Glenn!), so I decided to come here once more. I hope I'll be able to
> be answering some of the questions on my own soon. ;)

Welcome to Ruby.

> The problem I'm having now is converting the string from UTF-8 (I'm
> running Ubuntu with pt_BR.UTF-8 locale) into EUC-KR, run the
> Morphological Parser, read its output and process it. I have to parse
> a whole bunch of data "spidered" from the internet and it works great
> until the encoder comes across a Korean typo... Let's make this point
> clearer: the EUC-KR encoding does not cover all the possible
> combinations of initial and final consonants + vowels, as does
> unicode. Explaining: Unicode has 21708 codepoints for hangul whereas
> EUC-KR has only 11172. In fact, most of the extra chars are not used
> in day by day life, but still they can be used as abbreviations,
> slang, smileys or typos...
>
> In Python, I would simply throw these chars away but I really didn't
> manage to understand the "Ruby encoding way"...

I think we can throw them away in Ruby too.  See below.

> I know I'm missing something, but I can't seem to find enough info  
> around... Google
> doesn't seem to know much of this either...

I wrote a lot about Ruby's encoding engine on my blog:

http://blog.grayproductions.net/articles/understanding_m17n

> The part of my code which deals with this is as follows:
>
>  def run(txt)
>    txt = txt.encode("EUC-KR")

Try replacing the above line with:

   txt = txt.encode("EUC-KR", invalid: :replace, undef: :replace,  
replace: "")

>    kts_file = Tempfile::new('kts_text')
>    kts_file = open(kts_file.path, "w:EUC-KR")
>    kts_file << "#{txt}\n"
>    kts_file.close
>    cmd = "ktspell < #{kts_file.path}" # 2> /dev/null"
>    IO::popen(cmd, "r:EUC-KR").read.encode("UTF-8")
>  end

Hope that helps.

James Edward Gray II