From: Zev Blut <rubyzbibd@...>
Date: 2007-05-08T17:34:57+09:00
Subject: Re: separate Chinese and English! with Ruby

On Tue, 08 May 2007 17:22:11 +0900, John Joyce  
<dangerwillrobinsondanger@gmail.com> wrote:

>
> On May 8, 2007, at 3:54 PM, Nanyang Zhan wrote:
>
>> John Joyce wrote:
>>
>>> I don't know if the two main chinese sets are encoded as different
>>> ranges or simply declared in some way.
>>> In general in Unicode a character is the same character even when it
>>> appears in a different language.
>>
>> Many characters of these two set of Chinese(in fact, including Chinese
>> Characters in Japanese  and Korean...) are the same. Aren't they encoded
>> to the same codes when they are identical?
>>
>
> Yes. There is lots of overlap. So there is not always a clean separation  
> line. But, the Japanese and Korean phonetic characters will be in a  
> range. You might never use all the kanji/hanzi chinese characters, and a  
> few are Japanese only (very few).

Here is an API that might help "guess" if the text is Japanese, Korean or
Chinese:

http://raa.ruby-lang.org/project/libguess-ruby/

http://www.honeyplanet.jp/download.html#libguess

Cheers,
Zev