From: Nanyang Zhan <sxain@...>
Date: 2007-05-10T12:18:45+09:00
Subject: Re: separate Chinese and English! with Ruby

> NZ, could you share your final combined code? It might be useful to
> anyone using CJK, since Ruby originates in Japan that means a lot of
> people might find it useful. Might consider making a little gem out
> of it.


I do think it will be much helpful, because it only solve a very 
specified problem.
But anywhere, I paste it here. Maybe it could inspire somebody... who 
knows...

#This !!!!RoR!!!! snippet is used to separate Chinese phrase from 
specified formated strings:
#These strings may contain no Chinese:
#"a string without Chinese"
#or Chinese characters + space + western words: "中文 Western Words".
#"中文・另一些中文 western words" ＃Chinese characters may be separated by 
punctuations,
#or/and space like:
#"中文 前有空格 western words"
#Almost all Chinese phrases are at the beginning of the strings.
#But some may contain numbers, like:
#"2007年的日记 diary of 2007"
#or some time English or alphabets are used as part of Chinese
#phrases,like:
#"BB日记 diary of my baby"
#
#usage:
#separate_chinese("a string without Chinese") => "|||a string without 
Chinese"
#separate_chinese("2007年的日记 diary of 2007") => "2007年的日记|||diary of 
2007"
#chinese_str, other_str = separate_chinese("中文 前有空格 western 
words").split("|||")
#chinese_str => "中文 前有空格"
#other_str => "western words"

class Foo < ActiveRecord::Base
  def self.separate_chinese(n)
    ns = n.split(" ")
    i = ns.size
    ns.reverse.each do |p|
      i -= 1
      return ns.values_at(0..i).join(" ") + "|||" + ns.values_at((i + 
1)..(ns.size - 1) ).join(" ") if is_chinese(p)
    end
    "|||" << n
  end

  def self.is_chinese(n)
    cs = n.unpack("U*")
    chinese_character_num = 0
    cs.each do |unicode|
#comparing character's unicode to test if it is Chinese character
#19968-40869: unicode Chinese Character
#12288-12351: unicode CJK symbols and punctuation
#Note: as Eden Li have mentioned, there are a few more could be used in 
a Chinese Document.
      chinese_character_num += 1 if (unicode >= 19968 and unicode <= 
40869) or (unicode >= 12288 and unicode <= 12351)
    end
#if more the 29% of the characters a phrase contains is Chinese, it is 
Chinese phrase
#the value 29% servers well for my purpose, but use whatever you like.
    return true if chinese_character_num.to_f/cs.size > 0.29
    nil
  end
end

-- 
Posted via http://www.ruby-forum.com/.