From: Michal Suchanek <hramrach@...>
Date: 2007-05-08T21:43:42+09:00
Subject: Re: separate Chinese and English! with Ruby

On 08/05/07, Nanyang Zhan <sxain@hotmail.com> wrote:> Eden Li wrote:...> Thanks, Zev. but my current problem is about Chinese.> I am going to figure out a way to separate Chinese string from a string> mix with other characters.> What I mean other Characters are alphabets from English or/and other> languages, like Ô, é, á... (may I call them western words?)>> This string may be containing no Chinese:> "String without Chinese" ,I don't need to do anything about it, other> than identify such strings.> "中文 Western Words"  #Chinese characters + space + western words.> "中文・另一些中文 western words" ＃Chinese characters may be separated by> punctuations, or/and space like:> "中文 前有空格 western words"> Almost all Chinese phrases are at the beginning of the strings.> But some may contain numbers, like:> "2007年的日记 diary of 2007"> or some time English or alphabets are used as part of Chinese> phrases,like:> "BB日记 diary of my baby">> Eden Li wrote:> > Nooo! Those are the first BYTES of the UTF-8 encoding of the> > punctuation that you listed.>> Finally, I know what those number are. Thanks.> >so if you remove them from a givenstring, you're going to get back a poorly encoded UTF-8 string>> In fact, I wanted to use those number to test whether a character is> Chinese or not (if 'character[0]' fit the range of [226, 228, 229, 230,> 231, 233, 239], then it was likely to be a Chinese). (Now I know it may> be wrong.)> Then  depend on this judgment, if this part of string ( string would be> splited by space, divided into parts at the beginning) containing more> X%, say 60%, of this kind of characters, then I would mark this parts as> Chinese phrase, then take it out of string.>> I still want to use this strategy. but> As you point out, [226, 228, 229, 230, 231, 233, 239] are not safe to> identify Chinese, is there any other easy way to identify Chinese> characters?
I guess this should give you what you want:
irb(main):001:0> s = "大智若愚 asdfaf sdgs"=> "\345\244\247\346\231\272\350\213\245\346\204\232 asdfaf sdgs"irb(main):002:0> s.unpack "U*"=> [22823, 26234, 33509, 24858, 32, 97, 115, 100, 102, 97, 102, 32,115, 100, 103, 115]irb(main):003:0>
The "U*" specifier should give you a list of unicode codepoints (seethe high numbers for Chinese characters). You can use one of theunicode links mentioned earlier to find the codepoint range for EAscripts. 32 is a space so you can find the last EA character, thefirst space following that, and pack the two parts back into strings.
Thanks
Michal