From: "Mariusz Pękala" <skoot@...>
Date: 2007-05-08T19:04:17+09:00
Subject: Re: separate Chinese and English! with Ruby

--ZPt4rx8FFjLCG7dd
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-05-08 17:47:58 +0900 (Tue, May), Nanyang Zhan wrote:
> Thanks, Zev. but my current problem is about Chinese.
> I am going to figure out a way to separate Chinese string from a string=
=20
> mix with other characters.
> What I mean other Characters are alphabets from English or/and other=20
> languages, like =C3=94, =C3=A9, =C3=A1... (may I call them western words?)
>=20
> This string may be containing no Chinese:
> "String without Chinese" ,I don't need to do anything about it, other=20
> than identify such strings.
> "=E4=B8=AD=E6=96=87 Western Words"  #Chinese characters + space + western=
 words.
> "=E4=B8=AD=E6=96=87=E3=83=BB=E5=8F=A6=E4=B8=80=E4=BA=9B=E4=B8=AD=E6=96=87=
 western words" =EF=BC=83Chinese characters may be separated by=20
> punctuations, or/and space like:
> "=E4=B8=AD=E6=96=87 =E5=89=8D=E6=9C=89=E7=A9=BA=E6=A0=BC western words"
> Almost all Chinese phrases are at the beginning of the strings.
> But some may contain numbers, like:
> "2007=E5=B9=B4=E7=9A=84=E6=97=A5=E8=AE=B0 diary of 2007"
> or some time English or alphabets are used as part of Chinese=20
> phrases,like:
> "BB=E6=97=A5=E8=AE=B0 diary of my baby"

[...]

> In fact, I wanted to use those number to test whether a character is=20
> Chinese or not (if 'character[0]' fit the range of [226, 228, 229, 230,=
=20
> 231, 233, 239], then it was likely to be a Chinese). (Now I know it may=
=20
> be wrong.)
> Then  depend on this judgment, if this part of string ( string would be=
=20
> splited by space, divided into parts at the beginning) containing more=20
> X%, say 60%, of this kind of characters, then I would mark this parts as=
=20
> Chinese phrase, then take it out of string.
>=20
> I still want to use this strategy. but
> As you point out, [226, 228, 229, 230, 231, 233, 239] are not safe to=20
> identify Chinese, is there any other easy way to identify Chinese=20
> characters?

Just a random idea - maybe, if there is a problem with finding Chinese
characters, you can define the range of non-Chinese (defined for this purpo=
se as western)
characters?
Maybe just finding words composed of only Latin and Common scripts would
be enough?
Or do you plan to have a Chinese - Japanese pairs? You said about
'westen words' and your examples were in English..


--=20
No virus found in this outgoing message.
Checked by 'grep -i virus $MESSAGE'
Trust me.

--ZPt4rx8FFjLCG7dd
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6-ecc01.6 (GNU/Linux)

iD8DBQFGQEsTsnU0scoWZKARArKIAJ9oKYMTWx3nyddelVOMQOMA5qyDhgCgmrxz
p3jTwaPMxbfJH6HHs+3pfoo=
=ZPwL
-----END PGP SIGNATURE-----

--ZPt4rx8FFjLCG7dd--