From: Robert Klemme <shortcutter@...>
Date: 2007-07-30T18:26:00+09:00
Subject: Re: Text parser (text into sentences) that works with UTF-8 and multiple languages?

2007/7/30, mike b. <michael.w.bell@gmail.com>:
> I have to parse about 2000 files that are written in multiple
> languages (some English, some Korean, some Arabic and some Japanese).
> I have to split these UTF-8 encoded into individual sentences. Has
> anyone written a good parser that can parse all these non-Latin
> character languages or can someone give me some advice on how to go
> about writing a parser that can handle all these fairly different
> languages?

I would consider doing this in Java, as Java's regular expressions
support Unicode.  That might make the job much easier.  OTOH, if all
files use only dot, question mark etc. (i.e. ASCII chars) as sentence
delimiters then Ruby's regular expressions might as well do the job.

Kind regards

robert