From: Brylie Oxley <brylie@...>
Date: 2009-07-18T17:38:42+09:00
Subject: Re: Re-post: Parsing Japanese Language and Some Ruby Trivia

Hi,
I have a similar task. I have attached a text file with Japanese 
characters. Some of the words are fake and made to look similar to 
actually occurring words, as an experiment in linguistics.

Essentially what we are trying to accomplish is to get a count of every 
time one of the hiragana characters occurs adjacent to another hiragana 
character (including itself) within the context of a word.  The majority 
of the data in the file is frequency counts for the word over the course 
of several years, the numbers can be ignored for the purposes of this 
count.  At this point we are only concerned with the co-occurance of 
hiragana characters, but katakana and kanji may eventually be useful.

I currently have Ruby 1.8.7 installed and when I paste the characters 
into irb they print out as bytes and I am not sure where to begin 
figuring out how to write an effective regexp.
--Brylie

Attachments:
http://www.ruby-forum.com/attachment/3880/tinyJPsamplecorpus2.txt

-- 
Posted via http://www.ruby-forum.com/.