From: Heesob Park <phasis@...>
Date: 2009-07-18T20:03:05+09:00
Subject: Re: Re-post: Parsing Japanese Language and Some Ruby Trivia

Hi,

2009/7/18 Brylie Oxley <brylie@gnumedia.org>:
> Hi,
> I have a similar task. I have attached a text file with Japanese
> characters. Some of the words are fake and made to look similar to
> actually occurring words, as an experiment in linguistics.
>
> Essentially what we are trying to accomplish is to get a count of every
> time one of the hiragana characters occurs adjacent to another hiragana
> character (including itself) within the context of a word.  The majority
> of the data in the file is frequency counts for the word over the course
> of several years, the numbers can be ignored for the purposes of this
> count.  At this point we are only concerned with the co-occurance of
> hiragana characters, but katakana and kanji may eventually be useful.
>
> I currently have Ruby 1.8.7 installed and when I paste the characters
> into irb they print out as bytes and I am not sure where to begin
> figuring out how to write an effective regexp.


According to your attachment, I guess you want to handle Shift_JIS encoded text.

Here are some clues.

str.scan(/\w+/s)                    # => scan for word including Japanese
str.scan(/./s)                      # => scan for character including Japanese
str.scan(/\x82[\x9f-\xf1]/)         # => scan for one hiragana character
str.scan(/\x83[\x40-\x96]/)         # => scan for one katakana character
str.scan(/[\x88-\xee][\x40-\xfc]/)  # => scan for one kanji character (roughly)
str.scan(/(?:\x82[\x9f-\xf1])+/)    # => scan for two or more hiragana
characters

Refer to
http://blog.grayproductions.net/articles/bytes_and_characters_in_ruby_18
http://www.rikai.com/library/kanjitables/kanji_codes.sjis.shtml

Regards,

Park Heesob