From: matz@... (Yukihiro Matsumoto)
Date: 2001-03-04T23:45:16+09:00
Subject: [ruby-talk:11974] Re: text-processing with ext. characterset

Hi,

In message "[ruby-talk:11937] text-processing with ext. characterset"
    on 01/03/03, "Henning VON ROSEN" <hvrosen@world-online.no> writes:

|Questions:
|(1) unicode-support in Ruby? (Python got it from 1.6, but not so integrated
|into Regexes (not supporting word-boundaries)

As of Ruby 1.7, it supports only UTF-8, not Unicode (you know I
mean?).  I'm working on Ruby M17N, which enalbles a user to define
his/her own character encoding scheme using an extension.

You can fetch the prototype of Ruby M17N from CVS repositry, from
branch ruby_m17n.

For CVS access, See http://www.ruby-lang.org/~knu/cvsrepo-guide.txt

|(2) Scince Ruby is popular in Japan, where can i find something on trating
|Japanese?

Unicode is not yet popular in Japan.  We have our own multibyte
character encoding schemes.

|(3) I am a bit confused on whether or not to go for converting everything to
|16-bit unicode (Ruby?), or to stay in the utf-8 format.

Stay in UTF-8.  Since it is far easyer to handle ASCII compatible
encoding scheme.  Since USC2 is obsolete, 16bit Unicode (UTF-16) is no
longer fixed sized encoding.

|(4) If staying in utf-8; how difficult to add functionality, so that things
|like .upcase and sort does the right thing for various languages?

Using Ruby M17N, it's pretty easy (though it requires huge table).
Only thing you need is to define just 3 functions and register them.

|(5) I think integration to the in-built services of Ruby would allow a lot
|cleaner code. I mean: Why should non-english text-processing be more awkward
|than English? Considering Ruby's origin, I suppose Ruby to be easily
|extensible in this area...

Agreed.

|(6) Would it be time-inefficient to make the string-class utf-8 aware?
|(returning the "correct" length, not the physical, and so on... so that for
|example a c with circumflex, � or a kanji symbol, would be considered as of
|length 1?)

Yes and no.  Ruby M17N supports them.  It's little bit slower than
multibyte ignorant counterpart, but still in affordable performance.
It's buggy but proves the concept.  Ruby M17N will be merged into Ruby
1.7 in several months.

							matz.