Re: The face of Unicode support in the future

Note: this message contains UTF-8 characters.

On Thu, 13 Jan 2005 07:56:22 +0900, Yukihiro Matsumoto
<[e-mail]> wrote:
> In message "Re: The face of Unicode support in the future" on Thu,
> 13 Jan 2005 01:35:36 +0900, Christian Neukirchen
> <[e-mail] > writes:
>| This sounds likely to result in duplicated efforts... Do it
>| pragmatically; I don't think it should be very hard to provide a
>| default Character class that people can "customize" by
>| subclassing or method redefinition.
> A character might be represented by either:

> * code point
> * sequence of code points
> * or even set of attributes, without any code point
> * or something totally different

> But never mind. I'm no expert. I just don't want to repeat the
> argument again in English.

I'm not going to claim to be an expert, but in my work I have had to
look this over extensively -- mostly from the perspective of needing
to support Unicode, but also from the perspective of dealing with
unknown code pages. I'm not really understanding the last two points
in your list (a set of attributes, without any code point; something
totally different).

In Unicode encodings, at least, a code point can be multiple bytes
(or multiple words in UTF-16 with surrogates) -- and a character can
be multiple code points for the combining characters (e.g., can be
represented as combining-' + e, but there are explicit rules for how
combining characters can be specified in order).

It's important to note that in filenames, at least, Windows will
represent most characters as composed (e.g.,  and the Mac will
represent most characters as decomposed (e.g., 'e). These are,
however, the same Character. So when working with Unicode strings,
If I have 
  a = "r駸um蘂
then when I do:
  a[1]
I expect to get a Character of "蘂. With this Character, I should be
able to extract the composed codepoint(s) as well as the decomposed
codepoint(s) -- there are regular transformations available in
Unicode for this matter.

I do not necessarily expect to be able to do these transformations
on normal Strings -- but I do expect to be able to do these
transformations on I18N/M17N Ruby with multiple String encoding
support. Whether it's as a Character class or something that I apply
to a String, I don't particularly care.
  
I fully expect that you'll have a sensible API for this when you get
it added, but I would like such standard handling -- and a standard
way of adding new handling -- in Ruby 2.0.

Have you looked at ICUC (IBM's International Components for Unicode
for C)? It uses a UTF-16 encoding internally, but it supports quite
a bit of what I'm talking about already.

-austin
-- 
Austin Ziegler * [e-mail]
               * Alternate: [e-mail]