From: MikkelFJ <mikkelfj-anti-spam@...>
Date: 2002-08-02T00:12:38+09:00
Subject: Re: Unicode in Ruby now?


"Curt Sampson" <cjs@cynic.net> wrote in message
news:Pine.NEB.4.44.0208011945110.539-100000@angelic.cynic.net...
> On Thu, 1 Aug 2002, Tobias Peters wrote:
>
> > ...
> > We also need rules how to combine strings with different encoding then.
> > concatenating two strings encoded in koi8-r and iso-8859-1,
respectively,
> > may only be possible when the result is encoded in some unicode
> > representation.
>
> Yeah. This is getting into complex nightmare city. That's why I'd prefer
> to have the basic system just work completely in Unicode. One could have
> a separate character system (character and string classes, byte-stream
> to char converters, etc.) to work with this tagged format if one wished.

But isn't this what matz suggest?
Each stream is tagged, that is the same as having different types. It's
basically just  a different way to store the type while having a lot of
common string operations.
The only major issues I see are: fixed size characters versus variable
length characters - for example UTF-8 versus UCS-4.
I think a string class for each type might make sense.

I think UTF-8 is universal enough to warrant its own string class, but for
fixed size formats, why should it matter what encoding is used? The
important thing is that the encoding is stored, possibly along with the byte
width of the character.

BTW: Unicode is not a fixed with format. It is an almost fixed with format -
but there are escape codes and options for future extensions. Hence UCS-4 is
a strategy with limited timespan. That's why I prefer UTF-8 - it recognizes
the variable length issue up front, yet usually takes up less space than
UCS-2 or UCS-4 except if you are from Asia in which case you probably want a
different encoding.

One other detail about UTF-8 - its really nice to work with as raw 8 bit
character. When writing a lexer for UTF-8 you can define anything above 127
to be a letter and A-Za-z to also be a letter. Then you can forget
everything about UTF-8 and still parse everything correctly including
indentifiers. UCS-2, UCS-4 lexers get bloated or impossible due to the huge
lookup.

For practical string processing, it is actually very seldom that you
actually need to index a character sequence. Normally you simply break the
string into pieces instead.
You could have a iterator object for the string class which does not care
about variable length issues. It cannot index, but it can remember a
position even after a string is changed.

Mikkel