From: markjreed@...
Date: 2005-11-19T01:47:18+09:00
Subject: Re: Investigating Ruby - key limitations ?

> Ruby's strings are -- in many ways -- what other languages might call a ByteVector.

And that's the problem.  A String should be composed of *characters*,
not bytes.  I shouldn't need to care what bytes are used to store them
unless I'm dealing with a binary file format.  The advantage of Unicode
is that it's a superset of almost all other encodings, with a globally
unique identifier number for each of the characters in its repertoire
(with room for almost 1.5 million of them).

I understand the objections to Han unification, but compromises had to
be made to fit everything into a single codespace of a reasonable size.
 The resulting need for metadata indicating language is really no
different than it is across the many languages which use the Roman
alphabet.  And there are already extra Han Zi characters outside the
unified set in the BMP; I wouldn't be surprised to see more national
variants recognized that way over time.

As far as storage space, it's rapidly becomin a non-issue.  Two bytes
per character just isn't that bad, and most Unicode apps use UTF-16 in
memory; UTF-16 is also the native on-disk representation in Mac OS X.
Heck, I wouldn't be surprised to see things starting to use UTF-32 in
the near future.  There is a lot of UTF-8 online, and it is biased
toward Western characters, but that's not necessarily the worst choice
for source code in a programming language whose keywords are all
composed entirely of ASCII characters.  But here's always SCSU or
BOCU-1 if you want more efficient encoding of Eastern languages.

> This is, in fact, far  *superior* to what Java and Python do -- which are limited to UTF-8
> string representations (AFAIK).

Not true of Java or Python.  Or Perl, for that matter; I don't know if
the Perl5 interpreter will read source code in anything other than
ASCII/Latin-1/UTF-8, but a Perl program can certainly read and write
strings in just about any encoding you can think of via the "use
encoding" pragma.  Including Shift-JIS, EUC-JP, Big5, or anything else
.. . . while the interface presented to the Perl programmer always uses
Unicode.  That's what I think Ruby should do as well.