From: Austin Ziegler <halostatue@...>
Date: 2006-06-18T07:15:27+09:00
Subject: Re: Unicode roadmap?

On 6/17/06, Stefan Lang <langstefan@gmx.at> wrote:
> On Saturday 17 June 2006 16:16, Austin Ziegler wrote:
>> On 6/17/06, Stefan Lang <langstefan@gmx.at> wrote:
>>> Full ACK. Ruby programs shouldn't need to care about the *internal*
>>> string encoding. External string data is treated as a sequence of
>>> bytes and is converted to Ruby strings through an encoding API.
>> This is incorrect. *Most* Ruby programs won't need to care about the
>> internal string encoding. Experience suggests, however, that it is
>> *most*. Definitely not all.
> As long as one treats a character string as a character string, the
> internal encoding is irrelevant, and as soon as a decision for an
> internal string encoding is made, every programmer can read in the
> docs "Ruby internally encodes strings using the XYZ encoding".

And I'm saying that it's a mistake to do that (standardize on a single
encoding). Every programmer will instead be able to read:

  "Ruby supports encoded strings in a variety of encodings. The
  default behaviour for all strings is XYZ, but this can be
  changed and individual strings may be recoded for performance
  or compatibility reasons."

Language and character encodings are hard. Hiding that fact is a
mistake. That doesn't mean we have to make the APIs difficult, but that
we aren't going to be buzzworded into compliance, either.

> [...]
>> Unnecessarily complex and inflexible. Before you go too much further,
>> I *really* suggest that you look in the archives and Google to find
>> more about Matz's m17n String proposal. It's a really good one, as it
>> allows developers (both pure Ruby and extension) to choose what is
>> appropriate with the ability to transparently convert as well.
> I couldn't find much (in English, I don't understand Japanese), do you
> have a link at hand?

I do not. I've been reading about this, talking about this, and
discussing it with Matz for the last two years or so, and I've been
dealing with Unicode and other character encoding issues extensively at
work. However, the gist of it is that every String is still a byte
vector. Each string will also have an encoding flag. Substrings of a
single character width will always return the String required for the
*character*. The supported encodings will probably start with UTF-8,
UTF-16, various ISO-8859-* encodings, EUC-JP, SJIS, and other Asian
encodings.

> [...]
>> I think neither [IO change] is necessary and both would be a mistake.
>> It is, as I indicated to Juergen, sometimes *impossible* to determine
>> the encoding to be used for an IO until you have some data from the
>> IO already.
> That is easy to handle with the proposed scheme: Read as much as you
> need with the binary interface until you know the encoding and then do
> the conversion of the byte buffer to string. For file input, you can
> close the file when you have determined the encoding and reopen it
> using the "normal" (character oriented) interface.
>
> Or do you mean Ruby should determine the encoding automatically? IMO,
> that would be bad magic and error-prone.

I mean that what you're suggesting *exposes* problems with encoding
stuff extensively and unnecessarily. I certainly wouldn't want to
program in it if the API involved were as stupid as you're suggesting it
should be.


> [...]
>>> If the strings are represented as a sequence of Unicode codepoints,
>>> it is possible for external libraries to implement more advanced
>>> Unicode operations.
>> This would be true regardless of the encoding.
> But a conversion from [insert arbitrary encoding here] to unicode
> codepoints would be needed.

Why? What if the library that I'm interfacing with requires EUC-JP?
Sorry, but Unicode is *not necessarily* the right answer.

>>> Since IMO a new "character" class would be overkill, I propose that
>>> the String class provides codepoint-wise iteration (and indexing) by
>>> representing a codepoint as a Fixnum. AFAIK a Fixnum consists of 31
>>> bits on a 32 bit machine, which is enough to represent the whole
>>> range of unicode codepoints.
>> This does not match what Matz will be doing.
>>
>>   str = "Fran\303\247ais"
>>   str[5] # -> "\303\247"
>>
>> This is better than doing a Fixnum representation. It is character
>> iteration, but each character is, itself, a String.
> I wouldn't mind additionally having:
>
>     str.codepoint_at(5)     => a Fixnum

Since Ruby isn't *only* using Unicode, this isn't necessarily going to
be possible or meaningful.

> [...]
>>> * There is only one internal string (where string means a
>>>   string of characters) representation. String operations
>>>   don't need to be written for different encodings.
>> This is still (mostly) correct under the m17n String proposal.
> How does the regular expression engine work then? And all
> String methods that have to combine two or more strings in
> some way?

Matz will have that figured and detailed before he starts writing it.

> [...]
>>> * Separation of concerns. I always found it strange that most
>>> dynamic languages simply mix handling of character and arbitrary
>>> binary data (just think of pack/unpack).
>> The separation makes things harder most of the time.
> Why? In which cases?

In *reality*, the separation is not nearly as clean as people who
advocate such separations would like to pretend. It's less of a problem
in dynamic languages like Ruby, but it's also far less necessary in
dynamic languages like Ruby. I have found it far more useful to not have
to care whether I'm reading a binary or string value. I despise dealing
with C++ and Java where I am forced to care because of stupid API
design.

> [...]
>>> It seems that the main argument against using Unicode strings in
>>> Ruby is because Unicode doesn't work well for eastern countries.
>>> Perhaps there is another character set that works better that we
>>> could use instead of Unicode. The important point here is that there
>>> is only *one* representation of character data Ruby.
>> This is a mistake.
> OK, Unicode was enough for me until now, but I see that Unicode is not
> enough for everyone.

Thank you. Unicode needs to -- will -- work *very* well. I know enough
about Unicode handling to make sure that what I deal with *will*. But I
have come to believe that choosing a single encoding as your String
representation is a mistake, even if it means making your job harder by
defining and implementing rules for mixed-encoding handling.

>>> If Unicode is choosen as character set, there is the question which
>>> encoding to use internally. UTF-32 would be a good choice with
>>> regards to simplicity in implementation, since each codepoint takes
>>> a fixed number of bytes. Consider indexing of Strings:
>> Yes, but this would be very hard on memory requirements. There are
>> people who are trying to get Ruby to fit into small-memory
>> environments. This would destroy any chance of that.
> I can hardly believe that. There is still the binary IO interface and
> ByteString that I proposed. And I still think that the memory used for
> pure character data is a small fraction of the overall memory
> consumption of typical Ruby programs.

I can believe it; it's very domain and program specific, but you've just
proposed multiplying the memory usage of that amount of space by four.
(Rails would suffer terribly under your proposal to use UTF-32.)

-austin
-- 
Austin Ziegler * halostatue@gmail.com * http://www.halostatue.ca/
               * austin@halostatue.ca * http://www.halostatue.ca/feed/
               * austin@zieglers.ca