From: Izidor Jerebic <ij.rubylist@...>
Date: 2006-06-26T05:45:21+09:00
Subject: Re: Unicode roadmap?


On 25.6.2006, at 21:12, Austin Ziegler wrote:

> On 6/25/06, Izidor Jerebic <ij.rubylist@gmail.com> wrote:
>> If Ruby wants to move forward, it needs transparent String support  
>> and
>> hopefully separation of String and ByteArray, since this un-
>> separation brought us code which is mostly wrong (currently most of
>> existing Ruby code breaks if string encoding is honoured, as can be
>> seen from experience of brave people who modified String class).
>
> This is an incorrect and unsupportable statement. It is completely
> unnecessary to separate unencoded (e.g., binary) String support into
> String and ByteArray.

Well, if it is a byte array, it is not a String (an array of  
characters), is it?

If Ruby would have RegEx operations on byte arrays, there would be no  
need for untyped quasi String. API that has two incompatible things  
as one class is just plain ugly and wrong.

Reading jpeg image in a String is totally wrong.  You need bytes. You  
get characters, but they aren't really characters, they are bytes.  
Until something happens (maybe) and they are characters (maybe), or  
they are not (maybe). img_var[5] is what? 6th byte? 9th 2 bytes if  
encoding is utf8? What exactly? This is a clear API? There is no need  
for bytes masquerading as Strings. None. This practice just confuses  
the writer and the reader of the code. You need either bytes or  
Strings. Never both in the same variable. They are semantically  
totally different. At least they should be (we would not have  
problems if people would honour this distinction).

>
> Please don't try to assume that the problem is this completley
> unnecessary division. The problem is that existing strings are
> completely unencoded and have no way of being flagged with an encoding
> that is supported in any way across all of Ruby.

The problem is exactly this: the separation between bytes and  
characters. This is the general problem we have and discuss right  
now. API should help us solve the problem.

And you apparently missed all the attempts to extend String (also  
with encodings a la 1.9) that failed because of existing software,  
not because of Ruby.
>
> Ruby does not need a String with an internal representation in  
> Unicode;

Nobody says at this point of conversation that we need internal  
representation in unicode for all strings. We just want to avoid  
thinking about ANY encoding. We have other things to do. So having a  
transparent conversions between compatible encodings is a must.

> Ruby does not need a separate byte vector. An unencoded string can be
> treated as a byte vector with no problems.
> ; if it is determined to have
> textual meaning, it can be tagged with an encoding very simply

It can be, but it is not and will not be. Do you read emails? The  
problem is that people do not do things like that. And then other  
people have problems. If all the code you run is yours, then you are  
right. For many people that is not true.

> . There are times when the
> encoding is *not* best treated in Unicode, especially if there are
> potential conversion errors.

Why do you keep on about this?

Once again - WE DO NOT CARE WHAT ENCODING IS THERE. We just want the  
string operations to work without any extra programming work when  
operands have compatible encodings.

As written very well by Lugovoi Nikolai:

> What I, as Ruby user, wish for Unicode/M17N support:
> 1) reliability and consistency:
>  a) String should be abstraction for character sequence,
>  b) String methods shouldn't allow me to garble internal  
> representation;
>  c) treating String as byte sequence is handy, but must be explict  
> stated.
> 2) coding comfort:
>  a) no need to care what encodings have strings while working with  
> them;
>  b) no need to care what encodings have strings returned from third- 
> party code;
>  c) using explicit stated conversion options for external IO.
> 3) on Unicode and i18n : at least to have a set of classes for
> Unicode-specific tasks (collation, normalization, string search,
> locale-aware formatting etc.) that would efficiently work with Ruby
> strings.

Me too, please.


izidor