From: Vincent Isambart <vincent.isambart@...>
Date: 2008-01-01T17:19:48+09:00
Subject: Re: multibyte strings & bucket-of-bytes efficiency under 1.9.0


>> Well, it's actually not so difficult, and it will be needed.
>> We can't label a String as e.g. UTF-16, and claim that
>> UTF-16 is some kind of ASCII-compatible encoding.
>
> I'm not sure when matz introduce UTF-16.  I hope it is not
> just before a release.  At the time, dereferences of char*
> should be examined.

Does matz really intend to support UTF-16 in 1.9? I'm pretty sure I  
heard matz talking about only supporting directly encodings for which  
pure ASCII strings have the same representation as in ASCII, but maybe  
he changed his mind since...?

I know JRuby should support it UTF-16 strings natively as the Java  
world is UTF-16. So it would be probably better to have all  
implementations support it natively.
But in the C Ruby it will require some extensive modification of the  
string implementation and would not yield much benefits as UTF-16 ->  
UTF-8 conversion does not lose anything. For example concatenation of  
two strings, whatever their encoding is, works if the strings are  
ASCII-only because in all the supported encodings ASCII characters are  
represented the same way. But if one of the strings were UTF-16 and  
not the other it would require a conversion in one way or the other,  
and this be more or less contrary to the idea that concatenation is a  
simple operation that doesn't do any conversion (after all that's why  
there are exceptions when you try to concatenate for example UTF-8 and  
ASCII-8BIT). And yes the same problem would occur in JRuby. You could  
always forbid the concatenation of an UTF-16 with a non UTF-16 string  
but it would make UTF-16 behave differently from all other encodings...