From: Martin Duerst <duerst@...>
Date: 2008-01-01T19:59:12+09:00
Subject: Re: multibyte strings & bucket-of-bytes efficiency under 1.9.0

At 17:19 08/01/01, Vincent Isambart wrote:
>
>>> Well, it's actually not so difficult, and it will be needed.
>>> We can't label a String as e.g. UTF-16, and claim that
>>> UTF-16 is some kind of ASCII-compatible encoding.
>>
>> I'm not sure when matz introduce UTF-16.  I hope it is not
>> just before a release.  At the time, dereferences of char*
>> should be examined.

I don't understand the 'dereferences of char*' bit.
char* are pointers. Even pointers to UTF-16 strings
are just plain pointers, not NULL pointers.

Even now, I can write
$ ruby -e 'puts "abc\0def".length'
and I get 7. So I don't think we will have that many problems.

>Does matz really intend to support UTF-16 in 1.9? I'm pretty sure I  
>heard matz talking about only supporting directly encodings for which  
>pure ASCII strings have the same representation as in ASCII, but maybe  
>he changed his mind since...?

I don't think Matz changed his mind, but that's not what I talked
about. What I was talking is conversion to UTF-16, e.g. with
'abc'.encode('UTF-16'). We clearly know we want this, and I'll
implement it, probably in a couple weeks. But then, we need to
label the resulting string, i.e. we'll get
'abc'.encode('UTF-16').encoding => "UTF-16".
The question is whether UTF-16 will be derived from ASCII-8BIT
or not. I was arguing that deriving it from ASCII-8BIT isn't
appropriate.

>I know JRuby should support it UTF-16 strings natively as the Java  
>world is UTF-16. So it would be probably better to have all  
>implementations support it natively.
>But in the C Ruby it will require some extensive modification of the  
>string implementation and would not yield much benefits as UTF-16 ->  
>UTF-8 conversion does not lose anything. For example concatenation of  
>two strings, whatever their encoding is, works if the strings are  
>ASCII-only because in all the supported encodings ASCII characters are  
>represented the same way. But if one of the strings were UTF-16 and  
>not the other it would require a conversion in one way or the other,  
>and this be more or less contrary to the idea that concatenation is a  
>simple operation that doesn't do any conversion (after all that's why  
>there are exceptions when you try to concatenate for example UTF-8 and  
>ASCII-8BIT). And yes the same problem would occur in JRuby. You could  
>always forbid the concatenation of an UTF-16 with a non UTF-16 string  
>but it would make UTF-16 behave differently from all other encodings...

The machinery for doing this is already there. Encodings can be
'ASCII-compatible' or not.

Regards,   Martin.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst@it.aoyama.ac.jp