From: gwtmp01@...
Date: 2006-06-17T23:58:46+09:00
Subject: Re: Unicode roadmap?


On Jun 17, 2006, at 9:50 AM, Stefan Lang wrote:

> On Saturday 17 June 2006 13:08, Juergen Strobel wrote:
>> 2. Strings should neither have an internal encoding tag, nor an
>> external one via $KCODE. The internal encoding should be
>> encapsulated by the string class completely, except for a few
>> related classes which may opt to work with the gory details for
>> performance reasons. The internal encoding has to be decided,
>> probably between UTF-8, UTF-16, and UTF-32 by the String class
>> implementor.
>
> Full ACK. Ruby programs shouldn't need to care about the
> *internal* string encoding. External string data is treated as
> a sequence of bytes and is converted to Ruby strings through
> an encoding API.

I don't claim to be an Unicode export but shouldn't the goal be to  
have Ruby work with *any* text encoding on a per-string basis?  Why  
would you want to force all strings into Unicode for example in a  
context where you aren't using Unicode?  (The internal encoding has  
to be....).  And of course even in the Unicode world you have several  
different encodings (UTF-8, UTF-16, and so on).  Juergen, when you  
say 'internal encoding' are you talking about the text encoding of  
Ruby source code?

It seems to me that irrespective of any particular text encoding  
scheme you need clean support of a simple byte vector data structure  
completely unencumbered with any notion of text encoding or locale.   
Right now that is done by the String class, whose name I think  
certainly creates much confusion.  If the class had been called  
Vector and then had methods like:

	Vector#size		# size in bytes
	Vector#str_size 	# size in characters (encoding and locale considered)

I think this discussion would be clearer because it would be the  
behavior of the str* methods that would need to understand text  
encodings and/or locale settings while the underlying byte vector  
methods remained oblivious.  The #[] method is the most confusing  
since sometimes you want to extract bytes and sometimes you want to  
extract sub-strings (i.e consider the encoding).  One method, two  
interpretations, bad headache.

It seems that three distinct behaviors are being shoehorned (with  
good reason) into a single class framework (String):

	byte vector
	text encoding (encoded sequence of code points)
	locale	      (cultural interpretations of the encoded sequence of  
code points)

I'm just suggesting that these distinctions seem to be lost in much  
of this discussion, especially for folks (like myself) who have a  
practical interest in this but certainly aren't text-encoding gurus.


Gary Wright