From: Charles O Nutter Date: 2006-06-26T05:38:46+09:00 Subject: Re: Unicode roadmap? ------=_Part_36927_28170139.1151267922685 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline On 6/25/06, Austin Ziegler wrote: > > Ruby does not need a String with an internal representation in Unicode; > Ruby does not need a separate byte vector. An unencoded string can be > treated as a byte vector with no problems; if it is determined to have > textual meaning, it can be tagged with an encoding very simply and from > that point be treated as a meaningful string. There are times when the > encoding is *not* best treated in Unicode, especially if there are > potential conversion errors. > When is a ByteArray not a ByteArray? When is a String not a String? Is it correct to mingle the two concepts perpetually, when they each have fairly specific definitions? My problem with continuing to treat String as a byte vector is that it forces two somewhat incompatible concepts on the same class and the same methods. If you can use a String as both a byte vector and as a sequence of characters by calling the same methods, then setting or clearing encoding suddenly has the side-effect of changing how elements of String are to be treated. If you are providing separate methods for working with bytes as opposed to working with characters, then you are already splitting the two concepts. (As an aside, does it make sense that I read from a binary file into a String? Can I reliably assume that binary content in a String should be logically manipulable as text strings are? Should my binary String work anywhere and everywhere a text-based String does? I would think that binary content neither walks nor quacks like a String.) By your definition, a String can be treated as a ByteArray so long as its internal string does not have an encoding. What do I use if I want to have an encoding and still use byte vector semantics? It is appropriate that a String is no longer usable as a ByteArray as a result of changing some state? If there exists any state where String cannot be logically treated as a byte array, then String != ByteArray in the general case either. The encoding of a String's internal representation should not dictate the outward behavior of the String. If, however, you completely separate the two concepts, there's no dichotomy. In that case, a String deals with characters, and you do not have guarantees about byte-boundaries or indexed elements. You only have guarantees about characters, as it should be. Simultaneously, ByteArray would allow you to always work with a vector (array) of bytes, regardless of what those bytes contain. I'll end it off saying this: I think it's a no-brainer that for dealing with streams of bytes, there should be a non-string byte vector class. If folks are insistent on keeping them the same class, you can't logically continue to call it a String and have it fulfill the dual purposes of byte vector and character vector at the same time. If you plan to provide methods for supporting both behaviors, you're putting two distinct behaviors into the same type. I understand the unwillingness to move away from String as a byte vector, but with multibyte support coming you really can't have String == ByteArray without causing problems somewhere. They simply don't have the same behavior, and trying to pretend they do is asking for trouble. -- Charles Oliver Nutter @ headius.blogspot.com JRuby Developer @ www.jruby.org Application Architect @ www.ventera.com ------=_Part_36927_28170139.1151267922685--