From: Austin Ziegler Date: 2006-06-26T10:01:37+09:00 Subject: Re: Unicode roadmap? On 6/25/06, Izidor Jerebic wrote: > On 25.6.2006, at 21:12, Austin Ziegler wrote: >> On 6/25/06, Izidor Jerebic wrote: >>> If Ruby wants to move forward, it needs transparent String support >>> and hopefully separation of String and ByteArray, since this un- >>> separation brought us code which is mostly wrong (currently most of >>> existing Ruby code breaks if string encoding is honoured, as can be >>> seen from experience of brave people who modified String class). >> This is an incorrect and unsupportable statement. It is completely >> unnecessary to separate unencoded (e.g., binary) String support into >> String and ByteArray. > Well, if it is a byte array, it is not a String (an array of > characters), is it? It could be indistinguishable from such. Even a Unicode string is ultimately an array of bytes in memory. It just happens that there's a higher level abstraction that can be used to interpret that particular array of bytes. What you're asking for is rather like the difference between std::string and std::vector. They represent the same thing, but don't work the same. If you're going to have a String and ByteVector that work the same (except that the String also has the higher-level interpretation of characters), is it meaningfully a different object? I think not. Indeed, I think that having a separate object for these would increase the overall complexity and reduce the usability overall. >> Please don't try to assume that the problem is this completley >> unnecessary division. The problem is that existing strings are >> completely unencoded and have no way of being flagged with an >> encoding that is supported in any way across all of Ruby. > The problem is exactly this: the separation between bytes and > characters. This is the general problem we have and discuss right now. > API should help us solve the problem. > And you apparently missed all the attempts to extend String (also with > encodings a la 1.9) that failed because of existing software, not > because of Ruby. Excuse me? You don't know what you're talking about here. No existing version of Ruby has a String with encodings. Not even Ruby 1.9. Any extension which tries to do this *will fail* because there is no way to enforce this extension's semantics on all of Ruby and all extensions. Ruby 1.9 will be different because the m17n String will be a guaranteed behaviour in Ruby. The problem is not the separation between bytes and characters, but that there's no way *in Ruby* to distinguish between the two, at least not reliably. >> Ruby does not need a String with an internal representation in >> Unicode; > Nobody says at this point of conversation that we need internal > representation in unicode for all strings. We just want to avoid > thinking about ANY encoding. We have other things to do. So having a > transparent conversions between compatible encodings is a must. I think that you're confusing me with someone else. Most people who have advocated a separate ByteVector have been unable to articulate exactly what this would buy us, and most have also advocated an internal Unicode representation of Strings. I have been one of the ones who have advocated transparent conversions all along. Frankly, with coersion, it would be possible to upconvert to a compatible conversion between any encoding. >> Ruby does not need a separate byte vector. An unencoded string can be >> treated as a byte vector with no problems. ; if it is determined to >> have textual meaning, it can be tagged with an encoding very simply > It can be, but it is not and will not be. Do you read emails? The > problem is that people do not do things like that. And then other > people have problems. If all the code you run is yours, then you are > right. For many people that is not true. "Is not" is a useless term. OF COURSE IT ISN'T -- right now. In the future, with the m17n Strings, it could be -- and would be. And yes, I have read every single one of these emails about Unicode. Most of them have been ignorant of anything but their own narrow needs and clueless about good API design. >> There are times when the encoding is *not* best treated in Unicode, >> especially if there are potential conversion errors. > Why do you keep on about this? > > Once again - WE DO NOT CARE WHAT ENCODING IS THERE. We just want the > string operations to work without any extra programming work when > operands have compatible encodings. I suggest you look through the Unicode threads again. You'll find your statement is untrue. There are a lot of people who (foolishly) want Unicode to be the only internal representation of Strings in Ruby. > As written very well by Lugovoi Nikolai: >> What I, as Ruby user, wish for Unicode/M17N support: >> 1) reliability and consistency: >> a) String should be abstraction for character sequence, >> b) String methods shouldn't allow me to garble internal >> representation; >> c) treating String as byte sequence is handy, but must be explict >> stated. An unencoded -- raw -- String would be *only* interpretable as a byte sequence unless "recoded." Aside from that, everything said above would be true. >> 2) coding comfort: >> a) no need to care what encodings have strings while working with >> them; >> b) no need to care what encodings have strings returned from third- >> party code; >> c) using explicit stated conversion options for external IO. You'll always need to care, even if you're using Unicode. You can't *not* care and claim to be doing Unicode or m17n work. We can *reduce* those concerns, but you *CANNOT* be ignorant of this at any time. >> 3) on Unicode and i18n : at least to have a set of classes for >> Unicode-specific tasks (collation, normalization, string search, >> locale-aware formatting etc.) that would efficiently work with Ruby >> strings. > Me too, please. That would be useful. -austin -- Austin Ziegler * halostatue@gmail.com * http://www.halostatue.ca/ * austin@halostatue.ca * http://www.halostatue.ca/feed/ * austin@zieglers.ca