From: Austin Ziegler Date: 2006-06-17T22:52:24+09:00 Subject: Re: Unicode roadmap? On 6/17/06, Juergen Strobel wrote: > I empathically agree. I'll even repeat and propose a new Plan for > Unicode Strings in Ruby 2.0 in 10 points: > > 1. Strings should deal in characters (code points in Unicode) and not > in bytes, and the public interface should reflect this. Agree, mostly. Strings should have a way to indicate the buffer size of the String. > 2. Strings should neither have an internal encoding tag, nor an > external one via $KCODE. The internal encoding should be encapsulated > by the string class completely, except for a few related classes which > may opt to work with the gory details for performance reasons. > The internal encoding has to be decided, probably between UTF-8, > UTF-16, and UTF-32 by the String class implementor. Completely disagree. Matz has the right choice on this one. You can't think in just terms of a pure Ruby implementation -- you *must* think in terms of the Ruby/C interface for extensions as well. > 3. Whenever Strings are read or written to/from an external source, > their data needs to be converted. The String class encapsulates the > encoding framework, likely with additional helper Modules or Classes > per external encoding. Some methods take an optional encoding > parameter, like #char(index, encoding=:utf8), or > #to_ary(encoding=:utf8), which can be used as helper Class or Module > selector. Conversion should be possible at any time. An "external source" may be an extension that your Ruby program can't distinguish. Again, this point fails because your #2 is unacceptable. > 4. IO instances are associated with a (modifyable) encoding. For > stdin, stdout this can be derived from the locale settings. String-IO > operations work as expected. Agree, realising that the internal implementation of String must be completely different than you've suggested. It is also important to retain *raw* reading; a JPEG should not be interpreted as Unicode. > 5. Since the String class is quite smart already, it can implement > generally useful and hard (in the domain of Unicode) operations like > case folding, sorting, comparing etc. Agreed, but this would be expected regardless of the actual encoding of a String. > 6. More exotic operations can easily be provided by additional > libraries because of Ruby's open classes. Those operations may be > coded depending on on String's public interface for simplicissity, or > work with the internal representation directly for performance. Agreed. > 7. This approach leaves open the possibility of String subclasses > implementing different internal encodings for performance/space > tradeoff reasons which work transparently together (a bit like FixInt > and BigInt). Um. Disagree. Matz's proposed approach does this; yours does not. Yours, in fact, makes things *much* harder. > 8. Because Strings are tightly integrated into the language with the > source reader and are used pervasively, much of this cannot be > provided by add-on libraries, even with open classes. Therefore the > need to have it in Ruby's canonical String class. This will break some > old uses of String, but now is the right time for that. "Now" isn't; Ruby 2.0 is. Maybe Ruby 1.9.1. > 9. The String class does not worry over character representation > on-screen, the mapping to glyphs must be done by UI frameworks or the > terminal attached to stdout. The String class doesn't worry about that now. > 10. Be flexible. And little is more flexible than Matz's m17n String. > This approach has several advantages and a few disadvantages, and I'll > try to bring in some new angles to this now too: > > *Advantages* > > -POL, Encapsulation- > > All Strings behave exactly the same everywhere, are predictable, > and do the hard work for their users. Remember: POLS is not an acceptable reason for anything. Matz's m17n Strings would be predictable, too. a + b would be possible if and only if a and b are the same encoding or one of them is "raw" (which would mean that the other is treated as the defined encoding) *or* there is a built-in conversion for them. > -Cross Library Transparency- > No String user needs to worry which Strings to pass to a library, or > worry which Strings he will get from a library. With Web-facing > libraries like rails returning encoding-tagged Strings, you would be > likely to get Strings of all possible encodings otherwise, and isthe > String user prepared to deal with this properly? This is a *big* deal > IMNSHO. This will be true with m17n strings. However, your proposal does *not* work for Ruby/C interfaced items. Sorry. > -Limited Conversions- > > Encoding conversions are limited to the time Strings are created or > written or explicitly transformed to an external representation. This is a mistake. I may need to know the internal representation of a particular encoding of a String inside of a program. Trust me on this one: I *have* done some low-level encoding work. Additionally, even though I might have marked a network object as "UTF-8", I may not know whether it's *actually* UTF-8 or not until I get HTTP headers -- or worse, a tag. Assuming UTF-8 reading in today's world is doomed to failure. > -Correct String Operations- > Even basic String operations are very hard in the world of Unicode. If > we leave the String users to look at the encoding tags and sort it out > themselves, they are bound to make mistakes because they don't care, > don't know, or have no time. And these mistakes may be _security_ > _sensitive_, since most often credentials are represented as Strings > too. There already have been exploits related to Unicode. This is a misunderstanding on your part. Nothing about Matz's m17n Strings suggests that String users would have to look at the encoding tags. Merely that they *could*. I suspect that there will be pragma- like behaviours to enforce a particular internal representation at all times. > *Disadvantages* (with mitigating reasoning of course) > - String users need to learn that #byte_length(encoding=:utf8) >= > #size, but that's not too hard, and applies everywhere. Users do not > need to learn about an encoding tag, which is surely worse to handle > for them. True, but the encoding tag is not worse. Anyone who assumes that developers can ignore encoding at any time simply *doesn't* know about the level of problems that can be encountered. > - Strings cannot be used as simple byte buffers any more. Either use > an array of bytes, or an optimized ByteBuffer class. If you need > regular expresson support, RegExp can be extended for ByteBuffers or > even more. I see no reason for this. > - Some String operations may perform worse than might be expected from > a naive user, in both the time or space domain. But we do this so the > String user doesn't need to himself, and are problably better at it > than the user too. This is a wash. > - For very simple uses of String, there might be unneccessary > conversions. If a String is just to be passed through somewhere, > without inspecting or modifying it at all, in- and outwards conversion > will still take place. You could and should use a ByteBuffer to avoid > this. This is a wash. > - This ties Ruby's String to Unicode. A safe choice IMHO, or would we > really consider something else? Note that we don't commit to a > particular encoding of Unicode strongly. This is a wash. I think that it's better to leave the options open. After all, it *is* a hope of mine to have Ruby running on iSeries (AS/400) and *that* still uses EBCDIC. > - More work and time to implement. Some could call it over-engineered. > But it will save a lot of time and troubles when shit hits the fan and > users really do get unexpected foreign characters in their Strings. I > could offer help implementing it, although I have never looked at > ruby's source, C-extensions, or even done a lot of ruby programming > yet. I would call it the amount of work necessary. But the work needs to be done for a *variety* of encodings, and not just Unicode. *Especially* because of C extensions. > Close to the start of this discussion Matz asked what the problem with > current strings really was for western users. Somewhere later he > concluded case folding. I think it is more than that: we are lazy and > expect character handling to be always as easy as with 7 bit ASCII, or > as close as possible. Fixed 8-bit codepages worked quite fine most of > the time in this regard, and breakage was limited to special > characters only. > Now let's ask the question in reverse: are eastern programmers so used > to doing elaborate byte-stream to character handling by hand they > don't recognize how hard this is any more? Surely it is a target for > DRY if I ever saw one. Or are there actual problems not solveable this > way? I looked up the mentioned Han-Unification issue, and as far as I > understood this could be handled by future Unicode revisions > allocating more characters, outside of Ruby, but I don't see how it > requires our Strings to stay dumb byte buffers. No one has ever suggested that Ruby Strings stay byte buffers. However, blindly choosing Unicode *adds* unnecessary complexity to the situation. -austin -- Austin Ziegler * halostatue@gmail.com * http://www.halostatue.ca/ * austin@halostatue.ca * http://www.halostatue.ca/feed/ * austin@zieglers.ca