From: Austin Ziegler Date: 2006-06-17T23:16:29+09:00 Subject: Re: Unicode roadmap? On 6/17/06, Stefan Lang wrote: > Full ACK. Ruby programs shouldn't need to care about the > *internal* string encoding. External string data is treated as > a sequence of bytes and is converted to Ruby strings through > an encoding API. This is incorrect. *Most* Ruby programs won't need to care about the internal string encoding. Experience suggests, however, that it is *most*. Definitely not all. > I think the encoding/decoding API should be separated from the > String class. IMO, the most important change is to strictly > differentiate between arbitrary binary data and character > data. Character data is represented by an instance of the > String class. > > I propose adding a new core class, maybe call it ByteString > (or ByteBuffer, or Buffer, whatever) to handle strings of > bytes. > > Given a specific encoding, the encoding API converts > ByteStrings to Strings and vice versa. > > This could look like: > > my_character_str = Encoding::UTF8.encode(my_byte_buffer) > buffer = Encoding::UTF8.decode(my_character_str) Unnecessarily complex and inflexible. Before you go too much further, I *really* suggest that you look in the archives and Google to find more about Matz's m17n String proposal. It's a really good one, as it allows developers (both pure Ruby and extension) to choose what is appropriate with the ability to transparently convert as well. >> 4. IO instances are associated with a (modifyable) encoding. For >> stdin, stdout this can be derived from the locale settings. >> String-IO operations work as expected. > > I propose one of: > > 1) A low level IO API that reads/writes ByteBuffers. String IO > can be implemented on top of this byte-oriented API. [...] > 2) The File class/IO module as of current Ruby just gets > additional methods for binary IO (through ByteBuffers) and > an encoding attribute. The methods that do binary IO don't > need to care about the encoding attribute. > > I think 1) is cleaner. I think neither is necessary and both would be a mistake. It is, as I indicated to Juergen, sometimes *impossible* to determine the encoding to be used for an IO until you have some data from the IO already. >> 5. Since the String class is quite smart already, it can implement >> generally useful and hard (in the domain of Unicode) operations like >> case folding, sorting, comparing etc. > If the strings are represented as a sequence of Unicode codepoints, it > is possible for external libraries to implement more advanced Unicode > operations. This would be true regardless of the encoding. > Since IMO a new "character" class would be overkill, I propose that > the String class provides codepoint-wise iteration (and indexing) by > representing a codepoint as a Fixnum. AFAIK a Fixnum consists of 31 > bits on a 32 bit machine, which is enough to represent the whole range > of unicode codepoints. This does not match what Matz will be doing. str = "Fran\303\247ais" str[5] # -> "\303\247" This is better than doing a Fixnum representation. It is character iteration, but each character is, itself, a String. >> 7. This approach leaves open the possibility of String subclasses >> implementing different internal encodings for performance/space >> tradeoff reasons which work transparently together (a bit like >> FixInt and BigInt). > I think providing different internal String representations > would be too much work, especially for maintenance in the long > run. If you're depending on classes to do that, especially given that Ruby's String, Array, and Hash classes don't inherit well, you're right. > The advantages of this proposal over the current situation and > tagging a string with an encoding are: The problem, of course, is that this proposal -- and your take on it -- don't account for the m17n String that Matz has planned. The current situation is a mess. But the current situation is *not* what is planned. I've had to do some encoding work for work in the last two years, and while I *prefer* a UTF-8/UTF-16 internal representation, I also know that's *impossible* in some situations and you have to be flexible. I also know that POSIX handles this situation worse than any other setup. With the work that I've done on this, Matz is *right* about this, and the people claiming that Unicode is the Only Way ... are wrong. In an ideal world, Unicode would be the correct and only way. In the real world, however, it's a lot messier, and Ruby has to be aware of that. We can *still* make it as easy as possible for the common case (which will be UTF-8 encoding data and filenames). But we shouldn't make the mistake of assuming that the common case is all that Ruby should handle. > * There is only one internal string (where string means a > string of characters) representation. String operations > don't need to be written for different encodings. This is still (mostly) correct under the m17n String proposal. > * No need for $KCODE. This is true under the m17n String. > * Higher abstraction. This is true under the m17n String. > * Separation of concerns. I always found it strange that most dynamic > languages simply mix handling of character and arbitrary binary data > (just think of pack/unpack). The separation makes things harder most of the time. > * Reading of character data in one encoding and representing it in > other encoding(s) would be easy. This is true under the m17n String. > It seems that the main argument against using Unicode strings in Ruby > is because Unicode doesn't work well for eastern countries. Perhaps > there is another character set that works better that we could use > instead of Unicode. The important point here is that there is only > *one* representation of character data Ruby. This is a mistake. > If Unicode is choosen as character set, there is the question which > encoding to use internally. UTF-32 would be a good choice with regards > to simplicity in implementation, since each codepoint takes a fixed > number of bytes. Consider indexing of Strings: Yes, but this would be very hard on memory requirements. There are people who are trying to get Ruby to fit into small-memory environments. This would destroy any chance of that. [...] > Thank you for reading so far. Just in case Matz decides to implement > something similar to this proposal, I am willing to help with Ruby > development (although I don't know much about Ruby's internals and not > too much about Unicode either). I would suggest that you look for discussions about m17n Strings in Ruby. Matz has this one right. > I do not have a CS degree and I'm not a Unicode expert, so perhaps the > proposal is garbage, in this case please tell me what is wrong about > it or why it is not realistic to implement it. I don't have a CS degree either, but I have been in the business for a *long* time and I've been immersed in Unicode and encoding issues for the last two years. If everyone used Unicode -- and POSIX weren't stupid -- your proposal would be much more realistic. I *agree* that Ruby should encourage the use of Unicode as much as is practical. But it also shouldn't tie our hands like other programming languages do. -austin -- Austin Ziegler * halostatue@gmail.com * http://www.halostatue.ca/ * austin@halostatue.ca * http://www.halostatue.ca/feed/ * austin@zieglers.ca