From: Michal Suchanek Date: 2006-06-18T06:55:48+09:00 Subject: Re: Unicode roadmap? On 6/17/06, Stefan Lang wrote: > On Saturday 17 June 2006 16:16, Austin Ziegler wrote: > > On 6/17/06, Stefan Lang wrote: > > > Full ACK. Ruby programs shouldn't need to care about the > > > *internal* string encoding. External string data is treated as > > > a sequence of bytes and is converted to Ruby strings through > > > an encoding API. > > > > This is incorrect. *Most* Ruby programs won't need to care about > > the internal string encoding. Experience suggests, however, that it > > is *most*. Definitely not all. > > As long as one treats a character string as a character > string, the internal encoding is irrelevant, and as soon as a No, it is not. First for reasons of efficiency. If an application is going to perform lots of slicing and poking on strings it will want some encoding that is suiatble for that such as UTF-32. If an application runs on system with little memory it will want space-efficient encoding (ie UTF-8 or UTF-16 for Asian languages). And if an appliaction runs on system that uses some legacy codepage it can read, write, and process all strings in that codepage. And in JRuby it will be useful to convert strings to UTF-16 so that the native Java functions can be used for manipulation. Second, not all characters are equal. If you lived in world where everything was Unicode you would be fine. But it is not so. Unicode is suboptimal for encoding CJK characters. So some people might want to use another encoding for their texts (iirc TRON mentioned earlier is one of such encodings). In your model you can modify Ruby to use strings composed of TRON characters instead of Unicode characters. But how would Unicode Ruby and TRON Ruby exchange strings? And how would you write an application that handles _both_ TRON and Unicode? (I suspect TRON would not be much good ie for Runic script) Such appliaction has to be written very carefully because neither character set would be subset of the other so it is not possible converting strings forth and back without thinking. But in your model such application is not possible at all. > decision for an internal string encoding is made, every > programmer can read in the docs "Ruby internally encodes > strings using the XYZ encoding". > > [...] > > [...] > > > 2) The File class/IO module as of current Ruby just gets > > > additional methods for binary IO (through ByteBuffers) and > > > an encoding attribute. The methods that do binary IO don't > > > need to care about the encoding attribute. > > > > > > I think 1) is cleaner. > > > > I think neither is necessary and both would be a mistake. It is, as > > I indicated to Juergen, sometimes *impossible* to determine the > > encoding to be used for an IO until you have some data from the IO > > already. > > That is easy to handle with the proposed scheme: Read as much > as you need with the binary interface until you know the > encoding and then do the conversion of the byte buffer to > string. For file input, you can close the file when you have > determined the encoding and reopen it using the "normal" > (character oriented) interface. Why reopening or converting if you can simply tag a string that you had to read anyway? > > Or do you mean Ruby should determine the encoding > automatically? IMO, that would be bad magic and error-prone. No. But if you read part of html/xml document before the encoding was specified there is no reason why that part hes to be converted or reread. You apparently got it right if you were able to determine the encoding from what you read. > > [...] > > > If the strings are represented as a sequence of Unicode > > > codepoints, it is possible for external libraries to implement > > > more advanced Unicode operations. > > > > This would be true regardless of the encoding. > > But a conversion from [insert arbitrary encoding here] to > unicode codepoints would be needed. That will be needed anyway. You cannot expect all libraries to use the arbitrary encoding you chose for Ruby strings. But if you can choose the encoding of your strings there is nothing stopping you from converting your strings so that they best suit your library of choice. > > > > > * There is only one internal string (where string means a > > > string of characters) representation. String operations > > > don't need to be written for different encodings. > > > > This is still (mostly) correct under the m17n String proposal. > > How does the regular expression engine work then? And all > String methods that have to combine two or more strings in > some way? If they are both subset of Unicode I see no problem with converting both to Unicode. If they are incompatible things may break. But that is because of real incompatibility, not because of some restriction of the approach. > > [...] > > > * Separation of concerns. I always found it strange that most > > > dynamic languages simply mix handling of character and arbitrary > > > binary data (just think of pack/unpack). > > > > The separation makes things harder most of the time. > > Why? In which cases? Such as when you have to read sthe start of a HTML page as ByteBuffer and then convert it to String once you determine the encoding. Especially if string operations do not exist on the ByteBuffer to allow parsing it. > > > > If Unicode is choosen as character set, there is the question > > > which encoding to use internally. UTF-32 would be a good choice > > > with regards to simplicity in implementation, since each > > > codepoint takes a fixed number of bytes. Consider indexing of > > > Strings: > > > > Yes, but this would be very hard on memory requirements. There are > > people who are trying to get Ruby to fit into small-memory > > environments. This would destroy any chance of that. > > I can hardly believe that. There is still the binary IO > interface and ByteString that I proposed. And I still think > that the memory used for pure character data is a small > fraction of the overall memory consumption of typical Ruby > programs. It depends on the program. For programs that do only text processing the portion of memory taken by text may be large. Michal