From: Julian 'Julik' Tarkhanov Date: 2006-06-15T11:52:15+09:00 Subject: Re: Unicode roadmap? On 15-jun-2006, at 4:40, Austin Ziegler wrote: > On 6/14/06, Charles O Nutter wrote: >> I believe that Julik's way of solving the unicode problem (String#u >> providing access to a unicode helper) is very attractive. I have two >> questions related, for Julik and the rest of the peanut gallery: > >> 1. How does performance look with the unicode string add-on versus >> native >> strings (or as compared to icu4r, which is C-based)? >> 2. Is this the ideal way to support unicode strings in ruby? > > No. In fact, I believe that Matz has the right idea for M17N strings > in Ruby 2.0. The *reality* is that there's a *lot* of data out there > that isn't Unicode. It's very difficult for me to understand the implementation. What if we concat a Mojikyo string to a UTF8String? UnicodeDecodeError, ordinal not in range? I think Python folks proved that it's terrible (it is). Nothing is ideal. > I would suggest that JRuby could offer a JavaString that acts in every > way like a String except that it provides access to the native UTF-16 > implementation. Just what the ICU4R extension does. It's unusable to the point that you cannot concat a native string with a UString. To the point that you have to use special Regexp class for it. You end up having half of your Ruby script doing typecasting from one to the other. There is alot of data that isn't Unicode, indeed. Converted on input and converted on output if necessary - just as in any other case when the encoding of your system doesn't match your input or output. I don't know if it can be possible to have the "internal" encoding of a system switchable (seems to me this is what Matz wants) - then you can't safely refer to anything other than bytes. And then you get software that you can't use, because they had a different assumtpion than you had as to what encoding the user will be using. -- Julian 'Julik' Tarkhanov please send all personal mail to me at julik.nl