From: Austin Ziegler Date: 2006-03-10T13:34:59+09:00 Subject: Re: unicode in ruby On 3/8/06, Richard Gyger wrote:> so, you guys are telling me a language developed since the year 2000> doesn't support unicode strings natively? in my opinion, that's a> pretty glaring problem. Please note that Ruby itself is ten years old. Unicode has only*recently* (the last three or four years, with the release of WindowsXP) become a major factor, especially in Japan. Unix support for Unicodeis still in the stone ages because of the nonsense that POSIX put onUnix ages ago. (When Unix filesystems can write UTF-16 as their nativefilename format, then we're going to be much better. That will, however,break some assumptions by really stupid programs.) I've been following what Matz has had to say and have recently donequite a bit of work with Unicode. The reality is that Unicode is hard,and there's cultural and other reasons for Ruby *not* to have Unicode(UTF-16 or UTF-8) strings by default. I think that Matz's plans for M17Nstrings is far superior to assuming Unicode by default. Basically, Ruby will have the capabilities to work with UTF-8, UTF-16,and probably the ISO-8859-* encodings natively, as well as the existingSJIS and EUC-JP support. I wouldn't be surprised if it also includesother EUC-* encodings. Essentially, you'll be able to do: s = "école" s.encoding # -> :raw (or something like that) s.encoding = :iso8859_1 # "école" s.encoding = :utf8 # "Ã(c)cole" s.capitalize! # "École" s.encoding = :iso8859_1 # "École" More than that, using the same string: s[0] # "É" s.encoding = :utf8 s[0] # "É" I've shown everything as a byte string here. The point is, though, thatgoing from the raw encoding -- which may be the default, or the defaultmay be able to be set -- shouldn't cause any byte conversions. I suspectthat Matz will have a different way to get at the underlying bytes, butthat's what will be happening for Ruby 2.0. The last indication I had seen suggested that M17N strings were closer,but not yet done. I'm looking forward to them. -austin--Austin Ziegler * halostatue@gmail.com * Alternate: austin@halostatue.ca