From: Michael Selig Date: 2008-09-17T13:21:00+09:00 Subject: [ruby-core:18647] Re: Character encodings - a radical suggestion On Wed, 17 Sep 2008 12:51:14 +1000, James Gray wrote: > Can we do: > > ['"', val, '"'].map { |s| s.encode("UTF-16BE") }.join > > ? Yeah, that seems to work. It sucks, but it works. Yep, it sure sucks! I have been doing some more thinking about these ongoing issues.... Using Ruby SHOULD be making our lives easier, not harder. Other languages like Python have taken an easier route to m17n - represent all strings internally as unicode codepoints. Then there should never be a need to check encoding compatibility, right? I am not saying that this is a perfect solution either, by the way. But having to work around this "Encoding Compatibility Error" all the time is just a pain for apps which need to work in different countries with different locales. Unfortunately it is leading me towards the path of having to transcode everything to UTF-8, even though in 99% of cases all the data IS going to be compatible and be in the user's locale. I don't want so much of my time taken up, and be forced to write ugly code to take care of the remaining 1%. Maybe the problem is that Ruby is being too generous supporting all these different encodings internally! That was one reason why I raised the idea of removing UTF-16 & 32 support - at least that way I know that the ASCII strings from my program can work with any user data. But then the further problem: What if you need to work with (or at least take into account the possibility of) 2 or more non-ascii (but ascii compatible) encodings (eg: the user's locale & UTF-8)? What may solve this issue is if Ruby itself would automatically encode incompatible strings in a compatible encoding (UTF-8 I guess). The only time you should then get "Encoding Compatibility Errors" is when writing data to a file or network stream in a certain encoding and a character cannot be represented. That's it. Just a thought... Mike