From: Brian Candler <B.Candler@...>
Date: 2008-12-18T18:59:57+09:00
Subject: [ruby-core:20638] Re: 1.9 character encoding (was: encoding of symbols)

On Thu, Dec 18, 2008 at 06:22:28PM +0900, Daniel Cavanagh wrote:
> and you're against early conversion (ie. at I/O)

That's what ruby-1.9 does out-of-the-box. e.g. if you set the external
encoding to ISO-8859-1, and the internal encoding to UTF-8, then your
program will see the source as if it were a stream of UTF-8.

> why do i have to use /u 
> with regexes to match against unicode strings.

Only in Ruby 1.8. Or do you have a counter-example? Here in irb19 from
1.9.1-preview2:

irb(main):001:0> foo = "a�b"
=> "a�b"
irb(main):002:0> foo =~ /b/
=> 2
irb(main):003:0> foo =~ /�/
=> 1

> why do i have to specify 
> the type of the regex at all?

Only in 1.8

> why do i have to add in checks before i 
> have to be sure i can concatenate properly?

Not sure what you mean. If you have decided to read in one string in UTF-8,
and a different string in Shift-JIS, and you really insist on concatenating
them, then clearly you'll end up with a binary muddle. But what else do you
expect? If you wrote

    s = str1 + str2

how does Ruby know whether s should take the encoding of str1, or str2?

But if you convert all your data to one encoding at input time, so internal
processing is consistent, then the problem goes away.

At least, this is how I understand it at the moment :-)

Brian.