From: Yukihiro Matsumoto <matz@...>
Date: 2008-12-19T03:44:09+09:00
Subject: [ruby-core:20663] Re: 1.9 character encoding (was: encoding of symbols)

Hi,

In message "Re: [ruby-core:20637] Re: 1.9 character encoding (was: encoding of symbols)"
    on Thu, 18 Dec 2008 18:24:59 +0900, Brian Candler <B.Candler@pobox.com> writes:

|Concatenation raises interesting issues. For example:
|
|   data = "".force_encoding("UTF-8")
|   while chunk = file.read(1024)
|     data << chunk
|   end
|   # what is data.encoding ?

UTF-8 + ASCII-8BIT makes ASCII-8BIT.  Binary wins.

|irb(main):001:0> data = "".force_encoding("UTF-8")
|=> ""
|irb(main):002:0> data.encoding
|=> #<Encoding:UTF-8>
|irb(main):003:0> data << "\x61"
|=> "a"
|irb(main):004:0> data.encoding
|=> #<Encoding:UTF-8>
|irb(main):005:0> data << "\xc3"
|=> "a\xC3"
|irb(main):006:0> data << "\x9f"
|=> "a\xC3\x9F"
|irb(main):007:0> data.encoding
|=> #<Encoding:ASCII-8BIT>
|irb(main):008:0> data.force_encoding("UTF-8")
|=> "a�"

In this case, your source encoding (encoding of literals) seems to be
US-ASCII, but Ruby tokenizer automagically upgrade strings with 8bit
characters to ASCII-8BIT, so according to the above rule, the resulting
encoding should be ASCII-8BIT.  If your source encoding is, say UTF-8,
you will have UTF-8 result.

							matz.