From: Brian Candler <B.Candler@...>
Date: 2008-12-18T18:24:59+09:00
Subject: [ruby-core:20637] Re: 1.9 character encoding (was: encoding of symbols)

On Thu, Dec 18, 2008 at 08:43:45AM +0900, danielcavanagh@aanet.com.au wrote:
> concatenation could be extended too. if one string is a superset of the
> other then no actual conversion needs to be done

Concatenation raises interesting issues. For example:

   data = "".force_encoding("UTF-8")
   while chunk = file.read(1024)
     data << chunk
   end
   # what is data.encoding ?

Here each chunk of 1024 bytes may have split multibyte characters at start
or end. However it's OK to concatenate them, and as long as the file is read
to the end, the result would be valid UTF-8.

Ruby's current behaviour is to do the concatenation bytewise, but downgrades
the encoding to binary when concatenating binary onto the end of UTF-8 (and
File#read returns binary)

irb(main):001:0> data = "".force_encoding("UTF-8")
=> ""
irb(main):002:0> data.encoding
=> #<Encoding:UTF-8>
irb(main):003:0> data << "\x61"
=> "a"
irb(main):004:0> data.encoding
=> #<Encoding:UTF-8>
irb(main):005:0> data << "\xc3"
=> "a\xC3"
irb(main):006:0> data << "\x9f"
=> "a\xC3\x9F"
irb(main):007:0> data.encoding
=> #<Encoding:ASCII-8BIT>
irb(main):008:0> data.force_encoding("UTF-8")
=> "a�"