From: Brian Candler <B.Candler@...>
Date: 2008-12-15T20:32:48+09:00
Subject: [ruby-core:20567] Re: 1.9 character encoding (was: encoding of symbols)

On Mon, Dec 15, 2008 at 08:11:33PM +0900, Brian Candler wrote:
> irb(main):004:0> File.open("/bin/sh").gets.encoding
> => #<Encoding:UTF-8>

Sorry, that wasn't a good example. Using #read instead:

irb(main):001:0> File.open("/bin/sh").read(16).encoding
=> #<Encoding:ASCII-8BIT>
irb(main):002:0> File.open("/etc/passwd").read(16).encoding
=> #<Encoding:ASCII-8BIT>
irb(main):003:0> File.open("/bin/sh").gets.encoding
=> #<Encoding:UTF-8>
irb(main):004:0> File.open("/etc/passwd").gets.encoding
=> #<Encoding:UTF-8>

So I guess: the encoding declared in open is used for #gets, but #read
always returns ASCII-8BIT.

I see also that inline literals have variable encoding:

irb(main):029:0> "\x61\x62\x63".encoding
=> #<Encoding:US-ASCII>
irb(main):028:0> "\x61\xc3\x9f".encoding
=> #<Encoding:ASCII-8BIT>

From what I've heard so far, these will compare equal to data from my file
as long as US-ASCII and ASCII-8BIT are treated as 'compatible' encodings. It
still bothers me though :-)

#getc returns a single "character" (which can be more than one byte) and
tags it as UTF-8, although it will happily consume and return invalid UTF-8
too.

irb(main):025:0> File.open("tst","w") { |f| f.write "a\xc3" }
=> 2
irb(main):026:0> f = File.open("tst")
=> #<File:tst>
irb(main):027:0> 2.times { c = f.getc; puts "#{c.inspect} => #{c.encoding}" }
"a" => UTF-8
"\xC3" => UTF-8
=> 2

irb(main):033:0> File.open("tst","w") { |f| f.write "a\xc3\x9f" }
=> 3
irb(main):034:0> f = File.open("tst")
=> #<File:tst>
irb(main):035:0> 2.times { c = f.getc; puts "#{c.inspect} => #{c.encoding}" }
"a" => UTF-8
"�" => UTF-8
=> 2

Regards,

Brian.