From: Nathan Beyer <nbeyer@...>
Date: 2012-09-22T12:50:33+09:00
Subject: Re: Encoding question

On Fri, Sep 21, 2012 at 8:27 AM, Brian Candler <lists@ruby-forum.com> wrote:
> Nathan Beyer wrote in post #1076883:
>>> *No* transcoding takes place in the above example, regardless of the
>>> where it runs; the data is read into the string as-is. However, every
>>> String which you read from the file using f.gets or f.getc is marked to
>>> say that it is encoded using UTF-8. (But not Strings which you read
>>> using f.read)
>>
>> I'm not seeing this behavior.
>>
>> 1.9.3-p194 :001 > f = File.open('hello.txt', 'r:windows-1252')
>>  => #<File:hello.txt>
>> 1.9.3-p194 :002 > c = f.getc
>>  => "h"
>> 1.9.3-p194 :003 > c.encoding
>>  => #<Encoding:Windows-1252>
>
> The letter "c" has not been transcoded. It happens to be the same in
> windows-1252 and UTF-8, the single byte 0x63. However if you try it with
> a high character >0x80 you'll see that it is not transcoded into UTF-8.
>
> Note that a string in ruby 1.9 is a vector of two things: an array of
> bytes, and an encoding tag claiming that this string is encoded in a
> particular way. c.encoding only shows you the tag.
>

I'm aware of the implementation details of Ruby 1.9's String. What
I've been trying to figure out for a bit now is all of the
idiosyncrasies of the standard library APIs. As such, I'm very curious
about these details, so I performed a few more experiments.
Interestingly I'm still not seeing this behavior. Could this have
changed at some point between 1.9.0 and 1.9.3-p194? Am I running into
something on OS-specific?

I ran two tests. One with the inverted exclamation character, which is
code point U+00A0 and the euro sign, which is code point U+20AC. I
used these two characters, as the inverted exclamation has the same
code point value in Unicode, ISO-8859-1 and Windows-1252, but the byte
value is two bytes in UTF-8 and one byte in Windows-1252; the euro
sign is in Unicode and Windows-1252, but at different code points.

For the euro sign, i create a windows-1252 text file with a single
byte of 0x80 (the code point value) and then opened up IRB and ran the
following.

1.9.3-p194 :001 > f = File.open('euro_win1252.txt', 'r:windows-1252')
 => #<File:euro_win1252.txt>
1.9.3-p194 :002 > c = f.getc
 => "\x80"
1.9.3-p194 :003 > c.encoding
 => #<Encoding:Windows-1252>
1.9.3-p194 :004 > ct = c.encode('utf-8')
 => "€"

For the inverted exclamation point, i created a windows-1252 text file
with a single byte of 0xA1 (the code point value) and then opened up
IRB and ran the following.

 1.9.3-p194 :001 > f = File.open('inverted_win1252.txt', 'r:windows-1252')
  => #<File:inverted_win1252.txt>
 1.9.3-p194 :002 > c = f.getc
  => "\xA1"
 1.9.3-p194 :003 > c.encoding
  => #<Encoding:Windows-1252>
 1.9.3-p194 :004 > ct = c.encode('utf-8')
  => "¡"

Am I not working through this correctly?

FWIW - I'm running my tests on Mac OS X 10.8.2 with Ruby 1.9.3-p194
installed via RVM.

Thanks.