From: Brian Candler <lists@...>
Date: 2012-09-22T18:57:32+09:00
Subject: Re: Encoding question

Nathan Beyer wrote in post #1077055:
> I'm aware of the implementation details of Ruby 1.9's String. What
> I've been trying to figure out for a bit now is all of the
> idiosyncrasies of the standard library APIs.

Ah, well that's an open-ended question. Ruby's standard library is very 
large, and none of the encoding-related behaviour is documented. But 
File.open / getc are pretty fundamental to encoding behaviour.

> As such, I'm very curious
> about these details, so I performed a few more experiments.
> Interestingly I'm still not seeing this behavior. Could this have
> changed at some point between 1.9.0 and 1.9.3-p194? Am I running into
> something on OS-specific

I don't think so, unless it's behaviour of irb. I think you are just 
misinterpreting the results, and being confused by String#inspect.

Look at
    c.bytes.to_a
and
    c.pack("H*")
to see what's really in the String.

> I ran two tests. One with the inverted exclamation character, which is
> code point U+00A0 and the euro sign, which is code point U+20AC. I
> used these two characters, as the inverted exclamation has the same
> code point value in Unicode, ISO-8859-1 and Windows-1252, but the byte
> value is two bytes in UTF-8 and one byte in Windows-1252; the euro
> sign is in Unicode and Windows-1252, but at different code points.
>
> For the euro sign, i create a windows-1252 text file with a single
> byte of 0x80 (the code point value) and then opened up IRB and ran the
> following.
>
> 1.9.3-p194 :001 > f = File.open('euro_win1252.txt', 'r:windows-1252')
>  => #<File:euro_win1252.txt>
> 1.9.3-p194 :002 > c = f.getc
>  => "\x80"

That's the single byte you expected. However String#inspect has some 
hard-coded behaviour which treats bytes in the range 0x80-0x9f (I think) 
as unprintable, and therefore substitutes hex representation. "puts c" 
will squirt the string directly at the terminal, and because your 
terminal is UTF-8 but the string is invalid UTF-8, it will be 
unprintable. Your terminal will probably substitute some special 
character.

> 1.9.3-p194 :003 > c.encoding
>  => #<Encoding:Windows-1252>
> 1.9.3-p194 :004 > ct = c.encode('utf-8')
>  => "€"

You've transcoded it. Now ct contains two bytes, which is the UTF-8 
representation of that character. Then you've sent it to the screen.

By default ruby does *no* transcoding on output (i.e. it does not take 
into account the encoding of your terminal). Your terminal is in fact 
UTF-8, and so those two bytes get displayed as the one character you're 
sending.

(Because you're running OSX, your terminal is almost certainly UTF-8; 
mine is anyway)

> For the inverted exclamation point, i created a windows-1252 text file
> with a single byte of 0xA1 (the code point value) and then opened up
> IRB and ran the following.
>
>  1.9.3-p194 :001 > f = File.open('inverted_win1252.txt',
> 'r:windows-1252')
>   => #<File:inverted_win1252.txt>
>  1.9.3-p194 :002 > c = f.getc
>   => "\xA1"

Again that's one byte; for some reason String#inspect or irb is showing 
it in hex representation and I don't know why in this case. But if it 
didn't, it would be unprintable on a UTF-8 terminal.

>  1.9.3-p194 :003 > c.encoding
>   => #<Encoding:Windows-1252>
>  1.9.3-p194 :004 > ct = c.encode('utf-8')
>   => "¡"

Now ct contains 2 bytes, the UTF-8 representation of that character, and 
your terminal displays it properly.

Anyway, try running something like this from the command line and see if 
it's any clearer, because it eliminates any possible interaction with 
irb.

File.open("inverted_win1252.txt","wb") do |f|
  f.write "\xA1"
end
File.open("inverted_win1252.txt","r:windows-1252") do |f|
  c = f.getc
  puts c.bytes.to_a
  puts c.unpack("H*")
  puts c.encoding
  puts c.inspect
  puts c
  ct = c.encode("utf-8")
  puts ct.bytes.to_a
  puts ct.unpack("H*")
  puts ct.encoding
  puts ct.inspect
  puts ct
end

Regards,

Brian.

-- 
Posted via http://www.ruby-forum.com/.