From: Brian Candler <b.candler@...>
Date: 2010-03-11T06:19:40+09:00
Subject: Re: reading an UTF-8 encoded file

Une Bévue wrote:
> if i read and output to terminal an UTF-8 encoded file, i do not have
> the same result with ruby 1.8.x and ruby 1.9
> 
> with 1.8 i get "�" correctly, with 1.9 i get it wrong "é" even if i
> specify the encoding by :
> open(__FILE__, "r:UTF-8") do ...

In my web browser onto ruby-forum, I see what you say is the "correct" 
symbol as invalid above, and the "wrong" symbol is a valid one.

Are you in irb, or running code in a .rb file? Are you using "puts" or 
are you looking at the string values as returned by irb, after the => 
prompt?

In either case, show your actual code. Beware that things behave 
strangely in irb with 1.9. Some of the oddities I noticed in irb are 
documented in
http://github.com/candlerb/string19/blob/master/string19.rb
from about line 1648.

> what did i missunderstood ?

Remember that encodings by themselves don't actually change the sequence 
of bytes. If your code is something like this:

open("somefile.txt") do |f|
  while line = f.gets
    puts line
  end
end

and you run it as a .rb script, I would expect it to work the same in 
both 1.8 and 1.9. That is, it should read lines and squirt them back out 
to stdout unchanged. No transcoding is done. If they appear wrongly, it 
would be because the encoding of the file contents is not the same as 
the encoding of your terminal.

Furthermore, it makes no difference in 1.9 if you do this:

open("somefile.txt","r:UTF-8") do |f|
  while line = f.gets
    puts line
  end
end

In ruby 1.9, all this means is that the string 'line' will be tagged as 
being UTF-8, rather than some encoding picked up from the environment. 
However by default, the same sequence of bytes will be squirted out.

However in 1.9 you *can* cause the string to be transcoded, if:

(1) you specify a different internal and external encoding when reading 
the data (so it gets transcoded on input); or

(2) you specify an external encoding when writing the data (so it gets 
transcoded on output)

HTH,

Brian
-- 
Posted via http://www.ruby-forum.com/.