Re: Encoding question

Nathan Beyer wrote in post #1077055:
> I'm aware of the implementation details of Ruby 1.9's String. What
> I've been trying to figure out for a bit now is all of the
> idiosyncrasies of the standard library APIs.

Ah, well that's an open-ended question. Ruby's standard library is very =

large, and none of the encoding-related behaviour is documented. But =

File.open / getc are pretty fundamental to encoding behaviour.

> As such, I'm very curious
> about these details, so I performed a few more experiments.
> Interestingly I'm still not seeing this behavior. Could this have
> changed at some point between 1.9.0 and 1.9.3-p194? Am I running into
> something on OS-specific

I don't think so, unless it's behaviour of irb. I think you are just =

misinterpreting the results, and being confused by String#inspect.

Look at
    c.bytes.to_a
and
    c.pack("H*")
to see what's really in the String.

> I ran two tests. One with the inverted exclamation character, which is
> code point U+00A0 and the euro sign, which is code point U+20AC. I
> used these two characters, as the inverted exclamation has the same
> code point value in Unicode, ISO-8859-1 and Windows-1252, but the byte
> value is two bytes in UTF-8 and one byte in Windows-1252; the euro
> sign is in Unicode and Windows-1252, but at different code points.
>
> For the euro sign, i create a windows-1252 text file with a single
> byte of 0x80 (the code point value) and then opened up IRB and ran the
> following.
>
> 1.9.3-p194 :001 > f =3D File.open('euro_win1252.txt', 'r:windows-1252')=

>  =3D> #<File:euro_win1252.txt>
> 1.9.3-p194 :002 > c =3D f.getc
>  =3D> "\x80"

That's the single byte you expected. However String#inspect has some =

hard-coded behaviour which treats bytes in the range 0x80-0x9f (I think) =

as unprintable, and therefore substitutes hex representation. "puts c" =

will squirt the string directly at the terminal, and because your =

terminal is UTF-8 but the string is invalid UTF-8, it will be =

unprintable. Your terminal will probably substitute some special =

character.

> 1.9.3-p194 :003 > c.encoding
>  =3D> #<Encoding:Windows-1252>
> 1.9.3-p194 :004 > ct =3D c.encode('utf-8')
>  =3D> "=E2=82=AC"

You've transcoded it. Now ct contains two bytes, which is the UTF-8 =

representation of that character. Then you've sent it to the screen.

By default ruby does *no* transcoding on output (i.e. it does not take =

into account the encoding of your terminal). Your terminal is in fact =

UTF-8, and so those two bytes get displayed as the one character you're =

sending.

(Because you're running OSX, your terminal is almost certainly UTF-8; =

mine is anyway)

> For the inverted exclamation point, i created a windows-1252 text file
> with a single byte of 0xA1 (the code point value) and then opened up
> IRB and ran the following.
>
>  1.9.3-p194 :001 > f =3D File.open('inverted_win1252.txt',
> 'r:windows-1252')
>   =3D> #<File:inverted_win1252.txt>
>  1.9.3-p194 :002 > c =3D f.getc
>   =3D> "\xA1"

Again that's one byte; for some reason String#inspect or irb is showing =

it in hex representation and I don't know why in this case. But if it =

didn't, it would be unprintable on a UTF-8 terminal.

>  1.9.3-p194 :003 > c.encoding
>   =3D> #<Encoding:Windows-1252>
>  1.9.3-p194 :004 > ct =3D c.encode('utf-8')
>   =3D> "=C2=A1"

Now ct contains 2 bytes, the UTF-8 representation of that character, and =

your terminal displays it properly.

Anyway, try running something like this from the command line and see if =

it's any clearer, because it eliminates any possible interaction with =

irb.

File.open("inverted_win1252.txt","wb") do |f|
  f.write "\xA1"
end
File.open("inverted_win1252.txt","r:windows-1252") do |f|
  c =3D f.getc
  puts c.bytes.to_a
  puts c.unpack("H*")
  puts c.encoding
  puts c.inspect
  puts c
  ct =3D c.encode("utf-8")
  puts ct.bytes.to_a
  puts ct.unpack("H*")
  puts ct.encoding
  puts ct.inspect
  puts ct
end

Regards,

Brian.

-- =

Posted via http://www.ruby-forum.com/.=