From: Brian Candler <b.candler@...>
Date: 2009-09-02T18:42:40+09:00
Subject: Re: how to convert string to binary and back in Ruby 1.9?

Joe wrote:
> I have this UTF-8 string that I want to turn into binary, and then
> from binary into ISO-8859-1.

UTF-8 is a binary encoding of Unicode codepoints, so it's a sequence of 
binary bytes by definition. And you get the same as your Python code:

irb(main):001:0> '��������������'
=> "��������������"
irb(main):002:0> '��������������'.bytes.each { |x| print "%02x " % x }
d0 bf d0 be d0 bc d0 be d0 bd d0 b8 d0 ba => "��������������"
irb(main):004:0> '��������������'.force_encoding("BINARY")
=> "\xD0\xBF\xD0\xBE\xD0\xBC\xD0\xBE\xD0\xBD\xD0\xB8\xD0\xBA"

I think what's confusing you is this:

irb(main):005:0> str = '��������������'
=> "��������������"
irb(main):006:0> str.force_encoding("ISO-8859-1")
=> "��������������"

Here, Ruby is doing something strange. The string is tagged as a 
sequence of ISO-8859-1 characters, but this sequence of bytes is being 
squirted as-is to a UTF-8 terminal, and so the UTF-8 terminal is 
displaying them as the original characters.

You can get the behaviour you want like this, by transcoding to UTF-8:

irb(main):009:0> str.encode("UTF-8")
=> "����������������������������"

Given that irb is running in a UTF-8 environment, it is arguable that 
STDOUT should have an external encoding of UTF-8, which means text 
should be transcoded to UTF-8 automatically.

That is, you can also get the behaviour you want from this standalone 
program:

# encoding: UTF-8
STDOUT.set_encoding "UTF-8"    # << THE MAGIC BIT

str = '��������������'
str.force_encoding("ISO-8859-1")
puts str

It seems inconsistent to me that STDOUT doesn't get its 
external_encoding set automatically.

> So apparently in Ruby 1.9, binary isn't really binary?

Correct. In Ruby 1.9, binary is ASCII. I hate this.

I have documented a lot of the gory details at
http://github.com/candlerb/string19

Thanks for bringing another anomoly to my attention.
-- 
Posted via http://www.ruby-forum.com/.