From: Cameron Pope <camerooni@...>
Date: 2011-09-13T08:19:42+09:00
Subject: [ruby-core:39492] Re: File::BINARY does not behave as advertised

Thanks for the response. I admit I'm relatively new to how Ruby 1.9
handles character encoding.

> To put it another way: Opening a file in binary mode determines
> whether *Windows* performs character translations to data written to,
> or read from, the file. It makes no difference to the transformations
> *Ruby* performs.

It took me a few minutes to twist my brain around what you're saying,
but that makes sense. In my mind I was equating 'Binary' with 'not
transforming any data when writing'.

> I would expect Ruby to mark the file's encoding according to
> default_external, not default_internal. It apparently does so, if you
> use integer flags:
>
> Encoding.default_internal = 'UTF-8'
> Encoding.default_external = 'ASCII'
>
> File.open('test', File::CREAT | File::RDWR | File::BINARY) do |f|
> �puts "Integer Flags Encoding: #{f.external_encoding.to_s}" # => US-ASCII
> end

What seems to be happening is that the file's encoding is marked to
the external encoding if both the internal and external encodings are
set. Otherwise it will use the default internal encoding. I have not
dug into Ruby's implementation of encodings enough to understand why
that's the case. It seems backwards to me, but that's probably my own
ignorance.

> ...so I think this is the buggy behavior. It looks as if 'w+b' always
> results in ASCII-8BIT external encoding, whatever the value of
> Encoding.default_external.

I'm not sure that's buggy - it's at least consistent with what the
documentation for IO says:

  Binary file mode (may appear with any of the key letters listed above).
       |  Suppresses EOL <-> CRLF conversion on Windows. And
       |  sets external encoding to ASCII-8BIT unless explicitly
       |  specified.

> I guess you're not running Ruby on Windows?

Not with a development environment, so I downloaded the ruby installer
dev kit. You're absolutely right.

> O_BINARY is defined by the OS, if the feature exists, in <fcntl.h>.
> Hence the ifdefs in the Ruby source code.

It all makes sense now. Thanks!

> If PStore wants binary read/write, it ought to be specifying that Ruby
> should open the files as binary character encoding; opening them in OS
> binary mode is not sufficient to do that (and will be a null op on
> Unix).

Agreed!

-- 
Cameron Pope
camerooni@gmailcom