From: Eric Hodel <drbrain@...7.net>
Date: 2009-04-25T03:18:23+09:00
Subject: Re: reformatting a text file that has some binary in it

On Apr 24, 2009, at 03:49, Adam Akhtar wrote:
> well ive found some stuff out re: binary format.
>
> I was getting confused re: the "b" switch in File.open("file", "rb")  
> (as
> in "r**b**")
>
> I thought this was needed to tell ruby we were dealing with some funky
> "binary" file but its a lot simpler than that. There is no special
> binary file format (that im aware of). Binary is just written to a  
> file
> as text is but in unicode (im assuming).

In windows and on ruby 1.9 the 'b' flag says not to perform any  
conversions of bytes to characters on the text, that's all.  Just  
leave it as a stream of bytes.

> So why then do we have to set the "b" for binary mode flag in the
> File.open ?
> Sometimes binary can have the ^Z character in it. As binary its doing
> nothing more than any other character- representing some information  
> but
> in windows that character represents end of file.

Yes, ^Z is the NULL byte "\0" on windows.

> File.open expects text files so if it comes accross ^Z it will stop
> reading even if the text is actually representing binary. To stop ruby
> doing that you use "b" in your call to .open.

It'll also convert line endings, losing data that should be in a  
binary file.

> This is a windows only issue apparently.

It is also an issue on ruby 1.9 for any platform, but for different  
reasons.  Ruby will perform other character conversions.

> This will explain why i was getting different lengths with
>
> data_a = File.read('mn-scrape.txt')
> data_b = File.open("mn-scrape.txt", "rb").readlines.join("")
> data_a.scan(/./m).length ( ==> 170799 )
> data_b.scan(/./m).length ( ==> 767702 )

Yup.