From: Robert Klemme <shortcutter@...>
Date: 2006-12-14T21:20:08+09:00
Subject: Re: newbie: unpacking binary data

On 14.12.2006 12:14, Mark Van de vyver wrote:
> Hi,
> Thanks for the efforts put into making Ruby available!
> While new to Ruby it seems worth investing some time in.  Eventually I'd
> like to unpack some data from a binary file and insert it into a MySQL
> table.
> At the moment I'm having trouble looping over the source data file, and
> just viewing the data to make sure things are proceeding correctly.
> 
> FYI: eventually I need this to run quickly (many GB's to be processed),
> but for the moment I'll settle for getting anything to work :)
> 
> The ruby details are:
> $ ruby -v
> ruby 1.8.5 (2006-08-25) [i386-cygwin]
> 
> -----------
>  Attempt 1 ( an error )
> -----------
> File.foreach("./INPUT/T200510A.BIN", "rb") do |f|
>    print f.read(19).unpack('I4 I4 I4 I2 I2 a2 a1')
> end
> NoMethodError: [FATAL] failed to allocate memory

This does not work because the second argument to foreach is the /line 
separator/ - not the file mode.

> -----------
>  Attempt 2 ( an error )
> -----------
> File.open("./INPUT/T200510A.BIN", "rb").readlines do |f|
>   data = f.read(19).unpack('I4 I4 I4 I2 I2 a2 a1')
>   print data
> end
> /usr/lib/ruby/1.8/irb.rb:298:in `inspect': failed to allocate memory
> (NoMemoryError)

This fails because you use readlines - this will try to read the whole 
file into memory as /lines/ - and since there are no lines in a binary 
file this chokes while attempting to store the whole file or a large 
portion of it in a single line (i.e. String).

You should do this

File.open("./INPUT/T200510A.BIN", "rb") do |io|
   while ( data = io.read(19) )
     p data.unpack('I4I4I4I2I2a2a1')
   end
end

Regards

	robert