From: Tim Pease <tim.pease@...>
Date: 2007-01-31T02:49:06+09:00
Subject: Re: Correct way to process a file in Ruby?

On 1/30/07, Dave Hatton <rubylist@davehatton.it> wrote:
> Hi Chris,
>
> > I just slurp the file into an array (File.readlines) and then
> > split the array into separate arrays of each record type.
>
> I guess I should have mentioned that these files are binary format and
> can be quite large - anywhere from 3M to 50MB, so I was hoping to
> process them on disk.
>
> Would reading into an array still be appropriate in Ruby?
>

If you are on a Unix system, look into the Ruby mmap library.

<http://raa.ruby-lang.org/project/mmap/0.2.6>

This will only read in the parts of the file you need.  There is a
similar library for the windows platform (but I have not used that
one).

My solution to handling stuff like this is to create a parser class.


class FooParser

  def initialize( filename )
    @mmap = Mmap.new(filename, 'r')
  end

  def close
    return if @mmap.nil?
    @mmap.unmap
    @mmap = nil
  end

  def parse_info_type1
    # do stuff here to parse one type of information from the mmap object
  end

  def parse_info_type2
    # etc ...
  end

end


It works very well, and mmap allows me to handle gigabyte sized files
without hogging all the system memory.

Blessings,
TwP