From: Joel VanderWerf <vjoel@...>
Date: 2006-10-13T15:29:22+09:00
Subject: Re: Large File Reading and Writing

Daniel N wrote:
> Hi,
> 
> A project that I've currently got at work consists of reading large files
> and writing another larger file.  Unfortunately the machine I'm on is 
> pretty
> lame and it runs out of RAM pretty quick and so takes a long time.  
> (This is
> currently in perl and I'm not keen to play with it in perl.)
> 
> I have had a look around the docs and searched the mailing list ( a quick
> search ) for how to do buffered reading and writing but I've come up with
> nothing.
> 
> I have a bunch of text files.  I want to read some lines, and then write a
> line, the move on to the next line etc but I only want to have a few lines
> in memory at once.  Any clues as to how to achive this?
> 
> Cheers
> Daniel
> 

Something like this maybe...

history = []
ARGF.each do |line|
   history << line
   if enough_history_to_generate_some_output
     write_output
     history.clear
   end
end

The ARGF thingy is explained in 'ri IO':

      The global constant ARGF (also accessible as $<) provides an
      IO-like stream which allows access to all files mentioned on the
      command line (or STDIN if no files are mentioned). ARGF provides
      the methods #path and #filename to access the name of the file
      currently being read.

You can of course open a file by name from your code.

-- 
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407