From: Thomas Hafner <thomas@...>
Date: 2007-03-04T11:45:07+09:00
Subject: Re: Parsing a file with look ahead

"S. Robert James" <srobertjames@gmail.com> wrote/schrieb <1172104495.175931.201910@p10g2000cwp.googlegroups.com>:

> I need to parse a file line by line, and output the results line by
> line (too big to fit into memory).  So far, simple enough:
> file.each_line.
> 
> However, the parser needs the ability to peek ahead to the next line,
> in order to parse this line.  What's the right way to do this? Again,
> I really don't want to try to slurp the whole file into memory and
> split on newlines.

Sounds for me like it could be solved elegantly with a lazy stream of
input lines. For lazy streams see the Usenet thread starting with
article <9oib84-sf6.ln1@faun.hafner.nl.eu.org>, for instance.

The file will be split into lines, but lazily, and for that reason all
the lines don't need to be hold in memory at the same time. Old, i.e.
already consumed lines will be garbage collected soon, because the
application does no longer reference them. You can have as many
lookahead lines as you want (tradeoff: needs more memory, of course).

Regards
  Thomas