From: Gavin Sinclair <gsinclair@...>
Date: 2003-12-08T21:05:45+09:00
Subject: Re: handling large data sets

On Monday, December 8, 2003, 5:42:14 PM, Martin wrote:

> Hi...

> short version:
> What's your prefered way to handle large data sets used by Ruby scripts?

> [...]

> Any experiences to tell? :-)

Martin,

I can't answer the specific technical questions, but the one
experience I had with using Ruby to process large amounts of data
involved went something like this.  There was heaps of data in one set
of files, and heaps of data in another set, and I had to cross-match
the datasets to prepare data for insertion in a database.

But I needed to run it regularly on growing, rather than different,
files.  I can't remember what was particularly expensive, but I saved
a lot of time by caching pre-calculated results in a Marshal dump, so
that each time the program was run, it could work out what data it did
and didn't need to examine.

The data I was marshalling was slow in 1.6 and fast in 1.8.

It sounds like your data is an order of magnitude larger than mine,
and probably a different problem, but I'm sure some use of Marshal, or
YAML. can help.

It's probably worth experimenting in loading a 10000-line file into
internal data structures vs loading the pre-saved data structures.

Cheers,
Gavin