From: Armin Roehrl <armin@...>
Date: 2003-12-08T22:20:00+09:00
Subject: Re: handling large data sets

how large is the data?
what do you do with the data?

We looked at some logfiles (~ 100 gigaybtes; can't remember exact size)
and I noted that a single-pass over the data using a ruby-script was much
faster than using mysql.

probably obvious:
Only load in memory what you absolutely need.
Experiment with the data-chunks-sizes you read in.
Turn off GC temporarily (if it is feasible)
Write a few routines in C; (SWIG rules)

good luck,
    -A