From: Brian Adkins <lojicdotcom@...>
Date: 2010-12-08T14:05:16+09:00
Subject: Re: Q: what database would you suggest?

Diego Virasoro <diego.virasoro@gmail.com> writes:

> Hi,
> I am doing some scientific work in Ruby. So far the data is stored
> into a massive file (just Marshalled or YAMLed) and then once the run
> is over it is loaded back into memory for some data analysis.
>
> This (not very elegant method) has worked so far but as I will need to
> raise the scale to several GBs the time lost in loading and slicing
> the data in memory is getting painful. Hence I thought I could take
> advantage of the many DB that exist for Ruby. Alas... they are so
> many, and I've got no database experience, so I am not sure what would
> be best.
>
> The data can be thought of as hierarchical tables: i.e. tables in the
> SQL sense with *some* entries corresponding to entire new tables, and
> the others either strings or numbers (integers and float). I have only
> access to one computer, though with multiple processors, so I don't
> have any need for distributing the data. Finally I'd like speed, but
> given the database size I can't keep all the info in memory.
>
> The kind of operations I need to do involve getting slices of tables
> (all the rows but a subset of columns), often in all the "subtables"
> I've got. I also need to add a few rows to each table when the
> simulation is running (but not for analysis) however the rate is
> rather low so I don't expect this to be much of a constraint.
>
> Could people with more DB experience give a few suggestions? I've read
> of MongoDB, CouchDB, MySQL and PostgreSQL... and to be honest I am
> pretty lost, so I am sending this in the hope some fellow Rubyist can
> help a poor DB newbie. ;)
>
> Thank you to all in advance.
> Diego Virasoro

From what you've described, I recommend you spend some time looking into
MongoDB. I've been using relational databases for decades, and I've
recently developed a couple applications using MongoDB. It's fast,
handles hierarchical data well, has a powerful query language, good
indexing, etc.

One of my projects has over 40 million documents, and MongoDB is
handling it *much* better than postgres or mysql.

It does seem to take extra disk space than postgres (my typical rdbms),
but I accept that penalty for the increase in speed and flexibility in
programming.

-- 
Brian Adkins
http://lojic.com/