From: Kirk Haines <wyhaines@...>
Date: 2009-09-15T01:39:25+09:00
Subject: Re: Pre-allocate large amount of memory?

On Mon, Sep 14, 2009 at 10:05 AM, Robert Klemme
<shortcutter@googlemail.com> wrote:

> Maybe it's not such a good idea to read everything in memory. After
> all, retrieving large volumes of data from disk is where RDBMS's
> shine.  What about querying individual records and storing them in a
> LRU storage. You'll find such a beast here which can be used like a
> Hash:
>
> http://github.com/rklemme/muppet-laboratories/tree/9eb71ab7bf7d5db7396905b500bd3ac7cc153e0a/lib
>
> Then you don't burn large amounts of memory but retain the caching effect.

The super easy way to deal with this is to do what James suggested -- use redis.

It's absurdly easy to use, and quite fast.  You ARE limited by the
amount of data you can put into RAM, but you are doing that anyway, so
I assume that's a reasonable limitation.

Using some sort of an LRU cache may work, too.  It depends on whether
the query pattern is random access, or whether it it is clustered
around certain sets of data out of the whole, though.


Kirk Haines