From: Robert Klemme <shortcutter@...>
Date: 2009-09-15T01:05:12+09:00
Subject: Re: Pre-allocate large amount of memory?

2009/9/14 Carsten Gehling <carsten@sarum.dk>:
> I've created a small daemon, that serves certain data very fast to our
> remaining web-application. It does so by pre-loading a lot of data from
> our database into a special structure in memory. Boiled down, the
> pre-loading procedure looks like this:
>
> @data_hash = {}
> DBI.connect(dns, user, pass) do |dbh|
>  sth = dbh.execute("some-sql-returning >300.000 rows")
>  while row = sth.fetch_hash
>    hash_key = <calculated-from-row-data>
>
>    @owner_relations[hash_key] ||= []
>    @owner_relations[hash_key] << row
>  end
>  sth.finish
> end
>
> It gives a @data_hash with a structure like this:
>
> @data_hash = {
>  'key1' => [row1, row2, row3, ...]
>  'key2' => [row4, row5, ...]
>  ...
> }
>
> I have a problem with the speed of the pre-loading though. I am loading
> ~360.000 rows into memory (about 500MB). The first 50% of the rows are
> read and placed in the hash pretty quick. But after that it slows down.
>
> I suspect that Ruby's dynamic memory allocation for the Hash is to
> blaim. Can I somehow pre-allocate 500MB memory for the hash? Or tweak
> the way Ruby allocates memory?

Maybe it's not such a good idea to read everything in memory. After
all, retrieving large volumes of data from disk is where RDBMS's
shine.  What about querying individual records and storing them in a
LRU storage. You'll find such a beast here which can be used like a
Hash:

http://github.com/rklemme/muppet-laboratories/tree/9eb71ab7bf7d5db7396905b500bd3ac7cc153e0a/lib

Then you don't burn large amounts of memory but retain the caching effect.

Cheers

robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/