From: Lionel Bouton <lionel-subscription@...>
Date: 2007-10-28T07:46:38+09:00
Subject: Re: A memcached-like server in Ruby - feasible?

Tom Machinski wrote:
> Hi group,
>
> I'm running a very high-load website done in Rails.
>
> The number and duration of queries per-page is killing us. So we're
> thinking of using a caching layer like memcached. Except we'd like
> something more sophisticated than memcached.
>
> Allow me to explain.
>
> memcached is like an object, with a very limited API: basically
> #get_value_by_key and #set_value_by_key.
>
> One thing we need, that isn't supported by memcached, is to be able to
> store a large set of very large objects, and then retrieve only a few
> of them by certain parameters. For example, we may want to store 100K
> Foo instances, and retrieve only the first 20 - sorted by their
> #created_on attribute - whose #bar attribute equal 23.
>   

It looks like the job a database would do for you. Retrieving 20 large
objects with such conditions should be a piece of cake for any properly
tuned database. Did you try this with PostgreSQL or MySQL with indexes
on created_on and bar? How much memory did you give your database to
play with ? If the size of the objects is so bad it takes too much time
to extract from the DB (or the trafic is too much for the DB to use its
own disk cache efficiently) you could only retrieve the ids in the first
pass with hand-crafted SQL and then fetch the whole objects using
memcache (and only go to the DB if memcache doesn't have the object you
are looking for).

Lionel.