From: Lionel Bouton <lionel-subscription@...>
Date: 2007-11-10T19:26:05+09:00
Subject: Re: enterprise ruby

M. Edward (Ed) Borasky wrote the following on 10.11.2007 05:58 :
> Lionel Bouton wrote:
>> Useful ? Yes it would : I have some Rails actions that eat memory quite
>> happily (associations with hundreds of thousands of objects which
>> themselves have associations on which to work...). It would help if the
>> ruby processes would let go of the memory or at least let it live in
>> swap undisturbed once the action is done.
>
> Sounds to me like you're building a data structure in RAM to avoid
> making your RDBMS earn its keep. ;)

In some cases, yes because the code is easier to maintain that way. I
usually take the time to switch to SQL when it becomes a problem though
and pure SQL is powerful enough for the task (done that several times
last month).

But my current problem is that simply iterating other large associations
(to create a new object for each and every object on the other end of a
has_many association with complex business rules SQL can't handle for
example) is enough to use 100-300MB with hundreds of thousands of
objects. Usually I can split the task paginating through the whole set,
but in some cases it isn't possible : if inserts or deletes happens
concurrently you can miss some objects or try to process some twice (I'm
actually considering fetching all the primary keys in a first pass and
then paginate using windows in this set, which comes with other problems
though manageable ones in my case)...

A temporary 100-300MB spike isn't a problem, what is a problem is that :
1/ the memory isn't freed after completion of the task,
2/ it's kept dirty by the GC
->there's no way the OS can reuse this memory for another spike
happening in another process, only the original process can reuse it.

This is not a major problem : I can always move all these huge
processings in short-lived dedicated processes but it's kind of a downer
when the language keeps out of your way most of the time and then shows
one limitation.

> But seriously, unless you can restructure your application so it
> doesn't keep a lot of stuff in RAM, you're probably doomed to throw
> hardware at it.

Yes, I've done some simple code tuning that helps memory usage, but it
only helps with the wait for a bigger server.

> In other words, hard drives are where data that must live for extended
> periods (or forever) belong, *explicitly* filed there by your
> application code, not *implicitly* filed there by the swapper. RAM is
> for volatile information that is being used and re-used frequently.
>

As I understand it, the problem is that MRI keeps some unused memory
allocated and then the GC marks it dirty... So technically there's
information being used and re-used frequently but only by the GC :-(

Lionel.