From: "M. Edward (Ed) Borasky" <znmeb@...>
Date: 2007-11-11T05:47:19+09:00
Subject: Re: enterprise ruby

Lionel Bouton wrote:
> But my current problem is that simply iterating other large associations
> (to create a new object for each and every object on the other end of a
> has_many association with complex business rules SQL can't handle for
> example) is enough to use 100-300MB with hundreds of thousands of
> objects.

Ah ... complex business rules. That's the big problem with programming 
languages -- they make it possible to *have* complex business rules. 
Before computers were invented, we had to make do with "buy low, sell 
high, collect early, pay late" and double-entry bookkeeping. :)

 > A temporary 100-300MB spike isn't a problem, what is a problem is that :
> 1/ the memory isn't freed after completion of the task,
> 2/ it's kept dirty by the GC
> ->there's no way the OS can reuse this memory for another spike
> happening in another process, only the original process can reuse it.
> 
> This is not a major problem : I can always move all these huge
> processings in short-lived dedicated processes but it's kind of a downer
> when the language keeps out of your way most of the time and then shows
> one limitation.
> 

[snip]

> As I understand it, the problem is that MRI keeps some unused memory
> allocated and then the GC marks it dirty... So technically there's
> information being used and re-used frequently but only by the GC :-(


Well ... that sounds like an actual bug rather than a design issue in 
MRI. Is it that the GC can't tell it's unused?