From: Robert Klemme <shortcutter@...>
Date: 2007-01-04T00:55:13+09:00
Subject: Re: Memory Leak Madness

On 03.01.2007 15:46, Brandon Casci wrote:
> I'm having one hell of a time trying to find and stop a memory leak in a
> ruby daemon. It starts off using a tiny 14 MB of RAM, and 150+MB after a
> day. I've read through many forum and blog posts, and tried several
> profilers. I don't see anything unusual.
> 
> I even tried this guy
> http://scottstuff.net/blog/articles/2006/08/17/memory-leak-profiling-with-rails
> 
> That showed my overall object count going up and down as normal, but
> memory was not being released as objects came and went.
> 
> I've gone through my code made sure I'm squashing any unused objects.
> Well...I set them to nil, I just guessed that would help. I'm also
> calling .clear on any arrays and hashes. I noticed this helps with
> garbage collection. I also added a thread that runs does nothing but
> sleep and run garbage collection every 5 minutes, although I'm pretty
> sure this isn't helping much.
> 
> All I've managed to greatly slow the memory leak.  Before the daemon
> would consume 100+ MB in a few hours.

Is there some point where it stabilizes after a day or so?

> There are two things my daemon makes heavy use of that is out of
> control, hpricot and dbi (talking to ms sql server). I suspect one or
> both of these guys may be my problem. I'll write into both projects for
> advice, but is there anything else I can do so ruby will let go of
> unused objects?

It may actually be doing it.  The memory footprint you see on OS side 
need not be directly related to the number of objects currently in 
memory.  I am not sure whether Ruby ever gives back memory to the OS but 
if not the effect you see might actually be from a situation where there 
were needed many objects at once.

I definitively would not add a thread that does GC.  The interpreter 
will take care of this.

I'd probably do some object statistics (per class) and also summarize 
String, Array and Hash sizes.  If you encounter a significant increase 
in volume somewhere then this might indicate where the problem lies. 
You could try something like this:

require 'pp'

def stats_2
   counts = Hash.new(0)
   sizes = Hash.new(0)

   ObjectSpace.each_object(Object) do  |obj|
     counts[obj.class] += 1

     if obj.respond_to?(:size) && obj.method(:size).arity == 0
       sizes[obj.class] += obj.size
     end
   end

   pp "counts", counts, "sizes", sizes
end

Signal.trap :INT do
   stats_2
end


Kind regards

	robert