From: Eric Mahurin <eric_mahurin@...>
Date: 2005-10-16T22:40:40+09:00
Subject: Re: Help! define_method leaking procs...

--- Yohanes Santoso <ysantoso-rubytalk@dessyku.is-a-geek.org>
wrote:

> Eric Mahurin <eric_mahurin@yahoo.com> writes:
> 
> >> That is still not a valid way to detect leak as that still
> >> depends on
> >> VmSize value. Ruby could be freeing every allocation and
> the
> >> Vmsize
> >> would still grow. 
> >
> > But it is not.
> 
> Be careful, 1. there is no evidence yet that ruby is not
> freeing every
> allocation, and 2. I am not saying there is no memory leak
> caused by
> define_method.
> 
> What I have been saying is, you can't use VmSize as an
> indicator of
> memory leak. I have shown examples where the order of free()
> can
> affect VmSize.
> 
> Here is an alternative scenario that would have resulted in
> what you
> are seeing: ruby could have free()-ed all its allocations
> properly
> when you call GC.start, but may free() them in different
> order on each
> test case which result in you seeing different VmSize values.
> 
> Probably that is not what happens; Probably there really is a
> genuine
> memory leak; But you can't determine which scenario is
> happening from
> VmSize value.
> 
> > Have you tried this example?  On my 768MB machine, top
> shows it
> > filling my physical memory and then the the CPU drops to
> about 3%
> > because it starts swapping and my machine becomes very
> unresponsive.
> > I'd say that's a good indication that top is reporting the
> right
> > thing.
> 
> This only shows whether the free-ing is done immediately or
> not. Thus,
> why my example program focuses in the immediateness of
> freeing.
> 
> > You could consider this case a bug in ruby or a bug in the
> > simple test case above.  It doesn't matter where the fault
> > lies, it still shows a memory leak.
> 
> Memory leak is tricky to show since it requires one to prove
> that
> there is an allocation that is not free-ed by the time the
> process
> ends.

Just because a process frees all its memory before it
terminates doesn't mean there isn't a leak.  That doesn't help
too much if you run out of memory before the process is done. 
I've seen 2 definitions: a) when a program isn't freeing memory
that isn't used anymore, b) when a program loses all references
to memory that isn't being freed.  From the ruby script level
for this example, by both of these definitions it is a memory
leak.  From ruby itself, ruby probably still has references to
the memory, so (b) may not call it a "memory leak".  To me (a)
is the right definition because it describes symtoms not the
mechanism.  If any program keeps growing in memory and it
shouldn't, I'd say it has a memory leak.  It doesn't matter how
it comes about and whether or not the program could free the
memory if it was fixed, it is still "leaking" memory.

> However, there are evidences (search the archive for GC
> problems) that
> the GC is too conservative (read: lazy) and stupid in some
> cases. Sometimes, you need to drop the clue book on its head,
> like
> doing a=nil. Without the a=nil, the number of objects is not
> decreased
> by much after GC.
> 
> > The point is you better be careful with closures
> (block/lambda)
> 
> Yes, be careful as the GC is a lazy bum.

If you want to assign blame to ruby, the problem is not GC. 
The problem is that closures hold strong references to all
variables in the scope whether or not they are used.  One of my
solutions was to make references (in Proc#binding) to currently
unused variables (by the closure) "weak", so that they wouldn't
prevent GC.

> In any case, here is another 'fixed' version. 
> 
> ysantoso@jenny:/tmp$ ruby -e 'n=2**13;for i in (1..n) do
> a=(1..i).to_a;self.class.send(:define_method,:"f#{i}"){i*i}
> end;GC.start;
>
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
> 
> VmSize:	   11268 kB

This is because the "for" loop (which uses #each) doesn't
create the same type of block/lambda that the normal block
(after a method call) does.  In the above example, "a" is in
the outer scope.  With using #each and a block directly, "a" is
a local to each interation that it is yielded - it is a
different variable on each iteration.  The "for" loop you
showed actually doesn't even work the same because all of the
defined f* methods will use the same "i" because "i" is not
local for each iteration for "for" but it is for #each using a
normal block.



__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com