Re: WeakRef -- excessive memory usage?

Thanks for your help Rick.

I didn't realise irb was a bad tool to test memory usage. I've written a 
standard ruby program. Comments about the memory usage I observed at 
each step are included. This program indicates to me that the WeakRef 
implementation does require more memory than I'd assumed they would.



require 'WeakRef'

s = 'Hello'
a = []

GC.start

puts 'Start'
gets
   # Memory Usage: 2,652K

100.times do
   a << WeakRef.new(s)
end

GC.start

puts '100 WeakRef instances in array'
gets
   # Memory Usage: 53,360K

GC.start

puts 'Additional GC (ensure temporary data incurred by WeakRef instances 
is released)'
gets
   # Memory Usage: 53,360K

a = nil
GC.start

puts 'Array GCed'
gets
   # Memory Usage: 19,128K

GC.start

puts 'Additional GC (ensure release of WeakRefs)'
gets
   # Memory Usage: 18,996K

GC.start

puts '2nd Additional GC (ensure release of WeakRefs)'
gets
   # Memory Usage: 18,996K



Here's all I can think of that might be consuming memory in the WeakRef 
implementation:

WeakRef
1. Standard ruby object instance overhead
2. "Int" instance var (object_id of target)
3. Hash entry (WeakRef -> object_id of target)
4. Hash entry + array entry (object_id of target -> array of weakrefs)

It doesn't look like a look of data. But WeakRef also inherits from 
Delegator, maybe it's this class that's pushing up memory usage?

As a quick test I modified weakref.rb so that it no longer inherited 
from "Delegator," and I ran my test program again. The memory usage 
hardly moved from the value taken at startup. So I think "Delegator" is 
the culprit.

I can only guess at what might be causing it for now, but I've got a 
feeling it's the "eval" block in Delegator::initalize, which looks like 
it makes a new class method for each method in "obj" not belonging to an 
ancestor. Still, even if this is executed every time WeakRef is 
instantiated, I'd expect the data associated with the existing methods 
to be discarded every time the eval block executed again. Unless, maybe 
a new metaclass is created each time the methods are 'overridden' by the 
instantiation of the new WeakRef instance?

There's my ideas!

David



Rick DeNatale wrote:
> 2008/4/15 mail"@ruby-lang.org David Beswick <"david>:
>> Thanks for your reply Derek. I can definitely see that the intepreter is
>>  freeing memory over the course of the run.
>>
>>  In my original example, I made sure to add WeakRefs to an array to
>>  ensure that they wouldn't get garbage collected. The reason I still
>>  think WeakRef is using an unusual amount of memory is that when I assign
>>  that array reference to nil on the completion of the run and call
>>  GC.start, memory usage goes down from around 150mb to 20mb (for example).
>>
>>  So, I can see that ruby hasn't completely shrunk its heap back down to
>>  the original size (as you say, it must reserve the memory as per its
>>  memory management algorithm). But this also means that the WeakRef
>>  objects were using a large amount of the heap, since the majority of the
>>  heap was released once the references to the WeakRefs were released.
>>
>>  What it is about WeakRef's implementation that might cause this?
> 
> A few observations:
> 
> 1) The job of the GC is to preserve any reachable objects while
> reclaiming the space taken up by non-reachable objects.  The first
> part of this is a bit more important than the second, since freeing a
> referenced object can lead to mysterious errors when a reference is
> followed.
> 
> 2) Irb is not a good tool for doing experiments with GC. The reason is
> that it has references itself which can cause objects to hang around
> which you might not expect.
> 
> 3) WeakRefs themselves take up space.  In the 1.8 implementation each
> WeakRef is a ruby object which will have a class pointer, and an
> instance variable hash to hold it's one instance var which hangs on to
> the object id of the object it's referenceing.  Then there are two
> class variables which point to hashes.  One hash has an entry for each
> WeakRef object and maps it to the id of the object it references, the
> other hash maps the object id of each object referenced by a WeakRef
> to an array of WeakRefs referring to that object.  The entries in
> those hashes are set when the WeakRef is created, and manipulated by a
> finalizer, which deletes the appropriate entries in each of the hashes
> when a weakly referenced object is GCed.  I haven't added it all up
> but I suspect that each WeakRef takes more storage than your string
> "Hello"
>