From: Stephen Kellett <snail@...>
Date: 2005-07-19T18:15:55+09:00
Subject: Re: Memory profiling?

>NB - if you have not looked in the Java profiler space lately, it might
>be worth a glance.  Virtually every profiler package out there can tell
>you how many of each object are live, how many are dead, the line that
>produced each allocation, and the amount of VM taken by each.  Thus, you
>can easily find:
>   The lines that allocated the most memory
>   The lines that churned (allocated and freed) the most memory
>   The lines that allocated the most objects
>   The lines that churned the most objects
>   The objects holding on to the most data
>
>and so on.
>
>This kind of stuff usually makes it very easy to discover why an app is
>taking up so much RAM, which is a very good first step when deciding to
>improve both speed and footprint.

Ruby Memory Validator is in pre-beta at Software Verification. Currently
it has been released to one person profiling Rails. It can do all of the
above (except the last as Ruby is really impervious to queries like that
from C).

To implement RMV has involved some very intricate binary patches to
Ruby. We know what needs to be done, but Ruby's construction really
mitigates against doing this easily.

I think it should be implemented as a C-only callback so that we can
avoid creating Ruby objects as part of the callback. Additionally we'd
need to be able to provide a means to get a Ruby callstack from inside
the callback.

The callback would be called for the following events:
o Object creation. This would provide:
        VALUE - the object value
        char * - the object type
        Optional, but nice if possible - the size of the object
o Object destruction. When cleaned up using obj_free() or
rb_gc_force_recycle().
o Garbage collection start.
o Garbage collection end.

All of the above events are useful and should be supported. Java
supports them and as a result has many memory profilers. We have
implemented the above for Ruby, although getting the object type from
rb_newobj() is impossible so we've had to resort to a rather
un-satisfactory delayed approach to type detection.

Other very useful things that the C callable API should provide (again
without Ruby creating more VALUEs to be put through the GC): These
actions are required to implement the last item on the list provided by
Scott.
o Ability to determine all VALUEs referred to be a specific VALUE.
o Ability to determine all GC roots.
o Heap dump, listing all GC roots and identifying each VALUE and its
referred values.
The above 3 options allow the creation of very useful heap dumps showing
the reference graph of all objects. This allows "leaks" to be detected.

You will note I am very keen on a C callable API. The very action of
using a Ruby callable API will create and destroy many objects and will
invoke the Ruby gc many times (it operates very frequently!) just
servicing on call to the Ruby API. This will distort the reports quite
badly.

When we have the help available we will make RMV available as a beta.
Until then if you are interested please email betatest AT softwareverify
D0T com if you wish to test the current implementation.

RMV runs on Windows NT 4.0/W2K/XP or better.

Stephen
-- 
Stephen Kellett
Object Media Limited    http://www.objmedia.demon.co.uk/software.html
Computer Consultancy, Software Development
Windows C++, Java, Assembler, Performance Analysis, Troubleshooting