From: Mark Probert <probertm@...>
Date: 2005-02-15T05:33:03+09:00
Subject: More on memory and possible leaks ... (longish)


Hi ..

This is getting tough to track down!  

The basics are that my script is multi-threaded (potentially 20-30 threads), 
running remote data collection using telnet, for up to 300 nodes.  Some state 
information (basic info about the remote node) is stored using hashes within 
the class though all of the "real" data is spooled to disk for later 
processing.

Here are some facts, not necessarily all directly related to the problems I am 
seeing:


1.  I am using -lots- of memory under Solaris, without any exceptions being 
thrown (approx 200Mb per-thread reported for the full application).  When I 
trim everything down to do just login with no state saving, I am using about 
12Mb per thread.  This seems to be very high, though I am not really sure 
whether that amount of usage is or not.  Perhaps it is in keeping with Ruby's 
thread model.  


2.  When I use a fork() model, the amount of memory used per-process is around 
20Mb.  Again, this seems to be quite heavy, however, memory usage doubles to 
40Mb when I capture data in the hashes!  Now, I am -not- gathering a lot of 
data here:

      ### process card data
      if line =~ /^\s+([0-9]+)\s+(.*)$/ then
        @cards[$1] = $2
      end

as an example.  There are at a maximum 14 entries like this.  20Mb+ seems a 
lot of memory to use!


3.  I don't have, yet, a copy of Purify to check for leaks under Solaris, so, 
I am using valgrind under FreeBSD.  valgrind is crashing a lot due to the 
threads and sockets, but that is another issue :-(


4.  Before valgrind dies, it is reporting the following (many of these errors, 
I am just picking out the unique ones):

11:55 (kant)$ vgm --run-libc-freeres=no ruby test.rb
==26384== Memcheck, a memory error detector for x86-linux.
==26384== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward.
==26384== Using valgrind-2.1.2.CVS, a program supervision framework for 
x86-linux.
==26384== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward.
==26384== For more details, rerun with: -v
==26384== 
==26384== Conditional jump or move depends on uninitialised value(s)
==26384==    at 0x806EB35: is_pointer_to_heap (gc.c:605)
==26384==    by 0x806EAFB: mark_locations_array (gc.c:623)
==26384==    by 0x806FD2A: garbage_collect (gc.c:1352)
==26384==    by 0x806E7BF: rb_newobj (gc.c:381)
==26384== 
==26384== Conditional jump or move depends on uninitialised value(s)
==26384==    at 0x806EB7D: is_pointer_to_heap (gc.c:610)
==26384==    by 0x806EAFB: mark_locations_array (gc.c:623)
==26384==    by 0x806FD2A: garbage_collect (gc.c:1352)
==26384==    by 0x806E7BF: rb_newobj (gc.c:381)
==26384== 
==26384== Conditional jump or move depends on uninitialised value(s)
==26384==    at 0x806EDBE: rb_special_const_p (ruby.h:666)
==26384==    by 0x806ED12: gc_mark (gc.c:712)
==26384==    by 0x806EB11: mark_locations_array (gc.c:624)
==26384==    by 0x806EBFD: rb_gc_mark_locations (gc.c:637)
==26384== 
==26384== Use of uninitialised value of size 4
==26384==    at 0x806ED22: gc_mark (gc.c:713)
==26384==    by 0x806EB11: mark_locations_array (gc.c:624)
==26384==    by 0x806EBFD: rb_gc_mark_locations (gc.c:637)
==26384==    by 0x806FD3E: garbage_collect (gc.c:1354)
==26384== 
==26384== Use of uninitialised value of size 4
==26384==    at 0x806ED2F: gc_mark (gc.c:714)
==26384==    by 0x806EB11: mark_locations_array (gc.c:624)
==26384==    by 0x806EBFD: rb_gc_mark_locations (gc.c:637)
==26384==    by 0x806FD3E: garbage_collect (gc.c:1354)
==26384== 
==26384== Use of uninitialised value of size 4
==26384==    at 0x806ED40: gc_mark (gc.c:715)
==26384==    by 0x806EB11: mark_locations_array (gc.c:624)
==26384==    by 0x806EBFD: rb_gc_mark_locations (gc.c:637)
==26384==    by 0x806FD3E: garbage_collect (gc.c:1354)
==26384== 
==26384== Conditional jump or move depends on uninitialised value(s)
==26384==    at 0x806EE4F: gc_mark_children (gc.c:756)
==26384==    by 0x806EDA9: gc_mark (gc.c:729)
==26384==    by 0x806EB11: mark_locations_array (gc.c:624)
==26384==    by 0x806EBFD: rb_gc_mark_locations (gc.c:637)
==26384== 

These errors appear to be pointing at use of unprepared memory by the gc, 
which may or may not be an issue.  Perhaps someone with knowledge of that 
part of ruby would care to comment.


5.  There doesn't appear to be an easy way of tracking memory usage / 
allocations within ruby.  Are there any plans on adding something that might 
help?  Or is it all too system dependant?


So, the bottom line is that I haven't yet determined if there is any memory 
leaks happening.  However, I do seem to be using a lot of memory that, maybe, 
shouldn't be used.  My problem happens when I multiply these small number, 
like 20Mb, by 300.  Then I start to run into real problems.


Any thoughts, fellow rubyists?

Regards,

-- 
-mark.  (probertm at acm dot org)