From: Robert Klemme <shortcutter@...>
Date: 2009-10-22T17:40:09+09:00
Subject: Re: memory leak

On 10/22/2009 04:52 AM, Rob Doug wrote:
>> You could print out object statistics to get an idea about the source of
>> the leak.  Something along the lines
>>
>> require 'pp'
>>
>> def print_class_counts
>>    c = Hash.new 0
>>    ObjectSpace.each_object(Object) do |o|
>>      c[o.class] += 1
>>    end
>>    pp c
>> end
>>
>> Cheers
>>
>>   robert
> 
> I retest my script again on 10k+ database and seems it stops to grow 
> when reach 550mb size... I'm running 50 threads, so 10mb per thread. As 
> I understand this is still not good ?
> 
> Robert, I did what you propose and here is some data. I print all 
> objects that have more than 200 copies.
> 
> Here object after 100 URLs:
> "Class:   1039"
> "Thread:   395"
> "Gem::Version::Part:   343"
> "Regexp:   1463"
> "MatchData:   301"
> "Hpricot::Text:   31848"
> "Gem::Version:   424"
> "Array:   24977"
> "Nokogiri::XML::Element:   1636"
> "WWW::Mechanize::Chain:   203"
> "Gem::Requirement:   380"
> "WWW::Mechanize::Form::Field:   627"
> "String:   199615"
> "Hpricot::Elem:   21211"
> "Nokogiri::XML::SyntaxError:   1562"
> "Hash:   15884"
> "Proc:   949"
> "WWW::Mechanize::Form::Option:   458"
> 
> Here after 400 URLs:
> "Class:   890"
> "Gem::Version::Part:   343"
> "Regexp:   1089"
> "MatchData:   264"
> "Hpricot::BogusETag:   1573"
> "Hpricot::Text:   82593"
> "Gem::Version:   424"
> "Array:   53259"
> "Nokogiri::XML::Element:   2982"
> "WWW::Mechanize::Form::Button:   259"
> "WWW::Mechanize::Chain:   242"
> "Gem::Requirement:   380"
> "WWW::Mechanize::Form::Field:   1179"
> "String:   426729"
> "WWW::Mechanize::Form::RadioButton:   351"
> "Nokogiri::XML::SyntaxError:   6464"
> "Hash:   37025"
> "Hpricot::Elem:   55426"
> "Proc:   774"
> "WWW::Mechanize::Form:   338"
> "WWW::Mechanize::Form::Option:   601"
> 
> And here after 500 URLs:
> "Class:   991"
> "Thread:   354"
> "Gem::Version::Part:   343"
> "Regexp:   847"
> "MatchData:   409"
> "Hpricot::BogusETag:   915"
> "Hpricot::Text:   31707"
> "Gem::Version:   424"
> "Array:   24143"
> "Nokogiri::XML::Element:   2076"
> "WWW::Mechanize::Form::Button:   217"
> "WWW::Mechanize::Chain:   218"
> "Gem::Requirement:   380"
> "WWW::Mechanize::Form::Field:   874"
> "String:   197836"
> "Nokogiri::XML::SyntaxError:   2608"
> "Hash:   14719"
> "Hpricot::Elem:   21530"
> "Proc:   925"
> "WWW::Mechanize::Form:   266"
> "WWW::Mechanize::Form::Option:   416"
> 
> 
> As you can see there is no stable increasing amount of some object.
> 
> What can I do more?

Your number of threads is suspiciously high compared to the 50 threads 
that you intend to run.  That may be an indication that your code still 
suffers the issue that you stuff Thread instances into the Array and 
never remove them.

Other than that I cannot see something obvious on first sight.  Maybe it 
makes more sense to extend the memory dumping to dump only positive 
deltas.  An alternative is to decrease the dumping interval (e.g. just 
after fetching just one URL).

If you want to throw more on this you could use set_trace_func to 
capture invocations of methods named "new" along with the stack trace so 
you get the allocation sites.

Btw, what version of Ruby is this?  IIRC there was a bug with 
Array#shift up to 1.8.6 which could also cause these effects.

Kind regards

	robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/