From: Robert Klemme <shortcutter@...>
Date: 2009-10-21T17:40:14+09:00
Subject: Re: memory leak

On 21.10.2009 00:47, Rob Doug wrote:
>> As far as I can see you're never cleaning Array threads.  You are just
>> appending to the Array and so even terminated threads will stay in there
>> because you also do not reuse threads.
>>
>> Design criticism: usually you create threads upfront and pass work to
>> them via a Queue then you don't need the sleep crutch and use blocking
>> instead.  Also, you can make your code better handle large volumes of
>> URLs by using a more streamed approach along the lines Ryan presented by
>> limiting all resources (not only threads but also size of the queue).
>>
>> require 'thread'
>> ...
>>
>> THREADS = 50
>> q = SizedQueue.new(THREADS * 2)
>>
>> threads = (1..THREADS).map do
>>    Thread.new q do qq
>>      until qq.equal?(url = qq.deq)
>>        Common....
>>      end
>>    end
>> end
>>
>> File.foreach("bases/3+4.txt") do |line|
>>    line.chomp!
>>    q.enq(URI.parse(line))
>> end
>>
>> threads.size.times {|q| q.enq q}
> 
> Very thank you, your solution is great. Code become much clear and looks 
> good. :)
> 
> But, this is not solve memory leak :(
> 
> Any ideas or suggestions?
> 

You could print out object statistics to get an idea about the source of 
the leak.  Something along the lines

require 'pp'

def print_class_counts
   c = Hash.new 0
   ObjectSpace.each_object(Object) do |o|
     c[o.class] += 1
   end
   pp c
end

Cheers

	robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/