From: Rob Doug <broken.m@...>
Date: 2009-10-23T23:28:42+09:00
Subject: Re: memory leak

Well, seems error not in my code. I made simple version of crawler, 
there is just could not be my mistake. But, the program is still have 
memory leak :( ... you can check yourself, here is the code(after 2000 
random urls mem usage >200mb):

require 'rubygems'
require 'mechanize'
require 'thread'

mutex = Mutex.new

threads = []
$n = 0

THREADS = 50
q = SizedQueue.new(THREADS * 2)

threads = (1..THREADS).map do
   Thread.new q do |qq|
     until qq.equal?(myLink = qq.deq)
       mutex.synchronize do
         puts ($n +=1).to_s # + " : " + print_class_counts.to_s
       end
       begin
         agent = WWW::Mechanize.new{ |agent|
        agent.history.max_size=1
        agent.open_timeout = 20
        agent.read_timeout = 40
        agent.user_agent_alias = 'Windows IE 7'
        agent.keep_alive = false
    }
    page = agent.get(myLink)
    puts myLink
    puts page.forms.length

    page.forms.each do |form|
    end
    rescue
    end
     end
   end
end


File.foreach("bases/base.txt") do |line|
   line.chomp!
   q.enq(line)
end

threads.size.times { q.enq q}
sleep(120)

threads.each { |t| t.join}

It's very sad, because I like ruby very much, but seems it does not fit 
to my projects :(
-- 
Posted via http://www.ruby-forum.com/.