From: ara.t.howard@...
Date: 2006-09-28T03:26:09+09:00
Subject: Re: Threading and Deadlock

On Thu, 28 Sep 2006, Jordan McKible wrote:

> I'm making the back end for a feed reader.  I plan to have a daemon which
> periodically checks feeds for updates.  The daemon will have one refill
> thread responsible for maintaining a list of feeds to be updated.  It will
> also have n worker threads who pop feeds from the list and process them.
> When the list is empty, the refill thread will sleep for some amount of time
> (no need to update the feeds constantly), refill the list of feeds, and
> signal to the workers to start up again.
>
> Here's a crawler class as I have it so far.  I've removed some of the
> nonrelevant parts.

harp:~ > cat a.rb
require 'thread'

class Crawler
   DEFAULT = {
     'n_workers' => 4
   }

   def initialize opts = {}
     getopt = getopts opts
     @n_consumers = getopt['n_consumers']
     @q = Queue.new
     @producer = new_producer
     @consumers = Array.new(@n_consumers){ new_consumer }
   end

   def new_producer
     new_thread do
       loop do
         feeds = get_feeds
         feeds.each{|feed| @q.push feed}
       end
     end
   end

   def new_consumer
     new_thread do
       loop do
         feed = @q.pop
         process_feed feed
       end
     end
   end

   def get_feeds # placeholder
     %w( a b c )
   end

   def process_feed feed # placeholder
     trace "processing <#{ feed }>"
     sleep 1
   end

   def trace msg
     tid = Thread.current.object_id
     STDERR.puts "#{ msg } (#{ tid })"
   end

   def getopts opts
     lambda{|o| opts[o.to_s] || opts[o.to_s.intern] || DEFAULT[o.to_s]}
   end

   def new_thread &b
     Thread.new do
       Thread.current.abort_on_exception = true
       b.call
     end
   end
end

Crawler.new :n_consumers => 4

STDIN.gets



harp:~ > ruby a.rb
processing <a> (-609386786)
processing <b> (-609491016)
processing <c> (-609529486)
processing <a> (-609324242)
processing <b> (-609386786)
processing <c> (-609491016)
processing <a> (-609324242)
processing <b> (-609529486)
processing <c> (-609386786)
processing <a> (-609491016)
processing <b> (-609324242)
processing <c> (-609529486)



hth.




-a
-- 
in order to be effective truth must penetrate like an arrow - and that is
likely to hurt. -- wei wu wei