From: Robert Klemme <shortcutter@...>
Date: 2010-03-05T22:15:23+09:00
Subject: Re: Limit number of concurrent running threads in pool

2010/3/5 Joe Martin <jm202@yahoo.com>:
> Robert Klemme wrote:
>> 2010/3/4 Joe Martin <jm202@yahoo.com>:
>>> check how many threads are running at once, and if the number of running
>>> threads falls below the limit of 5, it takes the next thread out of the
>>> pool and runs it? �Not sure how I would go about doing this, pretty new
>>> to multithreading.
>>
>> Why do you create a pool much larger than the load you want to accept?
>>  Usually the pool size is used to limit concurrency.  Actually that is
>> the main purpose of thread pools.
>>
>> If you have different tasks for which you want to have different
>> limits on concurrency you could also create several pools with
>> different sizes.
>>
>> Kind regards
>>
>> robert
>
> When running this program, I will provide a list of items that need
> processing.  In some cases, this list can be as long as 250 items, in
> other cases well over 50,000.  The processing of each item can take
> anywhere from 15 to 60 seconds per item, so you can see there is a
> benefit to multithreading here.  In processing each item, there are also
> a number of database calls that occur, so I would like to put a cap on
> the number of actively running threads to avoid overwhelming the
> database.  Am I going about this the wrong way?  Is there a more
> effecient more suitable way of doing this?

For this scenario a thread pool with fixed size seems sufficient.

queue = Queue.new # or bounded queue

def cont(item) !item.nil? end

threads = (1..10).map do
  Thread.new do
    while (cont(item = queue.deq))
       # .. process
    end
  end
end

queue.enc "Task"

threads.size.times do
  queue.enq nil # terminate
end

threads.each {|th| th.join}

As simple as that.

Kind regards

robert


-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/