From: Robert Klemme <shortcutter@...>
Date: 2011-12-13T06:31:30+09:00
Subject: Re: [ANN] jobQueue 1.0.1 - Running stuff with a user defined number of threads

On Mon, Dec 12, 2011 at 5:18 PM, Ralf Mueller <ralf.mueller@zmaw.de> wrote:
> On 12/12/2011 01:23 PM, Robert Klemme wrote:
>>
>> I would change the API slightly to modify behavior of #push:
>>
>> 1. push(obj, method, *arguments)
>
> This was my first interface idea, too. But I skipped it to be able to push a
> whole list of items at once. But it really lacks readability - that's for
> sure. I guess I will let the user do the iteration over the items to push in
> favour of having a beautiful interface. I could keep the old push interface
> under a different name.

Or you just add #push_all(enum).

>> One could even extend behavior by providing a back channel for results:
>>
>> jq.push do |back_channel|
>>   back_channel<<  (1 + complicated_calculation() * 123)
>> end
>>
>> For that of course you must define how reply values are dealt with
>> (there could be a null back channel which just discards results if
>> configured that way).  Alternatively however just the result values of
>> method and block invocation could be used.  Maybe that's cleaner.
>
> Hm. It's definitely good for testing. Could you image a "real" use case for
> this?  Maybe parallel image processing or database requests.

Well, any farmer worker scenario where you need a single instance
composing all results into something complete.  Also, finding out that
all workers are finished could be viewed as one way of evaluating
results, too.

>> Also I would separate support for the call of system: Basically
>> invoking system is a special case which does not necessarily have
>> something to do with job queues in general.  So a better solution
>> would be to have a specialized job queue, e.g.
>>
>> class SystemJobs
>>   attr_reader :jq
>>
>>   def initialize(jq)
>>     @jq = jq
>>   end
>>
>>   def push(*args)
>>     jq.push do |back_channel|
>>       back_channel<<  system(*args)
>>       # we could use a variant of IO.popen here as well which
>>       # captures output
>>     end
>>   end
>> end
>>
>> You then could still do the pretty short
>>
>> sj.jq.push($stdout, :puts, "hello world")
>
> This would simplify the task handling in the JobQueue class. Running system
> commands and calling ruby methods in the same queue should not be the
> regular case.

My main argument would be separation of concerns.  Your basic JobQueue
is simply only responsible for executing tasks in concurrent threads.
Executing system commands is a special case which would be of no use
for someone who just needs to concurrent execution in the current
process.

> Many thanks!

You're welcome!

Kind regards

robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/