From: "ara.t.howard" <ara.t.howard@...>
Date: 2007-12-14T05:53:09+09:00
Subject: Re: [ANN] bj-0.0.2


On Dec 13, 2007, at 10:36 AM, Alex Young wrote:

> My word.  I think you've just saved me a ton of work.  Yet again.
>
> A quick question, though.  How difficult is it to set up parallel  
> job queues, so that a cluster node can pick up jobs from one queue,  
> process them, and submit them to the next in a chain?  Take a  
> search engine's spider as an example - from 20,000 feet you've got  
> a job that fetches a page, a job to parse the contents, followed by  
> a third to index the parsed structure.  Chances are that you want  
> different types of cluster node to work on each type of job, and  
> there's different data that you might want to attach at each  
> stage.  Is that easy to set up?



not exactly but this would be quite close:

1) i'd forget about having specialized nodes unless you have a very  
good reason - the death of one node will halt the entire processing  
chain otherwise.  it's nice if nodes are dumb from the perspective of  
robustness.  that said i'll add a feature where you can say

   Bj.submit 'job.exe', :runner => 'some.hostname'

to specify which host to run on.  this'll be two lines of code so i  
don't mind adding it.

2) bj supports priorities so here is what i would do.  say you've got  
a three stage job: a, b, c and 1000 initial 'a' tasks.  furthermore  
let's say you make a ./scripts/ directory in your rails_root (bj runs  
all jobs from the rails_root).  so you'll have something like

   ./scripts/task_a
   ./scripts/task_b
   ./scripts/task_c

then you'd do something like this in your rails app

   jobs = inputs.map{|input| "./scripts/task_a #{ input }}
   Bj.submit jobs, :priority => 10

now task_a is going to do this

   #! /usr/bin/env ruby
   input = ARGV.shift
   output = process_for_task_a input
   system "./script/bj submit ./scripts/task_b #{ output } -- 
priority=20"

(of course, if your processing needs to be run through ./script/ 
runner you'll just be able to use the api directly instead of the  
cli...  i'll be adding a feature shortly to allow for running ruby  
code through script runner directly)

task_b, for it's part, runs and submits task_c at priority=30.

so think about that for a minute and imagine you have three processes  
nodes - each will consume a task_a, run it, and then submit a  
priority=20 job.  therefore each node will probably then get one of  
those higher priority jobs, run that, and then find the priority=30  
task_c job in the queue.  when those are done there will nothing left  
except priority=10 task_a jobs and another batch will start.


so this will give you parallel processing of a host of tasks.

make sense?

a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being  
better. simply reflect on that.
h.h. the 14th dalai lama