Re: [ANN] bj-0.0.2

On Dec 13, 2007, at 8:34 AM, Giles Bowkett wrote:

> But why this instead of BackgrounDRb?

well, backgrounddrb was originally written by ezra on top of my slave  
lib, and ezra is one of the sponsors of bj so hopefully he'll chime  
in with his reasons, but here are mine

1) much better name.  gem install bj?  require 'bj'?  seriously giles...

2) backgrounddrb, afaik, is has proven to be a bit tricky for *non- 
experts* to manage and use in a production environment.

3) backgrounddrb aims to provide a 'rubyish' environment for code to  
execute in.  in otherwords you call methods on on objects, serialize  
ruby objects over the wire, etc.  this makes entire classes of  
problems easier to reason about, but it also comes with a price and  
that price is complexity.  for example, most (all?) people have to  
think about methods like this when using drb

   remote_object.each do |thang|
     thang.intense_computation
   end

now, on which cpu does 'intense' run?  in which process?  the answer  
is that it entirely depends on how the objects where setup and how  
DRbUundumped may or may not have been used.  as the maintainer of  
slave.rb i can tell you that the list of people who understand this  
is eric hodel and, um, eric hodel.  the point is that drb is not an  
rpc mechanism but a toolset for building servants.  using drb every  
process is potentially either a client or a server and generally  
both.  it's the block passing mechanism that gets people into trouble  
- blocks cannot go across the wire to drb does some magic to make  
them work.  the other issue with having a 'rubyish' environment to  
execute code in, in the case of using backgrounddrb with a rails app,  
is that rails' ruby code tends to do all sorts of nasty things like  
leak memory like a row boat full of hair trigger shotguns.

4) bj, on the otherhand, simply provides a way to fire and forget  
system calls.  these system calls just may happen to use ./script/ 
runner to run some code from within your rails environment, but  
that's up to you.  it may even contact a long running daemon like  
backgrounddrb to avoid loading your rails app over and over, but  
again that's up to you.  bj does *not* load your rails app or make  
that code available in any way.  all it does is connect to the db and  
run jobs from a queue - which is another big difference: bj is a  
priority queue, you can submit 100,000 jobs and forget about it, they  
will run serially in the background until they are complete.  another  
result of the design is that you can easily fire up runners on other  
hosts using bj - thereby creating a *cluster* of machines that run  
jobs on behalf of your front end(s) rails application.  and, of  
course, it's easy for development to submit jobs into a production  
queue and vise versa.  the last major difference is that bj is  
queuing job in the database whereas backgrounddrb is dealing with  
memory/context/closures - if you have backgrounded 100k credit card  
sales and your application crashes you can probably guess where  
having the jobs live would be best ;-)  with bj the act of submitting  
a job is a db transaction that's submitted a job which can run on  
it's own two feet so you *know* once submission is complete that, no  
matter what happens next, that job is recoverable - at least to the  
extent your database/fs are.


backgrounddrb, bj, and spawn (http://wiki.rubyonrails.org/rails/pages/ 
Tom+Anderson) all serve totally different purposes.  i think bj  
provides the lowest barrier of entry into doing background rails  
processing and, in cases where the user requires a rails_env and  
needs to wrap the methodology in a ./script/runner capable script  
makes up for making the user to a little work with promising that the  
application will not start leaking memory of having network issues in  
production once the script is working from the commandline.

i have not looked at the backgrounddrb code for some time - since the  
dependancy on slave.rb was removed - so i'm positive i've made a few  
errors in the above explanation - but i'm sure ezra can correct any  
serious mistakes i've made.

kind regards.


a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being  
better. simply reflect on that.
h.h. the 14th dalai lama