From: hemant kumar Date: 2007-12-14T02:33:47+09:00 Subject: Re: [ANN] bj-0.0.2 Hi, On Fri, 2007-12-14 at 01:53 +0900, ara.t.howard wrote: > On Dec 13, 2007, at 8:34 AM, Giles Bowkett wrote: > > > But why this instead of BackgrounDRb? > > well, backgrounddrb was originally written by ezra on top of my slave > lib, and ezra is one of the sponsors of bj so hopefully he'll chime > in with his reasons, but here are mine > > 1) much better name. gem install bj? require 'bj'? seriously giles... > > 2) backgrounddrb, afaik, is has proven to be a bit tricky for *non- > experts* to manage and use in a production environment. > > 3) backgrounddrb aims to provide a 'rubyish' environment for code to > execute in. in otherwords you call methods on on objects, serialize > ruby objects over the wire, etc. this makes entire classes of > problems easier to reason about, but it also comes with a price and > that price is complexity. for example, most (all?) people have to > think about methods like this when using drb > > remote_object.each do |thang| > thang.intense_computation > end > > now, on which cpu does 'intense' run? in which process? the answer > is that it entirely depends on how the objects where setup and how > DRbUundumped may or may not have been used. as the maintainer of > slave.rb i can tell you that the list of people who understand this > is eric hodel and, um, eric hodel. the point is that drb is not an > rpc mechanism but a toolset for building servants. using drb every > process is potentially either a client or a server and generally > both. it's the block passing mechanism that gets people into trouble > - blocks cannot go across the wire to drb does some magic to make > them work. the other issue with having a 'rubyish' environment to > execute code in, in the case of using backgrounddrb with a rails app, > is that rails' ruby code tends to do all sorts of nasty things like > leak memory like a row boat full of hair trigger shotguns. > > 4) bj, on the otherhand, simply provides a way to fire and forget > system calls. these system calls just may happen to use ./script/ > runner to run some code from within your rails environment, but > that's up to you. it may even contact a long running daemon like > backgrounddrb to avoid loading your rails app over and over, but > again that's up to you. bj does *not* load your rails app or make > that code available in any way. all it does is connect to the db and > run jobs from a queue - which is another big difference: bj is a > priority queue, you can submit 100,000 jobs and forget about it, they > will run serially in the background until they are complete. another > result of the design is that you can easily fire up runners on other > hosts using bj - thereby creating a *cluster* of machines that run > jobs on behalf of your front end(s) rails application. and, of > course, it's easy for development to submit jobs into a production > queue and vise versa. the last major difference is that bj is > queuing job in the database whereas backgrounddrb is dealing with > memory/context/closures - if you have backgrounded 100k credit card > sales and your application crashes you can probably guess where > having the jobs live would be best ;-) with bj the act of submitting > a job is a db transaction that's submitted a job which can run on > it's own two feet so you *know* once submission is complete that, no > matter what happens next, that job is recoverable - at least to the > extent your database/fs are. > > > backgrounddrb, bj, and spawn (http://wiki.rubyonrails.org/rails/pages/ > Tom+Anderson) all serve totally different purposes. i think bj > provides the lowest barrier of entry into doing background rails > processing and, in cases where the user requires a rails_env and > needs to wrap the methodology in a ./script/runner capable script > makes up for making the user to a little work with promising that the > application will not start leaking memory of having network issues in > production once the script is working from the commandline. > > i have not looked at the backgrounddrb code for some time - since the > dependancy on slave.rb was removed - so i'm positive i've made a few > errors in the above explanation - but i'm sure ezra can correct any > serious mistakes i've made. > I am maintaining backgruondrb in these younger days. And things have changed. Its written on top of event driven networking lib ( packet ) that i wrote. There are no threads anywhere. Everything is even driven, it still has real processes, but those have reactor loop of their own. I wrote a custom protocol for internal communication between workers and it works reasonably well. I agree that, bj , spawn they all have different purpose. If you find time, please look into code base and suggest any problems that you find. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org