Re: [ANN] ruby queue : rq-0.1.2

On Mon, 16 Aug 2004, Shashank Date wrote:

> Not right away ... but I will try and get a 2-node configuration working
> over the (already crowded ;-) week-end. No promises though.
>
> If I succeed, I will have Win XP (Home) on one node and SuSE 8.2 on the
> other. But please go ahead and email the script which determines if it works
> or not.

i'll tar it up and send it your way.  any chance you could try compiliing this
on windows?

   http://raa.ruby-lang.org/project/posixlock/

i don't have access to a windows machine with a compiler toolchain and don't
even know if windows offers a posix fctnl - but i'm hoping it does.  sqlite
compiles on windows so it must - but i may have to add some #ifdefs to that
code to get it working...  i should also add that you can simply pack a struct
to get fcntl working (thanks matz for this pure ruby solution) so a c
extension is not strictly needed for access to fcntl locking.  however, some
form of it is required so we should probably look into that pronto.  i have an
RCR out there (i think) for posixlocking in ruby but havn't had time to pursue
it - the lack of it is a problem IMHO....


> Hmmm... then may be I have mis-understood the nature of your project.  I was
> thinking of using it for CPU intensive (eg. image analysis) jobs over a
> cluster of heterogenous (in terms of CPU power, O.S. and in terms of
> functionality) nodes and wanted the ability to "farm" (push) work requests
> to least busy CPUs (provided there is a way to determine that, of course).
> Phil Tomson's TaskMaster comes to mind. See:
> http://raa.ruby-lang.org/project/taskmaster/
>
> [Phil: if you are reading this, will love some feedback from you:
> specifically are you planning to work on it in near future? Or is the
> project closed?]
>
> From your describtion above it appears to me that work will be "fetched"
> (pulled) by least busy CPUs. Am I correct? (We can take this discussion off
> line if it starts becoming [OT]).

exactly.  the flow is something like

   def feed
     daemon do
       loop do
         start_jobs until busy?
         reap_jobs
       end
     end
   end

but obviously a bit more compilcated.  the 'busy?' method only returns true if
a predefined number of jobs are already running (we set it to two for dual CPU
nodes) but i've got plans for this method to hook into resource monitoring so
a node may become 'busy' if some critical resource is maxed and, of course, so
that resources may be requested.

this approach totally avoids needing a scheduler since, as you correct state,
jobs are 'fetched' from the queue and the strongest fastest node simply get
more work done.  it is working __really__ good for us - we see our node
performance line up exactly as we would have predicted it as the number of
jobs grows.

i think this approach should work very well for the scenario you describe.
i'm assuming you've also checked out technologies like condor, sge, etc?  all
the software i looked at was extremely bloated, full of complexity bugs, and
didn't support one of the main features i'll eventually need (full boolean
resource requests) which is what lead me down this path.  if you get a chance
send a message offline (on online i suppose if it's relevant) detailing your
planned processing flow if you don't mind.  it's handy to know how people
might use your software since, sometimes, choices can be arbitrary and this
knowledge might help me make better ones.

cheers.

> This is very likely to be extremely specific to the problem we are trying to
> solve and may also be proprietory. I hope your licensing terms will permit
> me that.

ruby's license - so it should.


kind regards.

-a
--
===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: [phone]
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it. 
|   --Dogen
===============================================================================