Re: PStore vs. YAML::Store

On Thu, 13 May 2004, Kirk Haines wrote:

> On Thu, 13 May 2004 01:53:53 +0900, Ara.T.Howard wrote
>  
> > however, it __would__ be very cool if i could use it: the project is 
> > a distributed job manager (grid) that makes use of an on nfs disk 
> > priority job queue (pstore or yamlstore and accessed safely by means 
> > of my Lockfile class) and n consumer processes competing to run jobs 
> > from the queue.  the advantage of using yaml is that standard tools 
> > (eg. grep) might be used to check for the status of jobs... also it 
> > might make possible the construction of tool sets that utilize each 
> > others stdout/in a bit easier, eg.
> 
> Ara, I'm working on something similar right now, using a different 
> architecture.
> 
> I'm using a combination of DRb, Rinda/RingServer, and a TupleSpace to
> receive a queue of jobs that the consumers processes then take off of the
> tuplespace.  I'm planning on coupling the tuplespace to a persistent store
> so that if something happens to the process that holds the tuplespace, the
> queue isn't lost and can be restored when the tuplespace server restarts.

that's too funny -  i was just out running and thinking of __exactly__ this
scheme.  actually i've been thinking of it for quite a while; the hang up is
that opening ports here is a ***ing nightmare and everytime the sysads upgrade
they forget to re-open it.  my goal is to create clustering software that one
can have up and running in under five minutes that depends on nothing except
for having an NFS mounted directory... we do a lot of processing where we pool
10 or 15 nodes together for a few weeks/months and write custom scripts
driving ssh commands to 'cluster' them... this actually works really good:
right now i've got 36 nodes running jobs that take 5-10 days each.  i've
looked (extensively) into SGE (sun grid engine) for this purpose for the last
few months, but even the new beta6.0 has limitations that, IMHO, make is
useless for our purposes (this revolves around resource monitoring/requests -
see SGE mailing list for threads i've been having there).  in any case, your
plan sounds really neat.  one thing to consider is that it introduces a single
point of failure - if the master nodes blows a disk your entire cluster will
grind to a halt... SGE uses something called a shadow master for this purpose
(it's basically hot backup).  i think SGE might have a lot of good ideas that
apply to this type of arch if you haven't already looked at it.  one of things
i like about my approach is that if one one single node remains running the
cluster works, there is no central point of failure (well, the NFS server but
all the data resides there so we're dead if it's down anyhow).  do you know if
anyone has done anything to make a tuplespace highly available - like having
two nodes able to serve the space with some sort of negotiation between the
two in the case of failure?  that would be really cool.

> In my tests so far it seems to be a stable and flexible way to queue jobs
> since I can make use of objects in the queue to pass information from where
> the job is generate to the consumer process.  It, however, has the same 
> problem that your PStore based queue does in that simple tools like grep can 
> not be used to check the status of the queue.  Specialized tools have to be 
> written to do that.

yeah - i'm there now too.

> P.S. Is the weather as cold and wet down in Boulder today as it is up here
> in Wyoming?  Good day to be inside and writing some Ruby code, I think.

yes - my hands are so cold from my run that i can barely type!  where are you
in wy?

cheers.

-a
-- 
===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: [phone]
| ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
| URL     :: http://www.ngdc.noaa.gov/stp/
| TRY     :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done 
===============================================================================