From: "Ara.T.Howard" Date: 2004-05-13T04:33:53+09:00 Subject: Re: PStore vs. YAML::Store On Thu, 13 May 2004, Kirk Haines wrote: > On Thu, 13 May 2004 01:53:53 +0900, Ara.T.Howard wrote > > > however, it __would__ be very cool if i could use it: the project is > > a distributed job manager (grid) that makes use of an on nfs disk > > priority job queue (pstore or yamlstore and accessed safely by means > > of my Lockfile class) and n consumer processes competing to run jobs > > from the queue. the advantage of using yaml is that standard tools > > (eg. grep) might be used to check for the status of jobs... also it > > might make possible the construction of tool sets that utilize each > > others stdout/in a bit easier, eg. > > Ara, I'm working on something similar right now, using a different > architecture. > > I'm using a combination of DRb, Rinda/RingServer, and a TupleSpace to > receive a queue of jobs that the consumers processes then take off of the > tuplespace. I'm planning on coupling the tuplespace to a persistent store > so that if something happens to the process that holds the tuplespace, the > queue isn't lost and can be restored when the tuplespace server restarts. that's too funny - i was just out running and thinking of __exactly__ this scheme. actually i've been thinking of it for quite a while; the hang up is that opening ports here is a ***ing nightmare and everytime the sysads upgrade they forget to re-open it. my goal is to create clustering software that one can have up and running in under five minutes that depends on nothing except for having an NFS mounted directory... we do a lot of processing where we pool 10 or 15 nodes together for a few weeks/months and write custom scripts driving ssh commands to 'cluster' them... this actually works really good: right now i've got 36 nodes running jobs that take 5-10 days each. i've looked (extensively) into SGE (sun grid engine) for this purpose for the last few months, but even the new beta6.0 has limitations that, IMHO, make is useless for our purposes (this revolves around resource monitoring/requests - see SGE mailing list for threads i've been having there). in any case, your plan sounds really neat. one thing to consider is that it introduces a single point of failure - if the master nodes blows a disk your entire cluster will grind to a halt... SGE uses something called a shadow master for this purpose (it's basically hot backup). i think SGE might have a lot of good ideas that apply to this type of arch if you haven't already looked at it. one of things i like about my approach is that if one one single node remains running the cluster works, there is no central point of failure (well, the NFS server but all the data resides there so we're dead if it's down anyhow). do you know if anyone has done anything to make a tuplespace highly available - like having two nodes able to serve the space with some sort of negotiation between the two in the case of failure? that would be really cool. > In my tests so far it seems to be a stable and flexible way to queue jobs > since I can make use of objects in the queue to pass information from where > the job is generate to the consumer process. It, however, has the same > problem that your PStore based queue does in that simple tools like grep can > not be used to check the status of the queue. Specialized tools have to be > written to do that. yeah - i'm there now too. > P.S. Is the weather as cold and wet down in Boulder today as it is up here > in Wyoming? Good day to be inside and writing some Ruby code, I think. yes - my hands are so cold from my run that i can barely type! where are you in wy? cheers. -a -- =============================================================================== | EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov | PHONE :: 303.497.6469 | ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328 | URL :: http://www.ngdc.noaa.gov/stp/ | TRY :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done ===============================================================================