From: ara.t.howard@...
Date: 2006-09-30T00:44:54+09:00
Subject: Re: Forking job scheduler

On Sat, 30 Sep 2006, Francis Cianfrocca wrote:

> Sounds cool, Ara. How does it keep two copies of itself from running? Does
> it flock a file in /var/run or something like that?

yeah - basically.  it's under the users home dir though, named after the
queue.  the effect is 'one feeder per host per user' by default.  it really
works nicely because you can have a daemon process totally independent of
system space and without root privs.  dirwatch works the same way.  here's my
crontab on our nrt system:

   mussel:~ > crontab -l
   leader              = /dmsp/reference/bin/leader
   worker              = /dmsp/reference/bin/worker
   env                 = /dmsp/reference/bin/bashenv
   shush               = /dmsp/reference/bin/shush
   dirwatch            = /dmsp/reference/bin/dirwatch
   nrt                 = /dmsp/reference/bin/nrt
   nrtq                = /dmsp/reference/bin/nrtq
   nrtw                = /dmsp/reference/bin/nrtw
   nrts                = /dmsp/reference/bin/nrts
   beveldevil          = /dmsp/reference/bin/beveldevil
   sfctmp1p0           = /dmsp/reference/bin/sfctmp1p0
   afwa_watch          = /dmsp/nrt/dirwatches/data/incoming/afwa/dirwatch
   subscriptions_watch = /dmsp/nrt/dirwatches/subscriptions/dirwatch
   dmsp_watch          = /dmsp/nrt/dirwatches/data/incoming/dmsp/dirwatch
   night_files_watch   = /dmsp/nrt/dirwatches/data/incoming/night_files/dirwatch
   mosaic_watch        = /dmsp/nrt/dirwatches/data/incoming/mosaic/dirwatch
   www                 = /dmsp/nrt/www/root/
   qdb                 = /dmsp/nrt/queues/q/db
   show_failed         = /dmsp/reference/bin/show_failed


   #
   # mussel is the current leader
   #

   */15 *  * * * $leader $env $shush $afwa_watch start
   */15 *  * * * $leader $env $shush $subscriptions_watch start
   */15 *  * * * $leader $env $shush $dmsp_watch start
   */15 *  * * * $leader $env $shush $night_files_watch start
   */15 *  * * * $leader $env $shush $mosaic_watch start
   */15 *  * * * $leader $env $shush $beveldevil
   59   23 * * * $leader $env $shush $nrtq rotate

   #
   # clam, oyster, bismarck, scallop, shrimp are current workers
   #

   */15 * * * * $worker $env $shush $nrtq start


this same crontab is installed across our nrt cluster.  basically one node
runs a bunch of dirwatchs which trigger submits to the master queue.  the
workers, for their part, are completely stupid, all the have is a user account
and the '$worker' crontab entry that keeps a feeding process running at all
times, even after reboot.  it's a simple was to setup durable userland
daemons.

($leader and $worker are xargs style programs - $leader obviously only
executes it's command line if run on the leader, vise verse for worker)

regards.

-a
-- 
in order to be effective truth must penetrate like an arrow - and that is
likely to hurt. -- wei wu wei