From: the.liberal.media@...
Date: 2006-03-23T12:58:53+09:00
Subject: Re: Ruby IPC In an OpenMosix Cluster

Ok, so I went back and actually read through the entire rq article this
time (and noticed who wrote it -- many props Ara :).

>From what I understood, you're suggesting something like this:

1. Use dirwatch to wait for incoming data (files) on an NFS exported
dir
2. Inject jobs into rq for each incoming file
3. rq executes commands on each node that read in each file from the
NFS mount

How fast is a setup like this?  I would think there would be a lot of
overhead in forking processes for each job, and even more in the
NFS/file IO.  We're shooting for 100 jobs/second, starting with a
fairly small cluster and then scaling up.  Each piece of data is 4-10k.

My thought was to spawn a pool of processes once, then start feeding
them the data via [unknown IPC].  Seems like that would be a faster
solution as long as openMosix is efficient in redirecting the IO across
nodes.  Of course, this may be a development nightmare (learning
experience), since neither my team nor I have a lot of experience with
multiprocessing.

If rq would satisfy our speed requirements, then I would love to avoid
the extra development time.  Perhaps we'll just have to build a basic
prototype and run some tests. :)

Best,
Dan