From: ara.t.howard@...
Date: 2006-03-23T15:55:05+09:00
Subject: Re: Ruby IPC In an OpenMosix Cluster

On Thu, 23 Mar 2006 the.liberal.media@gmail.com wrote:

> Ok, so I went back and actually read through the entire rq article this time
> (and noticed who wrote it -- many props Ara :).
>
>> From what I understood, you're suggesting something like this:
>
> 1. Use dirwatch to wait for incoming data (files) on an NFS exported
> dir
> 2. Inject jobs into rq for each incoming file
> 3. rq executes commands on each node that read in each file from the
> NFS mount
>
> How fast is a setup like this?  I would think there would be a lot of
> overhead in forking processes for each job, and even more in the
> NFS/file IO.  We're shooting for 100 jobs/second, starting with a
> fairly small cluster and then scaling up.  Each piece of data is 4-10k.

yup this would defintely push the limits __unless__ you can batch them.  i'm
actually working with a group now that will be injesting data at almost that
exact same rate.  in there case it's sufficient to bundle jobs up  - higher
latency but also hight throughput.  so basically a dirwatch would watch an
incoming directory and, perhaps once per minute, scan the directory and submit
bunches of 500 files for processing.  for something really simply you might
not even need dirwatch - just move files to another directory once they've
been submitted - eg. sweep the directory as you submit.  rq now supports
providing stdin (and saving stdout and stderr) so submitting jobs that process
500 files is really easy.  anyhow, nfs scales pretty dang well on gigE with
tuned tcp/ip stacks and fast disk, we certainly abuse ours with little
problems.  however, we also use vsftpd to access data and this is very easy to
setup and extremely fast to use - about as fast as you can get.  in any case
you'll have the same issue with any cluster: distributing jobs is easier than
distributing data.  for instance, many of our inputs are 3gb-600gb - one has
to be careful with this sort of payload!  ;-)

> My thought was to spawn a pool of processes once, then start feeding them
> the data via [unknown IPC].  Seems like that would be a faster solution as
> long as openMosix is efficient in redirecting the IO across nodes.  Of
> course, this may be a development nightmare (learning experience), since
> neither my team nor I have a lot of experience with multiprocessing.

i did look into this a bit - if i recall openmosix makes io transparent and
can migrate process memory from node to node... in our case this would be a
disaster : code must follow data and not the other way around.  from what i
know at the moment i can't imagine that kernel level io network multiplexing
would be faster than pulling data across and sending it back in huge chunks...
the resident network expert here seems to think people need to move that way
to optimize newer networks with things like jumbo frames.  but i can't say for
certain i'm just a hacker!

> If rq would satisfy our speed requirements, then I would love to avoid the
> extra development time.  Perhaps we'll just have to build a basic prototype
> and run some tests. :)

yes.  we ran a simulation here, just prducing false input and running a busy
loop for a few seconds to appriximate the system that was similar to yours and
it seemed fine.  however they are not in production yet so i cannot say for
sure.  the good part is that all the bits are free and it should only take a
few hours to mock something up - even on a stock linux machine with no root
privs...

let me know if you go this route as i have upgrades to both dirwatch and rq
that you'll surely want.

regards.

-a
-- 
share your knowledge.  it's a way to achieve immortality.
- h.h. the 14th dali lama