From: ara.t.howard@...
Date: 2006-05-28T06:40:26+09:00
Subject: Re: Ruby Threads...

On Sun, 28 May 2006, Francis Cianfrocca wrote:

> You're making a very interesting point, one I've made many times: you're
> saying to write cooperative multiprocess rather than multithreaded programs.
> If you take aggregate costs into account (including time-to-market and
> lifecycle maintenance and support), this approach can be far better than
> multithreaded because it's so much more robust and easier to do.

i agree totally.

> Whether it's as fast, however, is a highly hardware and OS-dependent
> question. If you can specify multiprocessor or multicore hardware,
> multiprocess software design has a clear edge, IMO. And in a few years
> nearly all processors for general computation will be multicore.

true.  but, for me, it's totally moot.  'fast' for me requires 30-50 nodes.
right now we are doing some processing on 30 nodes that will last 5 days.
each node has 4 cpus.  so, when compared to a single processor cpu that
something like 600 days of processing on wall-clock-time.  whether or not the
code takes 30 minutes or 34 is largely besides the point.  the thing is to get
the jobs out there, using rq (ruby queue), and then to spread them across
cpus.  my approach is to keep the code simple and, when it's cpu bound spread
it out across the cluster.  saves brainpower.  also, we can add 10 nodes to
our cluster in about 2 hours.  i can't fix that many bugs in that time... so i
prefer to use brute force and be simple/stupid about such things.

> (This is a side point (and as we know, the side points always generate the
> hottest flames), but I happen to disagree with your choice of DRb. Not
> because of the communications model, but because distributed objects are
> fundamentally problematic. I'd encourage you to look at multiprocess
> event-driven systems. Watch for the upcoming pure-Ruby version of the
> eventmachine library on Rubyforge- it will have built-in constructs to
> explicitly support multiprocess event-driven programming.)

the problem here is the same as with clustering - it's easy to send
events/jobs around - it's the data that's hard.  i'd argue that an average
programmer working on a difficult multi-processing task could accomplish it
much faster using drb than events/signals, etc.  this is because state and
data become very, very important with logically difficult tasks and drb makes
this trivial to manage in an atomic way.  i don't like handling events one way
(signals, kpoll, whatever) and data (and atomic access to it) in two ways.  my
mind is feeble so doing it the braindead way lets me get it done now, get it
out on the cluster, let it run for three days before noticing a mistake, and
then to repeat that about 3 or 4 times (seriously, our stuff will have 100s of
config params so we almost never get it right the first time).  but, i'll
acknowledge that event driven programming is good for many applications and
that your work on even machine is certainly appreciated.

cheers.

-a
-- 
be kind whenever possible... it is always possible.
- h.h. the 14th dali lama