From: Robert Klemme <bob.news@...>
Date: 2005-11-21T06:27:23+09:00
Subject: Re: Pipelined Processing

Paulus Esterhazy <pesterhazy@gmx.net> wrote:
>>> this came up recently on IRC: question was how to chain processing
>>> so that each step runs concurrently to other steps.  While I don't
>>> see real benefit as long as there are no native threads in Ruby I
>>> played around a bit and this is the result (attached).  There's
>>> certainly rool for improvement.  Do with this whatever you like.
>
> Hello,
>
> it's funny you should bring this up because I just cooked up something
> similar (albeit less sophisticated and robust). What I needed was
> several tasks being carried out in parallel.

It's not really similar.  While my program executes several stages of 
processing in parallel you are actually doing similar things in parallel. 
At least that's what I understood from your explanation and code.

> In this particular
> instance I download several pages using open-uri and output a chunk
> to webrick as soon as its processing is finished (as responsiveness
> is critical). For network I/O, ruby's pseudo-threads seem to work
> well, though I agree native threads would be much better.

True, as soon as slow IO is involved ruby threads do ok as long as the 
processing doesn't use too many resources.

> Attached the ParallelEnumerate class along with a trivial test case.
> Bear with me, it's my first ruby script.

You try to tackle multithreading with your first script?  Wow!  I don't 
exactly understand why you use a thread per collection just to fill a queue. 
Maybe I'm missing something here but it looks a bit strange.  Are you sure 
this actually downloads in parallel?

If I would have done this I'd taken a different approach (but maybe I'm 
missing some of your requirements): I'd create a queue which receives URL's 
(or whatever tasks you have).  Then I'd set up n threads (n>0, probably 
depending on user input) and each thread reads elements from the queue and 
processes them in parallel.  You might as well combine both approaches, i.e. 
if a URL has been downloaded, the content is pushed onto another queue from 
which another number of threads (possible just 1) reads and processes.

Kind regards

    robert