From: Paulus Esterhazy <pesterhazy@...>
Date: 2005-11-21T07:05:47+09:00
Subject: Re: Pipelined Processing

Robert Klemme schrieb:
>> Attached the ParallelEnumerate class along with a trivial test case.
>> Bear with me, it's my first ruby script.
> You try to tackle multithreading with your first script?  Wow!  I don't
> exactly understand why you use a thread per collection just to fill a
> queue. Maybe I'm missing something here but it looks a bit strange.  Are
> you sure this actually downloads in parallel?

Yes it works for the purpose. The "collections" look like this:

class Source
  includes Enumerable
  def each
    @data = open_url("http://...").read
    while true
      element = get_next_element
      yield element
    end
  end
  def next_element
    # process @data
  end
end

It would probably have been cleaner to do this using a thread pool for
downloading the pages and processing the data in the main thread, as you
suggest - seperating the stages. I used an enumerator because it's
convenient - I wrap the enumerator in a pseudo IO object which I return
to webrick (which expects an object that supports the method "read").
That way, I get a kind of simple asynchronous data processing.

> 
> If I would have done this I'd taken a different approach (but maybe I'm
> missing some of your requirements): I'd create a queue which receives
> URL's (or whatever tasks you have).  Then I'd set up n threads (n>0,
> probably depending on user input) and each thread reads elements from
> the queue and processes them in parallel.  You might as well combine
> both approaches, i.e. if a URL has been downloaded, the content is
> pushed onto another queue from which another number of threads (possible
> just 1) reads and processes.

Thanks for the comment,
Paulus