From: Robert Klemme <shortcutter@...>
Date: 2008-02-08T19:17:40+09:00
Subject: Re: is lots of files with Threads faster?

2008/2/7, MenTaLguY <mental@rydia.net>:
> On Fri, 8 Feb 2008 05:21:51 +0900, Chris Richards <evilgeenius@gmail.com> wrote:
> > Im required to open 50+ files and parse the data in them.  WOuld using
> > multiple threads give me the best performance?  or is it best just to do
> > it sequentially?
>
> There's the same amount of IO bandwidth to go around no matter how many
> threads you throw at the problem (and in practice if you add more threads you
> start wasting bandwidth due to seeking and other overhead).  Given that,
> it's almost always best to do things sequentially.

... unless all files reside on different IO devices in which case
parallel reading *can* be faster than sequentially.  If they are on
the same filesystem I'd certainly prefer to read them sequentially.
There might be a slight performance gain by decoupling reading,
parsing (and probably output) into different threads.  But that mostly
depends on IO speed and processing complexity and the slowest part
determines throughput - no matter what.

> If you are using a native-threaded runtime (e.g. JRuby), and you can prove
> that you aren't consuming most of the available IO bandwidth yet (e.g. because
> parsing is taking longer than the IO), then _maybe_ consider using multiple
> threads, but then you need to be careful to only use enough to consume the
> available IO bandwidth and no more.  If you want to use your IO bandwidth most
> effectively, asynchronous IO (e.g. with libev, etc.) is often a better idea.

Good points.

Cheers

robert

-- 
use.inject do |as, often| as.you_can - without end