From: MenTaLguY <mental@...>
Date: 2008-02-08T05:42:31+09:00
Subject: Re: is lots of files with Threads faster?

On Fri, 8 Feb 2008 05:21:51 +0900, Chris Richards <evilgeenius@gmail.com> wrote:
> Im required to open 50+ files and parse the data in them.  WOuld using
> multiple threads give me the best performance?  or is it best just to do
> it sequentially?

There's the same amount of IO bandwidth to go around no matter how many
threads you throw at the problem (and in practice if you add more threads you
start wasting bandwidth due to seeking and other overhead).  Given that,
it's almost always best to do things sequentially.

If you are using a native-threaded runtime (e.g. JRuby), and you can prove
that you aren't consuming most of the available IO bandwidth yet (e.g. because
parsing is taking longer than the IO), then _maybe_ consider using multiple
threads, but then you need to be careful to only use enough to consume the
available IO bandwidth and no more.  If you want to use your IO bandwidth most
effectively, asynchronous IO (e.g. with libev, etc.) is often a better idea.

-mental