From: Paul Lutus <nospam@...>
Date: 2006-11-16T08:50:06+09:00
Subject: Re: *Fast* way to process large files line by line

devesh wrote:

/ ...

> I still don't see the reason why doing things in parrallel would screw
> things up ?

If you read from more than one file simultaneously, the drive heads must
constantly slew back and forth from the location of one file to another.
The more files open, the worse this becomes. Files are laid out on a drive
in a logical way, and the data should be read in that same logical way --
one file at a time.

Imagine shopping for groceries and washing your car simultaneously. A lot of
moving back and forth to do them at once, don't you think? Maybe washing
your car, then shopping, would be a better use of your time.

> As my assumption is that the disk head is anyway pretty 
> wild as the system will involve a lot of IO b/w context switches.

No matter how bad it is, you can always make it worse by opening multiple
files.
> 
> ii. Also why do you think that writing another file in the midst of
> reading (one or more) one is a bad idea ?.

Same reason -- disk thrashing.

-- 
Paul Lutus
http://www.arachnoid.com