From: ara howard <ara.t.howard@...>
Date: 2008-02-11T02:02:49+09:00
Subject: Re: is lots of files with Threads faster?


On Feb 10, 2008, at 5:19 AM, Francis Cianfrocca wrote:

> I basically gave up on optimizing hard-disk I/O long ago. (In
> Ruby/EventMachine, I started adding an event-driven interface for disk
> files, and will probably complete it someday, but initial profiling  
> showed
> relatively little benefit.)
>
> A big part of the problem is that different machines have different
> controller hardware, with a wide variance not only in raw  
> performance, but
> also in caching strategies and in the way they schedule the physical  
> seeks.
> Multispindle systems change the behavior yet again. You can develop  
> on one
> machine hoping to get some level of performance improvement, and  
> find a
> totally different behavior when you go to production.

good advice.  i've had quite a bit of experience optimizing large  
scale processing (really large) and seen that there is always an  
optimal io/cpu usage pattern (two processes per cpu in dual-cpu  
machines with dual disk controllers, etc) but also that it is *always*  
specific to the exact hardware setup.  i agree that it's mostly  
impossible to try to come up with a generic solution.

cheers.

a @ http://codeforpeople.com/
--
share your knowledge.  it's a way to achieve immortality.
h.h. the 14th dalai lama