From: Philip Rhoades <phil@...>
Date: 2008-06-27T16:07:35+09:00
Subject: Re: Performance improvement possible?

Ed, Ellie et al,


M. Edward (Ed) Borasky wrote:
> Eleanor McHugh wrote:
>> Regardless of what the profiler says, any program that's opening 32000 
>> files and writing to them is a prime candidate for IO optimisation...
> 
> Not to mention directory fragmentation in many filesystems. :) I'd be 
> looking at an ORM and SQLite rather than attempting something involving 
> 32,000 files.


The dir organisation is not so bad - 20 files in each dir in tree of 
50x32 dirs - however, this and previous comments from Ellie and others 
have caused me to think that more of the processing chain can be 
improved.  Although I am incurring a 90% time increase on this current 
step, the following steps have lots of awks, seds, pastes etc and it is 
VERY slow.  I think if I make use of DataMapper with SQLite (ie keep the 
DB in memory) and then massage the data with some more Ruby I can 
improve this stage still further and dramatically improve the next stage 
as well! (and also keep it in one Ruby program).  This is like a drug . .

Converting the first link in the chain, the main simulation C/C++ 
program would be a VERY big job though and I can't really justify the 
time for that at the moment although I am thinking about it . .

Thanks to all for their help!

Regards,

Phil.
-- 
Philip Rhoades

Pricom Pty Limited  (ACN 003 252 275  ABN 91 003 252 275)
GPO Box 3411
Sydney NSW	2001
Australia
E-mail:  phil@pricom.com.au