From: Ashley Moran <work@...>
Date: 2006-07-29T05:33:11+09:00
Subject: Re: For performance, write it in C

On Thursday 27 July 2006 10:13, Ashley Moran wrote:
> > Good luck. I recently got a Ruby program that aggregates several LDAP
> > directory-pulls with about a million entries down from a few hours to a
> > few seconds, without having to drop into C. It can be done, and it's
> > kindof fun too.
>
> Next time I get a morning free I might apply some of the tweaks that have
> been suggested.  Might be interested to see how much  I can improve the
> performance.


I looked at the source of the script today, and I made these changes:

- use FasterCSV instead of CSV
- don't buffer every row in the datasets, send them straight to 
Zlib:GzipWriter as they are processed
- don't do hash lookups in the middle of a 15 million row loop, do them once 
in advance!

Unfortunately I'm still stuck with a nasty "rows.each { |row| row.each { |col| 
col.strip! } }" type section, to fix the poor quality of the data, which 
would take a lot of time going through all the fields to thin out.

Despite this, I've got the run time down from over 2.5 hours to about 50 
minutes.  The smaller files are individually about 6x faster, but I'm happy 
with 3x faster overall.  It means we can realistically run it in the day if 
there are issues.

One curious thing is that while the real time was about 50 mins, the user time 
was only about 30 mins (negligible sys time if I remember).  Not sure where 
the other 20 mins has gone?

Ashley

-- 
"If you do it the stupid way, you will have to do it again"
  - Gregory Chudnovsky