From: Robert Klemme <bob.news@...>
Date: 2005-09-11T18:46:31+09:00
Subject: Re: Optimizing a single slow method

Glenn M. Lewis <noSpam@noSpam.com> wrote:
> Hi Robert!
>
> Yes... I found out that if I create a hash of contract names
> that I've already decided to ignore, then I can immediately test
> field[0] to see if I can totally ignore the line, like you had
> suggested.
> Oh, sorry... the only thing I was referring to was not being able
> to speed up the File I/O and parsing in another language...

I was probably not clear enough: I intended to convey that the pure IO part 
might not really speed with another language (the PL introduced overhead is 
usually small compared to the actual IO part).

> and I only
> said that because I had just finished another Ruby project where the
> original file parsing of about 80Megs of data would take over 10
> minutes, and when I rewrote the File I/O and parsing in C++, I could do 
> all of it in
> less than a second.

You're really making me curious... :-)

> But that is neither here nor there because the other optimizations
> recommended brought down the load time to 26 seconds, which is now
> acceptable and no longer worth the pain to create a C++ module that
> will communicate with Ruby in both directions.

Here's another suggestion that might have a small impact only: use the bang 
version of chomp:

File.open(file, "rb") do |io|
  io.each_line do |line|
    line.chomp!
    fields.split(/,/)
....

This saves you one object creation per line read.  If lines are short it 
might even make a noticeable difference.

> No, I never tried the DateParse... as Eric recommended that I no
> longer use Date, but switch over to Time... he was right... I got a
> significant speedup with switching to the Time class.

Cool!

> Thanks for your help!

You're welcome!

Kind regards

    robert