From: "Glenn M. Lewis" <noSpam@...>
Date: 2005-09-17T15:06:36+09:00
Subject: Re: Optimizing a single slow method

Just FYI for anyone following this thread...
I decided to go ahead and rewrite the file parsing
section of the code in C++ and interface it to Ruby
via SWIG (1.3.25), just because I thought the extra
effort would pay off in the long run with the ability
to perform a lot more experiments if it took less time
to run.

Now, with my current database of 16 files totalling
about 13Megs of information, it takes:

Ruby Only:  29 seconds
Ruby/C++ :  <2 seconds

So by doing this, I got approximately a 15x speedup.

Please don't misunderstand... I'm *NOT* complaining!!!
I'm tickled pink that I can use Ruby as a front-end
for all the cool processing that I want to do on
the data after I load it in!!!

I only posted this info for those of you who are
wondering if you can benefit from putting heavy-duty
I/O in a compiled language... yes you can.  :-)

-- Glenn

Glenn M. Lewis wrote:
> Yes indeed!  Thank you very much, Hugh, and all
> others who have helped!  The processing time is
> now a flat 26 seconds, which sure beats the original
> 9 minutes!!!
> 
> To summarize, the three most significant speedups:
> 1) Only process 20% of the lines based on a quick check
>    of relavency
> 2) Switch from using 'Date' class to 'Time' class
> 3) Minimize substring processing
> 
> Thanks again to all who have helped!
> -- Glenn
> 
> Hugh Sasse wrote:
> 
>> Did they help at all?
>>         Hugh
>>
>>