From: Austin Ziegler <halostatue@...>
Date: 2005-02-04T02:47:24+09:00
Subject: Re: yarv and dbi

On Fri, 4 Feb 2005 01:08:09 +0900, Michael Neumann <mneumann@ntecs.de> wrote:
> Austin Ziegler wrote:
> > On Thu, 3 Feb 2005 11:18:18 +0900, jm <jeffm@ghostgun.com > wrote:
> >
> >>Processing flow data from a router. The script caches user information
> >>from a database then processes a 25-30MB flow file captured from a
> >>router using flow-tools. This takes about 5 minutes on a 2.4GHz pentium
> >>4 unloaded by any other process running at 98% utilisation
> >>continuously. To put this into context each flow file is only 15
> >>minutes worth of data and the current perl version does it in a bit
> >>over 2 minutes. This perl version is showing it's lack of design in a
> >>variety of ways including the nightmare of trying to add features it
> >>was never designed to support. So this was a good opportunity to
> >>rewrite it in ruby to make it more maintainable, etc.
> >>
> >>While 5 minutes in within the time constraint that is on an unloaded
> >>machine and the machine it's destined for has other processes sharing
> >>the CPU.
> >
> >
> > Could your script be doing things that could be improved in performance?
> 
> Well, I think Ruby/DBI is not the fastest. It parses (and splits) each
> SQL statement, then joins it back into a string, even if you don't use
> '?' parameter markers. This should be delayed and omitted if no
> parameters were given. Not sure whether this is the reason for the slowness.
> 
> Regards,
> 
>    Michael
> 
> 


-- 
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca