From: schneik@...
Date: 2000-03-27T15:47:48-06:00
Subject: [ruby-talk:02197] Re: Scripting and OO -- thought question




David Douthitt writes:
> I have a set of applications that were written
> in Perl 4, which scan the UNIX system logs and generate color-coded
> HTML pages for them.  Whether using Perl or Ruby, they take a LONG
> time - especially for an application which runs every five minutes.
> They also suck an incredible amount of CPU time, slowing everything
> done noticibly (including terminal response time).
>
> All they do is scan the log (41,000 lines plus) and generate HTML
> files based on them.

Yes, but HOW do they do this?

> At one time I had them (Ruby version, Perl
> version) generating separate files for each system in the log;
> when I switched to using ksh and grep, the speed increase was incredible.

Again, HOW was this done? (Also recall that you already know that Ruby can
invoke grep too.) And what is incredible here? Was the speedup 50%? 100%?
500%?

> I'm still stuck though, since scanning for one particular host
> (with 41,000 lines!) can take over 3 minutes.

That doesn't seem (wild guess here) large enough to cause problems if you
are running on a moderately fast machine. Is the run time pretty much a
straight linear function of the number of lines of input?

> The application is quite simple really (two pages in Ruby) but
> the speed is in the tank.  Time for GNU Smalltalk?  Scheme?
> Eiffel?  Don't know.... still looking (and wanting to learn
> something new!)

The basic problem here is that if you don't show people your (suitably
sanitized if necessary) code (or at least representative critical pieces of
your code), the answers you get to such questions will be based largely on
people's imagination, which may or may not have anything to do with the
most relevant factors in this case. (Maybe you should have been using Perl
5 with compiled regular expressions instead of Perl 4. Maybe you were
somehow doing unnecessary or unbuffered I/O without realizing it. Maybe
your were building some sort of table or index for your HTML output that
inadvertently did something in an O(n**2) fashion. Maybe you stashed
everything in memory on a machine with insufficient RAM. Maybe any number
of other things....)

Conrad Schneiker
(This note is unofficial and subject to improvement without notice.)