From: Gavin Sinclair <gsinclair@...>
Date: 2002-11-29T00:05:37+09:00
Subject: A lesson learned

Folks,

Today I rediscovered the truth of that very important saying in software
developement: "Make it work, make it right, then make it fast."

In my project at work, I spent too much time trying to perform file scanning
operation fast: by reading it backwards until it determined that it didn't need
any more information, then committing to the database.

I've now done what I should have in the first place: read the file forwards,
grabbing *all* the information, and then updating the database as necessary.
This involves grabbing potentially 99% redundant information from a HUGE file,
then comparing against lots of info in the database before knowing what to
commit.  It's a dumb approach, but it's more reliable (no dependence on smarts
to determine when to stop scanning), and more flexible (can run it on old files
usefully and safely).  In the end, it was these features I needed, so I
scrapped my "optimised" approach that I had spent perhaps a week on a few weeks
ago.

Furthermore, the same degree of "optimisation" can be achieved by controlling
an external factor: restarting a certain process every night so the log file
doesn't grow too large.  Then the original "need" to read it backwards (itself
an expensive task) vanishes.

Oh well, lesson learned.  And although this has nothing directly to do with
Ruby, I share it herre in the hope that by warning others, I can balance my
karma and not have it happen to me again :)

Cheers,
Gavin