From: Srinivas Jonnalagadda <srinivas.j@...>
Date: 2005-11-07T12:51:13+09:00
Subject: Re: Help requested -- regexp

William Ramirez wrote:
> Personally, and maybe because I'm a product of different experiences, I'd
> approach this problem differently. I see your problem and it just screams to
> me "database".
> 
> Is there a reason you couldn't parse the text file and upload it to some
> sort of SQL database and then let your users query that?
> 

Yes.  Each dump file is between 800 MB and 1.4 GB.  I had indeed setup
a database to hold this data.  Here are a few reasons why I tried the
approach that I did:

1. The 'load' (with all relational constraints turned off) was still
    taking enormous amount of time (of the order of 3-4 hours per dump
    file).

2. The dump file's schema changes rather frequently.  This induced
    frequent DBA overhead into this process, to keep the database
    schema synchronized.

3. The sparse nature of the data (not all fields being mandatory in
    all records) has resulted in a high storage overhead (about 2.5X).

4. SQL queries on the resulting database are not a serious option
    since my users do not know SQL, and would not be able to interpret
    any error diagnosis.

5. When wrapped with objects in Ruby, the same queries take 6-10X
    longer (as compared to the regular expression approach).  Profiling
    shows this to be mostly because of object creation/initialization
    overhead.

    And, to answer James' question -- it was indeed a hash that was
    employed in each object.

So, I sought a solution that worked faster, and ended up with the
current regular expression approach.

Hope that clarifies your question.

Best regards,

JS