From: Daniel Sheppard <daniels@...>
Date: 2005-11-07T13:09:33+09:00
Subject: Re: Help requested -- regexp

> It worked for the likes of the example above, but started 
> having problems for clauses with multiple fields in each 
> 'condition', like:
> 
>      (field1 > 4) and ((field4 < 2.25) or (field3 + field8 > 8.0))

I'll go out on a limb and say (field3 + field8 > 8.0) cannot be handled
by a regex. No way. No how. You're going to have to throw some
programming at that. Regex's might be getting you part way there, but
it's not going to cut it.

I'd say to throw a crapload of memory at a box, have a process read the
file into objects and then expose those objects over drb for client
processes to query. You'd probably need to throw in some sort of
indexing and query caching, basically building a database tuned
specifically for what you want.

If that doesn't get you fast enough, or you can't pull that sort of
memory, try using something closer to the bone than ruby - either some c
extensions or (blasphemy) shift the whole thing to some other language. 

But really, this thing does scream database - there's got to be a way to
make the loading and querying of that thing faster. Try using an isam
based database (ie. mysql) for that extra speed. You don't need the full
weight of a transactional relational database there.
#####################################################################################
This email has been scanned by MailMarshal, an email content filter.
#####################################################################################