From: Greg Willits <lists@...>
Date: 2009-07-08T02:10:41+09:00
Subject: Re: file.seek and unused bytes

Robert Klemme wrote:
> 2009/7/7 Greg Willits <lists@gregwillits.ws>:
>> James Gray wrote:
>>> The more I read in this thread, the more I feel it's crying out for a
>>> database
>>
>> I think if you saw the whole thing you'd see why the db is more of a
>> hinderance than an assett. I've described a small piece of it, and not
>> the piece that has the most compelling reasons to avoid a db -- at least
>> a traditional db. Some of these other ones that have brought up may be a
>> different story.
> 
> I have to admit that I thought of a database as well.  Keywords that
> especially triggered this were: index, aggregation, huge, speed.  You
> make it really sound as if we are talking about a huge beast of which
> we are getting just a tiny glimpse. :-)

Heh. No, it's not huge, certainly not by today's standards of huge.

The first version of this did everything by database, but we realized we 
were using little of the DB's strengths, and that in-memory indexes 
would be more efficient to both create and use. Most of our indexes 
aren't normal indexing of single fields, but rather combinations of 
pieces of various fields. Those ones are used to determine matches and 
probabilities of matches between two data sets.

During aggregation, we do a lot of work with just the indexes 
themselves, something that wasn't as effective when using a DB.

When we replaced the DB with random access text files for the data 
itself, and replaced all the other tasks with our own techniques, the 
system as whole was faster, and more flexible to code new ideas for 
matching and aggregation.

Now, whether something like redis or couchdb has value added compared to 
a traditional db because they behave more like what we've done, I don't 
know. Haven't looked into that yet, but it does look interesting.

Still, what we have works quite well, is entirely in Ruby, and requires 
no other resources except to connect to the application's db for the 
final data update. So there's some advantages to that simplicity as 
well.

-- gw


-- 
Posted via http://www.ruby-forum.com/.