From: Jamey Cribbs <cribbsj@...>
Date: 2006-05-17T22:48:53+09:00
Subject: Re: ruby sdbm library reliable?

Francis Cianfrocca wrote:
> I've been interested for a long time in doing a fast system for 
> datasets up
> to a few million rows with journalled updates. (I wrote the journal-arena
> already, I use it for fast access to log entries.) If you like, I'd be 
> happy
> to help out also.
Sure thing!

I've been doing some thinking about Logan's idea of offering a choice of 
multiple backends.  That sounds intriguing.  I could see that idea, 
coupled with something like qdbm, to provide different options for the user.

The default backend could remain the plain-text files that KirbyBase 
currently uses.  But, if the user has qdbm (as an example) installed, 
KirbyBase could optionally use that as the backend resulting in a 
noticeable (I would hope) performance increase.

By the way, does anyone have experience using qdbm?  It looks like the 
library is well maintained and there is plenty of documentation.  I'm 
curious if it is reliable and does it seem to scale.

As far as the indexing goes, I've recently become infatuated with 
Skiplists.  In some informal testing, my very crude skiplist 
implementation seemed to be a nice balance between a hash and an array 
for general lookups.  I've also thought about trying out a B-Tree 
implementation for the indexing, with the added possibility of 
storing/accessing the indexes from disk as opposed to loading them 
completely into memory before use.  I think Logan mentioned that he had 
some thoughts in that direction.

I would definitely be interested in seeing your work with journaling.  I 
was thinking of possibly using journaling in conjunction with having the 
entire table in memory.  Updates and deletes would be made directly to 
the table in memory as well as written to a journal file, so that, when 
it came time to write the table back out to disk, I would only have to 
update the table's disk file with the contents of the journal file as 
opposed to having to write out the memory-based table.  Like I said, 
this is just brainstorming.  It may not even make sense to try to keep 
the entire table in memory.

Jamey

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient(s), you are hereby notified that any dissemination, unauthorized review, use, disclosure or distribution of this email and any materials contained in any attachments is prohibited. If you receive this message in error, or are not the intended recipient(s), please immediately notify the sender by email and destroy all copies of the original message, including attachments.