From: Francis Cianfrocca <garbagecat10@...>
Date: 2006-05-17T21:33:52+09:00
Subject: Re: ruby sdbm library reliable?

------=_Part_114923_30583508.1147869228380
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

I've been interested for a long time in doing a fast system for datasets up
to a few million rows with journalled updates. (I wrote the journal-arena
already, I use it for fast access to log entries.) If you like, I'd be happ=
y
to help out also.

On 5/16/06, Jamey Cribbs <cribbsj@oakwood.org> wrote:
>
> Berger, Daniel wrote:
> > The dbm/gdbm/sdbm libraries are toys, imho.  I wouldn't trust them to
> > store more than a few thousand records, let alone a million.  I've seen
> > them fall down horribly on Windows.  Just ask Hal Fulton.  As far as I
> > can tell, the dbm/gdbm/sdbm wrappers were included because thats what
> > Perl does.
> >
> >
> Thanks, Dan.  That's the kind of info I was looking for before I spent a
> lot of time testing.
> > I'm very curious Jamey - why aren't you using KirbyBase?
> >
>
> Well, here's the deal.  :-)
>
> I'm looking at trying to improve KirbyBase's performance, to make it
> scale better.  One way I am doing this is I have started playing around
> with using Skiplists for KirbyBase indexes, rather than arrays.  For a
> lot of queries, this has greatly speeded up the part of the query that
> grabs all matching keys.
>
> But, I'm still running into the bottleneck of taking those keys from the
> query and using them to pull the actual records in from the flat text
> file where KirbyBase stores the actual data.
>
> So, I've been looking at ways to get around this.  I would basically
> look at replacing the back-end flat file format of KirbyBase with
> something faster.  The user wouldn't see any difference in how they
> interacted with KirbyBase.  One of the possibilities was to store the
> data records in some kind of dbm-like file (i.e. gdbm, qdbm, etc.) and
> sdbm looked attractive because it is included with Ruby.  The downside
> to this is that the data is no longer kept in easily-editable plain-text
> files.
>
> Another option would be to keep the entire table in memory as a hash and
> use some kind of journal file to handle updates/deletes.  This might be
> an issue on memory-constrained systems, but I'm thinking this is less
> and less of an issue on modern pcs.
>
> I'm just kind of brainstorming right now.  KirbyBase performs pretty
> well, but I would like to see it scale better.
>
> Jamey
>
> Confidentiality Notice: This email message, including any attachments, is
> for the sole use of the intended recipient(s) and may contain confidentia=
l
> and/or privileged information. If you are not the intended recipient(s), =
you
> are hereby notified that any dissemination, unauthorized review, use,
> disclosure or distribution of this email and any materials contained in a=
ny
> attachments is prohibited. If you receive this message in error, or are n=
ot
> the intended recipient(s), please immediately notify the sender by email =
and
> destroy all copies of the original message, including attachments.
>
>

------=_Part_114923_30583508.1147869228380--