From: ara.t.howard@...
Date: 2006-08-09T23:08:09+09:00
Subject: Re: Looking for a Fast Persistent Store

On Wed, 9 Aug 2006, Bob Hutchison wrote:

> Hi,
>
> I'm looking for a persistent store, where:
> * I can use it from Ruby
> * it is fast
> * it is transactional
> * it can update more than one store in a single transaction
> * simple String -> String mappings in the store are sufficient
> * ideally uses files in the filesystem
>
> These kinds of stores are normally implemented as persistent hashes or
> BTrees (or both). I know that Sleepycat's Berkeley DB
> <http://www.sleepycat.com/> does this, and I've used Java based systems that
> do this. I also know of some C based things but they don't have Ruby
> wrappers. I can't find anything in the Ruby world, and I don't care too much
> if it isn't pure Ruby.
>
> I know about Purple <http://purple.rubyforge.org/> and the QDMB Ruby wrapper 
> <http://qdbm.sourceforge.net/>. Neither do the multiple hash/BTree in a 
> transaction thing.
>
> The trick appears to be with transactions.

check out joel's fsdb - it's very nice if you want to go pure ruby.  i've used
it on several projects.

> I know this can be done easily enough in a Relational DB, but I know that,
> for example, JDBC/MySQL combination is significantly slower at what I need
> to do than Perst (a pure Java thing that's startlingly fast). I've taken a
> look at sqlite, which satisfies all requirements but I can't shake this
> feeling that things can be better.
>
> Does anyone have any ideas?

sqlite is hard to beat - i use it in at least 20 production systems and have
never had a single error.  ruby queue uses it under the hood for the cluster
job store and this is used over nfs - we've been through disk failures and
power failures and come through unscathed.

it's also very fast, esp if you use in memory tables, but on linux it's just
as easy to copy the db to /dev/shm and then go...

if you end up using it try with my arrayfields package - james' sqlite binding
detects it automatically and the tuples with come back as arrays with named
field access - it's faster and requires less memory than a hash, and is also
really convenient to code with.

cheers.

-a
-- 
to foster inner awareness, introspection, and reasoning is more efficient than
meditation and prayer.
- h.h. the 14th dali lama