From: Joel VanderWerf <vjoel@...>
Date: 2005-03-03T06:46:36+09:00
Subject: Re: A wish: Simple database

Hal Fulton wrote:
> Michael Neumann wrote:
> 
>>
>> Something like FSDB?
>>
>> http://raa.ruby-lang.org/list.rhtml?name=fsdb
>>
>> Theoretically, it could be imported into Ruby if the author and matz 
>> would agree (and there's need for it in the base distribution).
> 
> 
> I've used FSDB and I like it.

Well, thank you, Hal. And thanks to Michael, too.

> But I was always a little nervous about its disk usage. Is it
> excessive?

Have you seen any problems? The disk usage should be equal to your data
(stored using your choices of serialization methods), plus the extra
dirs to provide the hierarchical db structure, plus some small 
(typically 4 bytes each for versioning data) files with names like 
"..fsdb.meta.yourfilename".

So the overhead (due to the file system's block size, for instance) will 
be bad if you have lots of little files. (Maybe Reiser FS can help with 
that.)

> I know I said I wasn't terribly worried about performance, but
> it does need to be *reasonable* in terms of speed and space.

The benchmarks report 900+ transactions per sec with 1 process, 1 thread 
on 850MHz Pentium. It degrades a little when you add processes, and it 
degrades a bit more as the thread count increases.

That brings up another advantage of fsdb over PStore: fsdb is both 
process and thread safe. That's assuming the OS has a reasonable file 
lock (FSDB is not process-safe on WinME). Process safety means that FSDB 
can be used for _persistent_ asynchronous IPC in situations where some 
delay is acceptable. Also note that FSDB uses its own mutex classes that 
don't break when you fork from a multithreaded app, so you can easily 
use FSDB for communicating among parent and child processes. If you fork 
with a db object in scope, you can just keep talking to it.

My only objection to putting fsdb in the standard library is that I've
only tested it on Windows (noting the problem on ME/98/95), Solaris, and
Linux. Someone was trying to get it to work on OSX, but we couldn't get
file locking to work (I didn't have access to a mac, and I couldn't
really debug remotely).

It's already Ruby-licensed.

Regarding Hal's feature list:

> 1. Distributed as part of Ruby

No. BTW, FSDB is in RPA.

> 2. Need not store entire db in memory

Yes, need not. FSDB uses a cache that can be cleared and only loads 
requested objects into the cache. If you want, you can subclass the 
CacheEntry class to use weak references, so that ruby GC can clear the 
cache as needed. (I may make this standard if it is not too much of a 
speed hit--it's on my todo list to investigate.)

> 3. No SQL requirement

None. Unfortunately, there is no SQL interface, either.

> 4. No special efficiency requirement

None. It's pretty a basic ruby wrapper on top of file system calls.

> 5. Available cross-platform

Sorta. See above. The difficulty is that each platform generates file 
exceptions differently (and it may even differ from file system to file 
system AFAIK). For instance, WinME raises Errno::ENOENT if you try to 
open a dir. FSDB maps this to reasonably platform-independent behavior.

> 6. Database files are readable cross-platform

Yes, and you have your choice (per file, using regex matching on the 
db-relative file path) of YAML, Marshal, ASCII, tabular data, CSV, etc.

Of course, if you want the whole database in one file, this file tree 
stuff isn't gonna make you happy.

But that's another nice thing about FSDB: if you have an existing file 
tree, even with a mix of file formats, you can treat it as a database. 
It's a nice way to have ruby interact somewhat safely with external 
programs that have their own ideas about where data files go and what 
format they are in.

Conversely, you can treat a FSDB database as a file tree: you can grep 
it, rsync it, etc.

More details on:

http://redshift.sourceforge.net/fsdb/doc/api/index.html