From: Igal Koshevoy <igal@...>
Date: 2008-07-07T07:32:24+09:00
Subject: Re: implementing a simple and efficient index system

Joel VanderWerf wrote:
> Igal Koshevoy wrote:
>> 1. PStore standard library: Put your objects into a magical hash, 
>> that's automatically persisted to a file. Probably the quickest and 
>> easiest solution. See 
>> http://www.ruby-doc.org/stdlib/libdoc/pstore/rdoc/classes/PStore.html
>
> PStore writes the whole file at once, not incrementally. Not really 
> what OP is looking for, IMO.
It takes ~2s for my machine to read or write the 50MB PStore file. This 
isn't a big deal if the original poster (OP) doesn't mind keeping the 
program running to process multiple sequences at once.

>> 2. Lightweight SQL database: Maybe store sequences in SQLite as 
>> BLOBs. Probably the best long-term solution, but will require you to 
>> work harder to transform data to and from storage. See 
>> http://sqlite-ruby.rubyforge.org/
>
> Not clear that would be better than files. Maybe so, if the individual 
> strings are short. Would be interesting to get some benchmarks on this 
> question.
Files would probably be faster, but with such a small dataset, we're 
probably talking about less than a second of difference for processing 
the full dataset. I like using SQLite for stuff like this because it 
provides a standard, out-of-the-box solution for working with 
persistence, incremental processing, structured data, queries, and the 
ability to easily add more fields to a record.

>> 3. Marshal core class: Dump objects to and from strings, and then 
>> files. Useful if you need something more than PStore, but still want 
>> to persist objects directly. See 
>> http://ruby-doc.org/core/classes/Marshal.html
>
> PStore uses Marshal, so it's odd to say that Marshal is more than PStore.
Working directly with Marshall allows greater flexiblity than using the 
PStore wrapper, for example, if they decided to write a filesystem 
database class. :)

> If you're looking for a way to manage marshalled (or string or 
> yaml...) data in multiple files, using file paths as db keys, look no 
> further than: http://raa.ruby-lang.org/project/fsdb/
Cool project, thanks for writing it. Sounds useful.

-igal