From: Igal Koshevoy <igal@...>
Date: 2008-07-07T05:39:27+09:00
Subject: Re: implementing a simple and efficient index system

Janus Bor wrote:
> No, I didn't try it and it might actually work: Every sequence has a 
> size of ~1kb, so 50 000 sequences would probably be around 50mb. But 
> getting all this data will take hours, so I need to implement a system 
> that will not lose all data if the program is terminated abnormally.
>   
Here are some simple alternatives for persisting and retrieving your 
data in the order I'd recommend them based on what you've described so far:

1. PStore standard library: Put your objects into a magical hash, that's 
automatically persisted to a file. Probably the quickest and easiest 
solution. See 
http://www.ruby-doc.org/stdlib/libdoc/pstore/rdoc/classes/PStore.html

2. Lightweight SQL database: Maybe store sequences in SQLite as BLOBs. 
Probably the best long-term solution, but will require you to work 
harder to transform data to and from storage. See 
http://sqlite-ruby.rubyforge.org/

3. Marshall core class: Dump objects to and from strings, and then 
files. Useful if you need something more than PStore, but still want to 
persist objects directly. See http://ruby-doc.org/core/classes/Marshal.html

Best of luck.

-igal