From: Robert Klemme <shortcutter@...>
Date: 2008-07-27T19:24:46+09:00
Subject: Re: Faster Marshaling?

On 27.07.2008 07:44, David Masover wrote:
> On Sunday 27 July 2008 00:33:56 Greg Willits wrote:
> 
>> We have talked about skipping Marshaling and going straight to standard 
>> text files on disk and then using read commands that point to a specific 
>> file line.
> 
> If the files aren't changing, you probably want to seek to a specific byte 
> offset in the file, rather than a line -- the latter requires you to read 
> through the entire file up to that line.

Array#pack and String#unpack come to mind.  But IMHO this is still 
inferior to using a relational database because in the end it comes down 
to reimplementing the same mechanisms that are present there already.

> For what it's worth, ActiveRecord does work on SQLite. So does Sequel, and I 
> bet DataMapper does, too.

But keep in mind that AR and the like introduce some overhead of 
themselves.  It might be faster to just use plain SQL to get at the data.

But given the problem description I would definitively go for a 
relational or other database system.  There is no point in inventing the 
wheel (aka fast indexing of large data volumes on disk) yourself.  You 
might even check RAA for an implementation of B-trees.

Kind regards

	robert