From: Greg Willits <lists@...>
Date: 2009-07-07T04:47:48+09:00
Subject: Re: file.seek and unused bytes

> reading this I wonder why you don't dump the records into a database
> at this point. For example SQLite3 would be lightweight, fast and
> flexible. You could probably save a lot of effort not having to build
> index structures manually and you could query your dataset using SQL
> or the ORM of your choice afterwards...

That would actually require a much larger rewrite than what I've done. 
It wasn't too big a leap to change the APIs from "here's all the 
records" to "here's some of the records." A few order of operations had 
to be juggled, but overall it's not a major logical leap, nor a major 
implementation leap. That probably goes for all the other DB options as 
well.

At this point, the change from all-in-RAM to chunks-in-RAM has been a 
relatively straight forward retrofit, and based on previsou versions 
which did use a DB, I think this is still faster or as fast. Another 
option we will explore is 64-bit ruby (which I started another thread 
for a few days ago).

>What about storing all file offsets
>in an Array and write it to a file

That's a possibility that would be an easy retrofit and probably w/o 
breaking the current interfaces. It would use a little more RAM, but 
would eliminate calculating that offset each time during reads. That's 
not a real pinch point, but worth an experiment just because it's a 
simpler concept to recognize.

> Did you consider to make your dsl generate the code 
> or is that what you are doing?

If I interpret the question correctly, that's what we're doing. I may be 
using "DSL" a little loosely here though. It's more like an "active 
config" language. I write setup files that are a mixture of k-v pairs 
for certain things, and expressions for other things. All of that gets 
processed at the start of every aggregation run which results in a 
number of Ruby Module files being created which get included 
dynamically.

Many thanks for all the thoughts & feedback.

-- gw


-- 
Posted via http://www.ruby-forum.com/.