From: James Edward Gray II <james@...>
Date: 2007-05-03T23:04:06+09:00
Subject: Re: Beyond YAML? (scaling)

On May 3, 2007, at 8:50 AM, Bil Kleb wrote:

> Hi,
>
> I've been using YAML files to store hashes of numbers, e.g.,
>
>  { ["O_Kc_01"] => [ 0.01232, 0.01212, 0.03222, ... ], ... }
>
> This has worked wonderfully for portability and visibility
> into the system as I've been creating it.
>
> Recently, however, I've increased my problem size by orders
> of magnitude in both the number of variables and the number
> of associated values. The resulting YAML files are prohibitive:
> 10s of MBs big and requiring 10s of minutes to dump/load.
>
> Where should I go from here?

Some random thoughts:

* If they are just super straightforward lists of numbers like this a  
trivial flat file scheme, say with one number per line, might get the  
job done.
* XML can be pretty darn easy to output manually and if you use  
REXML's stream parser (not slurping everything into a DOM) you should  
be able to read it reasonably quick.
* If you are willing to sacrifice a little visibility, you can always  
take the step up to a real database, even if it's just sqlite.  These  
have varying degrees of portability as well.
* You might want to look at KirbyBase.  (It has a younger brother  
Mongoose, but that uses binary output.)

Hope something in there helps.

James Edward Gray II