From: Jamey Cribbs <jcribbs@...>
Date: 2007-05-03T23:04:37+09:00
Subject: Re: Beyond YAML? (scaling)

Bil Kleb wrote:
> Hi,
>
> I've been using YAML files to store hashes of numbers, e.g.,
>
>  { ["O_Kc_01"] => [ 0.01232, 0.01212, 0.03222, ... ], ... }
>
> This has worked wonderfully for portability and visibility
> into the system as I've been creating it.
>
> Recently, however, I've increased my problem size by orders
> of magnitude in both the number of variables and the number
> of associated values. The resulting YAML files are prohibitive:
> 10s of MBs big and requiring 10s of minutes to dump/load.
>
> Where should I go from here?

Hey, Bil.  If you don't mind a couple of shameless plugs, you might want 
to try KirbyBase or Mongoose.

KirbyBase should be faster than YAML and it still stores the data in 
plain text files, if that is important to you.

Mongoose is faster than KirbyBase, at the expense of the data not being 
stored as plain text.

I don't know if either will be fast enough for you.

HTH,

Jamey Cribbs