From: Bil Kleb Date: 2007-05-04T00:35:04+09:00 Subject: Re: Beyond YAML? (scaling) Brian Candler wrote: > > Use a SQL database? I always suspect that I should be doing that more often, but as my experience with databases is rather limited and infrequent, I always shy away from those as James already knows. Regardless, I should probably overcome my aggressive incompetence one day! > It all depends what sort of processing you're doing. If you're adding to a > dataset (rather than starting with an entirely fresh data set each time), > having a database makes sense. In this point, I'm generating an entirely fresh data set each time, but I can foresee a point where that will change to an incremental model... > Put it another way, does your processing really require you to read the > entire collection of objects into RAM before you can perform any processing? Yes, AFAIK, but I suppose there are algorithms that could compute statistical correlations incrementally. > You could consider using something like Madeleine: > http://madeleine.rubyforge.org/ > This snapshots your object tree to disk (using Marshal by default I think, > but can also use YAML). You can then make incremental changes and > occasionally rewrite the snapshot. Probably not a good fit as I won't change existing data, only add new... Thanks, -- Bil Kleb http://fun3d.larc.nasa.gov