From: Bob Hutchison Date: 2006-08-13T22:15:45+09:00 Subject: Re: Looking for a Fast Persistent Store Hi, On Aug 12, 2006, at 8:27 AM, Francis Cianfrocca wrote: > > That's a pretty constant tradeoff, speed for space, and it shows up > in many > different ways. I'm convinced that's why there's no single answer > to this > problem- every application will need different optimizations. > However, this > is somewhat less true than it once was, for two reasons: journaling > filesystems, and the fact that disk storage is still getting > cheaper every > year at a rapid rate- it's the only element in the computing chain > that > still is. So my inclination (subject to change depending on the > particular > case) is to waste space if it will save time (development time and/or > runtime). I absolutely agree which is why I've implemented the filesystem route first. There is *no* performance problem with it. The problem is that I've got to write a bunch of files out and if for some reason even one file failed to write I'd have a consistency problem. There are easy ways to mitigate these risks but they can take an excruciatingly long time to run. Moving to a transactional store (like Perst in Java, or SQLite in ruby) gives me the guarantees of consistency that I'm looking for, but SQLite in ruby is 3-5 times slower than direct to the file system on OS/X with journaling on (unless there's something wrong in my testing... silly benchmarks forthcoming). In Java, Perst is *much* faster than the filesystem. Now, I've thought of implementing the same kind of techniques that some persistent stores use to assure consistency but directly in the filesystem. The only quick-enough techniques that I can think of would require a recovery after failure, but that shouldn't be too big a problem (especially the frequency of failure will be low judging from my current experience where I've not had any failures yet). ---- Bob Hutchison -- blogs at Recursive Design Inc. -- Raconteur -- xampl for Ruby --