From: Bob Hutchison Date: 2006-08-10T00:07:42+09:00 Subject: Re: Looking for a Fast Persistent Store Hi Ara, On Aug 9, 2006, at 10:08 AM, ara.t.howard@noaa.gov wrote: > On Wed, 9 Aug 2006, Bob Hutchison wrote: > >> Hi, >> >> I'm looking for a persistent store, where: >> * I can use it from Ruby >> * it is fast >> * it is transactional >> * it can update more than one store in a single transaction >> * simple String -> String mappings in the store are sufficient >> * ideally uses files in the filesystem >> >> These kinds of stores are normally implemented as persistent >> hashes or >> BTrees (or both). I know that Sleepycat's Berkeley DB >> does this, and I've used Java based >> systems that >> do this. I also know of some C based things but they don't have Ruby >> wrappers. I can't find anything in the Ruby world, and I don't >> care too much >> if it isn't pure Ruby. >> >> I know about Purple and the QDMB >> Ruby wrapper . Neither do the >> multiple hash/BTree in a transaction thing. >> >> The trick appears to be with transactions. > > check out joel's fsdb - it's very nice if you want to go pure > ruby. i've used > it on several projects. I'm already using that (version 0.5 -- I can't get to RAA right now for some reason and there are no files on Rubyforge for fsdb so I don't know if there is a more recent version). In version 0.5 the transactions were not sufficient I think (it would be nice if I was wrong). > >> I know this can be done easily enough in a Relational DB, but I >> know that, >> for example, JDBC/MySQL combination is significantly slower at >> what I need >> to do than Perst (a pure Java thing that's startlingly fast). I've >> taken a >> look at sqlite, which satisfies all requirements but I can't shake >> this >> feeling that things can be better. >> >> Does anyone have any ideas? > > sqlite is hard to beat - i use it in at least 20 production systems > and have > never had a single error. ruby queue uses it under the hood for > the cluster > job store and this is used over nfs - we've been through disk > failures and > power failures and come through unscathed. That's good to hear. Don't misunderstand, I *like* SQLite but I don't know that it is suitable. If I can't find anything else I'll use it. The thing that makes me nervous is that I can do what I need with just two tables. One that has (key, value-kind, value), and another that has (index-name, index-value, key, value-kind). Each column is a String. The vast majority of queries would be based on key and value- kind on the first table. The remaining queries would be a select/join kind of thing. And I can very easily jam the key and value-kind into the same field (but that would be an optimisation that may not be necessary). The trouble is that the first table might get to be very very large. I don't know how SQLite behaves with a single huge table. I suppose I'm going to have to find out. > > it's also very fast, esp if you use in memory tables, but on linux > it's just > as easy to copy the db to /dev/shm and then go... That's interesting. > > if you end up using it try with my arrayfields package - james' > sqlite binding > detects it automatically and the tuples with come back as arrays > with named > field access - it's faster and requires less memory than a hash, > and is also > really convenient to code with. > I noticed that the other day. Nice. Cheers, Bob > cheers. > > -a > -- > to foster inner awareness, introspection, and reasoning is more > efficient than > meditation and prayer. > - h.h. the 14th dali lama > ---- Bob Hutchison -- blogs at Recursive Design Inc. -- Raconteur -- xampl for Ruby --