From: Robert Feldt <feldt@...>
Date: 2004-10-06T07:43:57+09:00
Subject: Re: Large disc-based "hash"?

Ara.T.Howard@noaa.gov wrote:

> On Wed, 6 Oct 2004, Robert Feldt wrote:
>
>> Hi,
>>
>> I need a simple way to have a potentially very large hash mapping
>> Strings to Ruby objects. It needs to scale to on the order of 1 million
>> entries with up to 1kb of data per entry (even though it might end up
>> with maybe 2-300 bytes per entry for starters). Access time is not
>> critical but low-to-medium mem usage is. Deletion of (and insert of new)
>> keys/values will be fairly frequent so should be supported.
>
>
> if 'fairly frequent' is once per week look at CDB too - it's 
> blindingly fast
> and updates (atomic replacements actually) are suprisingly fast.  
> reads are
> absolutely insanely fast.
>
fairly frequent is more like "1-5% of keys per hour" but thanks for the 
pointer, very interesting. Seems it is unix only but I need windows also 
so maybe not.

> bdb is also great.
>
Yes, seems it can handle this load. I tried with a BTree and the 
db-file-size-to-added-key+value-bytes ratio seems to stay at 2.5 at 
least up to 1e5 entries. Running a test with 1e6 added entries now; 
insertion time does not seem to scale linearly but we'll see about file 
size.

Thanks to both Rich and you for the pointers.

Downside of going with something like bdb is ease of install/support on 
multiple platforms (ie windows since everything works on linux... :)) 
but I guess bdb should be well supported also on Windows so maybe that's 
what it'll be.

> sqlite with adequate indexing may also work - but probably not.
>
Ok, thanks for that info.

/R