From: Francis Cianfrocca Date: 2006-08-12T08:00:12+09:00 Subject: Re: Looking for a Fast Persistent Store ------=_Part_138072_33448763.1155337209503 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline I was so interested in this idea that I cobbled up a quick test on a workstation machine with a medium-bandwidth IO channel. 100,000 1K files brought the shared-memory filesystem to its knees almost immediately. On an oxide-based journalling filesystem, it seemed to do 10 to 20 thousand files a second pretty consistently. That's a decent but not huge fraction of the available channel bandwidth. Haven't tried a million files. I'm still interested though. On 8/11/06, khaines@enigo.com wrote: > > On Sat, 12 Aug 2006, Francis Cianfrocca wrote: > > > With all the times I've reinvented this wheel, I've never tried > > storing millions of data elements each in its own file. Do you have > > performance metrics you can share? Did you tune it for a particular > > filesystem? > > Not really. I have not stored the results of any tests to date. The > largest test, however, created a million item cache, and I had no > problems. > > The main consideration if one were planning on storing large numbers of > elements is making sure that the filesystem is built with large numbers of > small files taken into consideration. > > However, a lot of files can fit into a surprisingly non-messy directory > structure. > > Consider an SHA512 hash for a key that has the first few characters of: > > dd232b224979c0e > > If I am running a cache with a bucket depth of 2, and a bucket width of 2, > the file for that cache is going to go into the directory > > dd/23/ > > Given an even distribution of hashes, with a million records, one should > have 256 directories at the top level, 256 at the second level, and 15 or > 16 files under each of those. > > That's pretty easy to handle efficiently. > > > Kirk Haines > > ------=_Part_138072_33448763.1155337209503--