From: Bob Hutchison <hutch@...>
Date: 2006-08-12T20:48:01+09:00
Subject: Re: Looking for a Fast Persistent Store


On Aug 11, 2006, at 8:12 PM, Francis Cianfrocca wrote:

> On 8/11/06, Bob Hutchison <hutch@recursive.ca> wrote:
>>
>>  External fragmentation can be
>> a problem.
>
>
> Clearly a directory hierarchy is the only way to make this work,  
> and Kirk's
> idea of partitioning the space by segmenting the hash seems as  
> reasonable as
> any. So far I'm very surprised to find that performance may be  
> reasonable.

So was I but it actually is quite good (I'll publish some  
unscientific benchmarks in the next couple of days).

>
> But what do you mean by "external fragmentation"? If you're  
> thinking of NFS,
> I'd rather cut myself than attempt something like this on NFS, even  
> though
> Kirk says it's supported.

Sorry, jargon from 3 decades ago. External fragmentation, in this  
case is the disk space wasted due to small files being put into a  
larger segment (e.g. 1k file in a 4k disk segment is a 3k loss of  
space (external fragmentation)). Things like Perst, qdbm, Purple,  
SQLite-as-persistent-store will avoid almost all external  
fragmentation, at the cost of internal fragmentation. In the case of  
the file system, there isn't any space wasted *within* the 1k file,  
but the persistent stores all have this problem (and you'll see them  
talking about free space management, best-fit, first-fit, garbage  
collection, and so on).

Cheers,
Bob

----
Bob Hutchison                  -- blogs at <http://www.recursive.ca/ 
hutch/>
Recursive Design Inc.          -- <http://www.recursive.ca/>
Raconteur                      -- <http://www.raconteur.info/>
xampl for Ruby                 -- <http://rubyforge.org/projects/xampl/>