From: Bill Kelly <billk@...>
Date: 2006-08-13T11:24:34+09:00
Subject: Re: Looking for a Fast Persistent Store

From: "Francis Cianfrocca" <garbagecat10@gmail.com>
>
> I was so interested in this idea that I cobbled up a quick test on a
> workstation machine with a medium-bandwidth IO channel. 100,000 1K files
> brought the shared-memory filesystem to its knees almost immediately. On an
> oxide-based  journalling filesystem, it seemed to do 10 to 20 thousand files
> a second pretty consistently. That's a decent but not huge fraction of the
> available channel bandwidth. Haven't tried a million files. I'm still
> interested though.

Hi,

What is being measured?  Access time for files that already exist?
Creation of new files?  Scanning the directory structure for a list
of existing files?

At a prior gig, we used to split a couple hundred thousand 
encyclopedia articles up as 12/34/56.xxx sort of format.  It worked
adequately for our needs--our batch-oriented processing was expected
to run overnight anyway--but my impression was that as long as the
filename was known, accessing file 12/34/56.xxx seemed quick,
whereas directory scans to enumerate the existing filenames were
pretty slow.

(This was on sunOs/solaris systems circa 2000... i don't know if
the filesystem was journalled or not.)


Regards,

Bill