From: Bill Kelly <billk@...>
Date: 2001-09-08T01:58:23+09:00
Subject: [ruby-talk:20985] Re: creating lots of files


From: "David Tillman" <dtillman@ozarkaircraftsystems.com>
> 
> >>>>> Joseph McDonald <joe@vpop.net> writes:
> 
> > I'm going to be dealing with a lot of files, some operating
> > systems don't like to have too many files in a single directory.
> > So I thought that if I created a directory with 256 entries,
> > each of which was another directory with 256 directories each
> > and each of those held 256 files, I could handle 256^3
> > (16,777,216) files without the OS choking.
> <snip>
> 
> 
>   This isn't an answer to your question - How large are the files?
> 
>   Have you considered using postgresql or dbm?
> 
>   I had a similar situation under Solaris, dealing with an application
>   that generated 25,000 files a day and we needed to keep three days worth.
> 
>   Once we hit 30,000 things got seriously boggy.

FWIW we have had little problem with 1,000,000's of files in
Solaris with a directory structure similar to that described
by Joseph.  Access (esp. creation and deletion) isn't as speedy
as what I imagine gdbm should be, but it definitely perked up
a lot when we found the fastfs command to turn on async metadata
writes:

ftp://www.wins.uva.nl/pub/solaris/fastfs.c.gz

It's been great for us - might want to peruse Google groups for
some discussion about increased chance of filesystem corruption
in the event of a system crash or power failure though . . .

> 
>   The simple solution: I added a gdbm interface and stored the files with
>   the filename as the key.
> 
>   With 75,000 files the gdbm db was 200Mb and *fast*.
> 
>   Not really Ruby related...
> 
> 
>   -David Tillman

Bill