From: Joseph McDonald <joe@...>
Date: 2001-09-07T11:12:34+09:00
Subject: [ruby-talk:20969] Re: creating lots of files


BFF> Here's a simple algorithm for you, if you'd like to use real filenames
BFF> instead of just file numbers:

BFF> def make_filename(source)
BFF>         [source.hash].pack('I').unpack('C*').collect {|c|
BFF>             format("%.02x", c)
BFF>         }.push(source).join('/')
BFF> end

BFF> If you're going to use 32-bit numbers as filenames, you can get away with
BFF> simply:

BFF> def make_filename(source)
BFF>         [source].pack('I').unpack('C*').collect {|c| "%.02x" % c}.join('/')
BFF> end

Interesting ideas.  I would of course change join('/') to
join(File::SEPARATOR) now :-)


BFF> However, that certainly won't give you even distribution if you use 
BFF> sequential numbering, instead of a pseudo-random sequence.

I'm not necessarily looking for even distribution, in fact I want
files to be grouped close to their creation time -- they will be used
together most of the time and hopefully the OS/disk controller caching
would help if I grouped them in the same directory.

BFF> Personally, I'd just let the OS maintain the hash tables itself for large 
BFF> directories and only concentrate on making sure to divide the names 
BFF> generated so that they don't get too excessively large.  Even nicer would be

Are you saying just have 1 directory with millions of files?  I have not
tried that recently, but I know that years ago when running nntp
servers, that FreeBSD/Linux and SunOS didn't like it *at all*.  The
best solution back then was to buy a NetApp which touted its ability
to deal with many files in a directory.

BFF> just to use a filestore that simply uses sequential numbered files and has 
BFF> no concept of directories in the first place.

Just 1 directory?

thanks,
-joe