From: "Brian F. Feldman" <green@...>
Date: 2001-09-07T10:37:31+09:00
Subject: [ruby-talk:20967] Re: creating lots of files

Joseph McDonald <joe@vpop.net> wrote:
> 
> I'm going to be dealing with a lot of files, some operating systems
> don't like to have too many files in a single directory.  So I thought
> that if I created a directory with 256 entries, each of which was
> another directory with 256 directories each and each of those held 256
> files, I could handle 256^3 (16,777,216) files without the OS choking.
> 
> Does this look like a good way to do it (assuming that each file has
> a number as a name starting at 1 and incrementing for each new file):
> 
> def make_filename(file_num)
>     first_level = file_num%256
>     second_level = (file_num%65536)/256
>     filename = [first_level, second_level, file_num].join("/")
> end

Here's a simple algorithm for you, if you'd like to use real filenames 
instead of just file numbers:

def make_filename(source)
	[source.hash].pack('I').unpack('C*').collect {|c|
	    format("%.02x", c)
	}.push(source).join('/')
end

If you're going to use 32-bit numbers as filenames, you can get away with 
simply:

def make_filename(source)
	[source].pack('I').unpack('C*').collect {|c| "%.02x" % c}.join('/')
end

However, that certainly won't give you even distribution if you use 
sequential numbering, instead of a pseudo-random sequence.

Personally, I'd just let the OS maintain the hash tables itself for large 
directories and only concentrate on making sure to divide the names 
generated so that they don't get too excessively large.  Even nicer would be 
just to use a filestore that simply uses sequential numbered files and has 
no concept of directories in the first place.

-- 
 Brian Fundakowski Feldman           \  FreeBSD: The Power to Serve!  /
 green@FreeBSD.org                    `------------------------------'