From: Joel VanderWerf <vjoel@...>
Date: 2008-07-07T03:38:15+09:00
Subject: Re: implementing a simple and efficient index system

Janus Bor wrote:
> Hello everyone,
> 
> I'm pretty new to Ruby and programming in general. Here's my problem:
> 
> I'm writing a program that will automatically download protein sequences
> from a server and write them into the corresponding file. Every single
> sequence has a unique id and I have to eliminate duplicates. However, as
> the number of sequences might exceed 50 000, I can't simply save all
> sequences in a hash (with their id as key) and then write them to hd
> after downloading has finished. So my idea is to write every sequence to
> the corresponding file immediately, but first I have to check if it has
> been processed already.

Can you simply use the id as a filename, and check for file existence 
before writing? If you file system doesn't handle huge dirs well, then 
split the id into several terms. But I'd try the hash or set approach 
first, to avoid all the system calls.

-- 
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407