From: "ara.t.howard" <ara.t.howard@...>
Date: 2008-07-07T13:52:39+09:00
Subject: Re: implementing a simple and efficient index system


On Jul 6, 2008, at 12:22 PM, Janus Bor wrote:

> Hello everyone,
>
> I'm pretty new to Ruby and programming in general. Here's my problem:
>
> I'm writing a program that will automatically download protein  
> sequences
> from a server and write them into the corresponding file. Every single
> sequence has a unique id and I have to eliminate duplicates.  
> However, as
> the number of sequences might exceed 50 000, I can't simply save all
> sequences in a hash (with their id as key) and then write them to hd
> after downloading has finished. So my idea is to write every  
> sequence to
> the corresponding file immediately, but first I have to check if it  
> has
> been processed already.
>
> I could save all processed id's in an array and then check if the  
> array
> includes my current id:
>
> sequences = []
> some kind of loop magic
> if sequences.include?(id)
>  process file
>  sequences << id
> end
> end
>
> But I suspect that sequences.include?(id) would iterate over the whole
> array until it finds a match. As this array might have up 50 000
> positions and I will have to do this check for every sequence, this
> would probably be very inefficient.
>
> I could also save all processed id's as keys of a hash, however I  
> don't
> have any use for a value:
>
> sequences = {}
> some kind of loop magic
> if sequences[id]
>  process file
>  sequences[id] = true
> end
> end
>
> Would this method be more efficient? Is there a more elegant way?  
> Also,
> can Ruby handle arrays/hashes of this size?
>
> Thanks in advance!
> -- 
> Posted via http://www.ruby-forum.com/.
>


the simplest and most robust method is probably going to be to use  
sqlite to store the id of each sequence.  this will help you in the  
case if a program crash and as you develop.  for example:


cfp:~ > ruby a.rb


cfp:~ > sqlite3 .proteins.db 'select * from proteins'
42|ABC123


cfp:~ > ruby a.rb
a.rb:27:in `[]=': 42 (IndexError)
         from /opt/local/lib/ruby/gems/1.8/gems/amalgalite-0.2.1/lib/ 
amalgalite/database.rb:477:in `transaction'
         from a.rb:24:in `[]='
         from a.rb:6


cfp:~ > sqlite3 .proteins.db 'select * from proteins'
42|ABC123





cfp:~ > cat a.rb

db = ProteinDatabase.new

id, sequence = 42, 'ABC123'

db[id] = sequence



BEGIN {

   require 'rubygems'
   require 'amalgalite'

   class ProteinDatabase
     SCHEMA = <<-SQL
       create table proteins(
         id integer primary key,
         sequence blob
       );
     SQL

     def []= id, sequence
       @db.transaction {
         query = 'select id from proteins where id=$id'
         rows = @db.execute(query, '$id' => id)
         raise IndexError, id.to_s if rows and rows[0] and rows[0][0]
         blob = blob_for( sequence )
         insert = 'insert into proteins values ($id, $sequence)'
         @db.execute(insert, '$id' => id, '$sequence' => blob)
       }
     end

   private
     def initialize path = default_path
       @path = path
       setup!
     end

     def setup!
       @db = Amalgalite::Database.new @path
       unless @db.schema.tables['proteins']
         @db.execute SCHEMA
         @db = Amalgalite::Database.new @path
       end
       @sequence_column =  
@db.schema.tables['proteins'].columns['sequence']
     end

     def blob_for string
       Amalgalite::Blob.new(
         :string => string,
         :column => @sequence_column
       )
     end

     def default_path
       File.join( home, '.proteins.db' )
     end

     def home
       home =
         catch :home do
           ["HOME", "USERPROFILE"].each do |key|
             throw(:home, ENV[key]) if ENV[key]
           end

           if ENV["HOMEDRIVE"] and ENV["HOMEPATH"]
             throw(:home, "#{ ENV['HOMEDRIVE'] }:#{ ENV['HOMEPATH'] }")
           end

           File.expand_path("~") rescue(File::ALT_SEPARATOR ? "C:/" :  
"/")
         end

       File.expand_path home
     end
   end

}



a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being  
better. simply reflect on that.
h.h. the 14th dalai lama