From: "ara.t.howard" <ara.t.howard@...>
Date: 2008-08-15T05:56:46+09:00
Subject: Re: Mutithreading to implement near 7000 to 10000 mssage per min


On Aug 14, 2008, at 1:10 PM, Martin DeMello wrote:

> On Thu, Aug 14, 2008 at 12:06 PM, ara.t.howard  
> <ara.t.howard@gmail.com> wrote:
>>
>> buffer them and insert them in a transaction 1000 at a time.  even  
>> with ruby
>> this should be a peice of cake.
>
> Do any of the ruby db libraries offer support for doing this  
> efficiently?
>
> martin

pretty much all of them


cfp:~/rails_root > cat a.rb
size = Integer(ARGV.shift || 10_000)

messages = Array.new(size).map{ rand.to_s }

Db = "#{ RAILS_ROOT }/db/#{ RAILS_ENV }.sqlite3"


# using sqlite directly
#
   Message.delete_all
   sql = messages.map{|message| "insert into messages(content)  
values(#{ message.inspect });"}.join("\n")

   a = b = response = nil
   IO.popen("sqlite3 #{ Db } 2>&1", "r+") do |sqlite3|
     a = Time.now.to_f
     sqlite3.puts "begin;"
     sqlite3.puts sql
     sqlite3.puts "end;"
     sqlite3.flush
     sqlite3.close_write
     response = sqlite3.read
     b = Time.now.to_f
   end

   abort response unless $?.exitstatus.zero?

   puts "using sqlite3"
   puts "elapsed: #{ b - a }"
   puts "count: #{ Message.count }"

# using ar
#
   Message.delete_all
   a = Time.now.to_f

   Message.transaction do
     messages.each{|message| Message.create! :content => message}
   end

   b = Time.now.to_f

   puts "using ar"
   puts "elapsed: #{ b - a }"
   puts "count: #{ Message.count }"



cfp:~/rails_root > ./script/runner a.rb
using sqlite3
elapsed: 0.222311019897461
count: 10000

using ar
elapsed: 7.75591206550598
count: 10000

0.2 seconds for 100000 records seems plenty fast to me.  7 seconds not  
so much.



a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being  
better. simply reflect on that.
h.h. the 14th dalai lama