From: Eric Hodel <drbrain@...7.net>
Date: 2006-07-05T05:29:48+09:00
Subject: Re: open-uri speed

On Jul 3, 2006, at 9:51 PM, Elliot Temple wrote:

> I just discovered this. Lots of people already know, I'm sure, but  
> maybe some don't.

Your benchmark is not very illustrative of the problem, try this one:

$ cat timing.rb
require 'net/http'
require 'open-uri'

def timing(name, n)
   start_time = Time.now
   n.times do yield end
   end_time = Time.now
   puts "#{name} ran in: #{end_time - start_time} seconds."
end

def test(uri, n)
   timing 'raw socket', n do
     s = TCPSocket.open uri.host, uri.port
     s.write "GET #{uri.request_uri} HTTP/1.0\r\nHost: #{uri.host}\r\n 
\r\n"
     s.read.split("\r\n\r\n", 2).last
     s.close
   end

   Net::HTTP.start uri.host do |http|
     timing 'net/http cheat', n do
       r = http.get uri.request_uri
     end
   end

   timing 'net/http', n do
     Net::HTTP.start uri.host do |http|
       r = http.get uri.request_uri
     end
   end

   timing 'open-uri', n do
     uri.open do |x|
       x.read
     end
   end
end

n = 100

uri = URI.parse 'http://localhost/manual/'

p uri.read.length

test uri, n

uri = URI.parse 'http://localhost/manual/mod/mod_rewrite.html'

p uri.read.length

test uri, n

$ ruby timing.rb
9187
raw socket ran in: 1.184571 seconds.
net/http cheat ran in: 1.809506 seconds.
net/http ran in: 2.137558 seconds.
open-uri ran in: 2.606976 seconds.
87071
raw socket ran in: 1.729406 seconds.
net/http cheat ran in: 7.434297 seconds.
net/http ran in: 7.740268 seconds.
open-uri ran in: 13.605024 seconds.

You shouldn't cheat and have Net::HTTP reuse its connection.  (It  
seems socket setup/teardown costs 3ms over loopback on my machine.)

open-uri and Net::HTTP's performance both degrade significantly on  
larger files.  I believe this is due to their implementation, they  
both read into a buffer rather than fetching the entire response.   
open-uri buffers differently and provides progress callbacks which is  
probably the reason it performs worse the larger the file.

I tend to use open-uri because it has a simpler API.  I don't have to  
worry about handling redirects because it all gets taken care of for me.

-- 
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com