From: Luke Burton <luke@...>
Date: 2006-07-17T11:54:44+09:00
Subject: Re: net/http performance


Hi all,

Thanks to the many thoughtful suggestions here, I have implemented an 
easy workaround to this problem that doesn't involve giving up the 
net/http library completely.

If you go back to my original benchmark testing code in the original 
post, I have made the following changes. Basically I override the 
necessary methods to tweak the buffer size:

class OverrideInternetMessageIO < Net::InternetMessageIO
  def rbuf_fill
    timeout(@read_timeout) {
      @rbuf << @socket.sysread(65536)
    }
  end
end

class NewHTTP < Net::HTTP
  def NewHTTP.socket_type
     OverrideInternetMessageIO
  end
end

Benchmark.bm(10) do |time|
  out = File.new("/tmp/net.tar.bz2", "w")
  time.report("net/http - bigbuffer") do
    NewHTTP.start uri.host, uri.port do |http|
      http.request_get(uri.request_uri) do |response|
        response.read_body do |segment|
          out.write(segment)
        end
      end
    end
  end
  out.close
end

After making all those changes, we see the following new results for the 
10 MB file transfer from WEBrick:

                           user     system      total         real
net/http - big buffer  0.360000   0.390000   0.750000 (  0.991848)

That's still twice as slow as a raw TCPSocket, but it's now definitely 
in the realm of "usable for large file transfers".

I couldn't make any real recommendations on what the buffers size should 
be. I imagine it's a trade off between the OS kernel's buffer size, TCP 
packet size, and memory footprint of your application. Do HTTP clients 
normally automatically negotiate a buffer? Do they pick one based on 
content type? What are the common optimisations, and should net/http 
follow them?

I tested a couple of values and found anything past 65536 bytes started 
giving negligible returns, on my G5 running OS X 10.4.7 (ruby 1.8.2 - 
i.e. the default OS X install).

Thanks again for all the pointers and commentary, and I hope to see a 
more robust solution in a future version :)

Regards,

Luke.


-- 
Posted via http://www.ruby-forum.com/.