From: Brian Candler <B.Candler@...>
Date: 2007-08-30T21:30:27+09:00
Subject: Backport BufferedIO size increase?

A colleague of mine is using ruby-1.8.6-p36 and Net::HTTP on OpenWrt (a
small MIPS platform) to retrieve relatively large files.

He found performance to be poor. Using the test program below, using
Net::HTTP to retrieve a 3.2MB file from a machine on the same LAN took 64
seconds.

However, Net::HTTP uses a class called BufferedIO when reading data from a
socket, defined in net/protocol.rb, and BufferedIO has a fixed 1024-byte
buffer. I increased the block size, and this made some huge performance
gains:

Blocksize     Time
 (bytes)      (secs)

  1024         64.0
 16384          5.5
 65536          3.5

Looking in trunk, I see that in March (r12091) a constant BUFSIZE was
introduced, which now defaults to 16KB.

So, is there any reason not to back-port this to 1.8? As far as I can tell,
BufferedIO is only used by Net::HTTP, and it's marked "internal use only".
However, it does mean of HTTPResponse#read_body would get data in larger
chunks.

Thanks,

Brian.

--------------------------------------------------------------------------
#!/usr/bin/ruby -w
require 'net/http'

uri_str = "http://mypc.example.com/foo"  # 3.2MB file

URI.parse(uri_str)
count = 0
total = 0
starttime = nil
Net::HTTP.new(uri.host, uri.port).start { |http|
  http.request_get(uri.request_uri) { |response|
    response.read_body { |chk|
      if count == 0
        puts "Starting..."
        starttime = Time.now
      end
      count += 1
      total += chk.size
    }
  }
}
puts "Time taken: #{Time.now - starttime}"
puts "Chunks: #{count}"
puts "Total size: #{total}"