From: Florian Frank <flori@...>
Date: 2006-07-16T11:15:15+09:00
Subject: Re: net/http performance

Luke Burton wrote:

>I just think that net/http's abysmal speed is somewhat anomalous. I am 
>confident there is a simple explanation - maybe some tight loop in there 
>doing something not particularly clever - but I haven't had the time as 
>yet to dive right into net/http and find the reason. And I haven't had 
>much success with Ruby profilers either. More suggestions welcome here!
>  
>
I think you're right. In the GET case (for both open-uri and net/http) I 
have found a possibility to speed up the download. The method 
BufferedIO#rbuf_fill in net/protocol.rb uses a fixed buffer size of 
1024. I changed this to 8192:

    def rbuf_fill
      timeout(@read_timeout) {
        @rbuf << @io.sysread(8192)
      }
    end

This lead to almost the same speed I could achieve with the use of 
direct sockets.

>I have gone ahead and changed the critical section of my code to use 
>TCPSocket instead. That solved the HTTP GET problem, but I still 
>struggle with PUTs:
>
>file = File.open(resultFile, "r")
>http = Net::HTTP.new(@uri.host, @uri.port)
>http.put("/put/" + URI.escape(File.basename(resultFile)), file.read)
>
>Now that's not real pleasant because it relies on snarfing the whole 
>file into memory first. I would have liked to do something like:
>
>http.put("/put/" + URI.escape(File.basename(resultFile))) do 
>|datasocket|
>    while file.eof? == false
>        datasocket.write(file.read(4096))
>    end
>end
>  
>
You can do this by setting Put#body_stream= to your IO object. The only 
problem is, that the related method also sets a very small buffer size:

    def send_request_with_body_stream(sock, ver, path, f)
      raise ArgumentError, "Content-Length not given and 
Transfer-Encoding is not `chunked'" unless content_length() or chunked?
      unless content_type()
        warn 'net/http: warning: Content-Type did not set; using 
application/x-www-form-urlencoded' if $VERBOSE
        set_content_type 'application/x-www-form-urlencoded'
      end
      write_header sock, ver, path
      if chunked?
        while s = f.read(1024)
          sock.write(sprintf("%x\r\n", s.length) << s << "\r\n")
        end
        sock.write "0\r\n\r\n"
      else
        while s = f.read(1024)
          sock.write s
        end
      end
    end

I think this is rather unfortunate. It would be better, if those methods 
would use higher buffer values and/or make them tweakable if necessary.