From: Daniel Martin <martin@...>
Date: 2006-07-17T23:21:07+09:00
Subject: Re: net/http performance

Luke Burton <luke@burton.echidna.id.au> writes:

> After making all those changes, we see the following new results for the 
> 10 MB file transfer from WEBrick:
>
>                            user     system      total         real
> net/http - big buffer  0.360000   0.390000   0.750000 (  0.991848)
>
> That's still twice as slow as a raw TCPSocket, but it's now definitely 
> in the realm of "usable for large file transfers".

Does this improve noticeably if you use the read_nonblock suggestion
given earlier in the thread?  Say:

class OverrideInternetMessageIO < Net::InternetMessageIO
  def rbuf_fill
    begin
      @rbuf << @io.read_nonblock(65536)
    rescue Errno::EWOULDBLOCK
      if IO.select([@io], nil, nil, @read_timeout)
        @rbuf << @io.read_nonblock(65536)
      else
        raise Timeout::TimeoutError
      end
    end
  end
end

As for the appropriate buffer size, for what it's worth apache uses
this structure to read into when it's acting as a proxy server and
reading someone else's output:

    char buffer[HUGE_STRING_LEN];

Where HUGE_STRING_LEN is defined in various apr (Apache Portable
Runtime) headers as 8192. (In Apcahe 1.3 it was in 'httpd.h')

I don't have time to track through the mozilla source to find out what
buffer size they use.