[ruby-core:40900] [ruby-trunk - Feature #5605] [PATCH] net/http: use IO.copy_stream for requests using body_stream

Issue #5605 has been updated by Eric Hodel.


=begin
For the chunked? half, would looping over (({IO.copy_stream f, sock.io, 1024})) be faster? 
=end

----------------------------------------
Feature #5605: [PATCH] net/http: use IO.copy_stream for requests using body_stream
http://redmine.ruby-lang.org/issues/5605

Author: Eric Wong
Status: Open
Priority: Low
Assignee: 
Category: lib
Target version: 2.0.0


This significantly reduces both user and system CPU usage in the
client while making uploads.  When using plain HTTP with a
known Content-Length, IO.copy_stream may also use sendfile() to
further reduce user CPU time.

I tested using the following script to upload a 1 gigabyte file.

------------------- net-http-upload.rb -------------------------
require "net/http"
require "benchmark"
require "tempfile"
tmp = Tempfile.new("sparse")
tmp.sysseek(1024 * 1024 * 1024) # 1 gigabyte
tmp.syswrite(".")
host = "127.0.0.1"
port = 1234
res = nil

Net::HTTP.start(host, port) do |http|
  put = Net::HTTP::Put.new("/")
  tmp.sysseek(0)
  put.body_stream = tmp
  put.content_length = tmp.size
  puts "with Content-Length: #{tmp.size}"
  bm = Benchmark.measure { res = http.request(put) }
  p [ res, res.body ]
  printf("utime=%0.3f stime=%0.3f\n", bm.utime, bm.stime)
end

Net::HTTP.start(host, port) do |http|
  put = Net::HTTP::Put.new("/")
  tmp.sysseek(0)
  put.body_stream = tmp
  put["Transfer-Encoding"] = "chunked"
  puts "with Transfer-Encoding: chunked"
  bm = Benchmark.measure { res = http.request(put) }
  p [ res, res.body ]
  printf("utime=%0.3f stime=%0.3f\n", bm.utime, bm.stime)
end

------------------------ Results -------------------------------

before
------
with Content-Length: 1073741825
[#<Net::HTTPOK 200 OK readbody=true>,
 "dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
utime=2.660 stime=1.680
with Transfer-Encoding: chunked
[#<Net::HTTPOK 200 OK readbody=true>,
 "dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
utime=6.230 stime=1.900

after
-----
with Content-Length: 1073741825
[#<Net::HTTPOK 200 OK readbody=true>,
 "dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
utime=0.010 stime=0.410
with Transfer-Encoding: chunked
[#<Net::HTTPOK 200 OK readbody=true>,
 "dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
utime=0.320 stime=0.620

----------------------- Server setup --------------------------
Any server that'll accept chunked uploads works, I'm most
familiar with unicorn.  I used unicorn 4.1.1 with
rewindable_input disabled, running t/sha1.ru from the Rainbows!
source tree.  t/sha1.ru returns the SHA1 of the uploaded body to
verify the upload is sending correct data.

  $ git clone git://bogomips.org/rainbows
  $ unicorn -E none -p 1234 -c unicorn.conf rainbows/t/sha1.ru

unicorn.conf.rb only had one line to disable rewindable input:

  rewindable_input false

This was to prevent disk I/O from slowing the overall runtime
of the test, it doesn't have a significant effect on measured
CPU time in the client process even on the same box.



-- 
http://redmine.ruby-lang.org