From: "naruse (Yui NARUSE)" Date: 2012-04-09T05:23:59+09:00 Subject: [ruby-core:44197] [ruby-trunk - Feature #5605] [PATCH] net/http: use IO.copy_stream for requests using body_stream Issue #5605 has been updated by naruse (Yui NARUSE). The original patch looks better than [ruby-core:40903]'s. drbrain, do you have any idea? If no one object it, I'll merge original one. ---------------------------------------- Feature #5605: [PATCH] net/http: use IO.copy_stream for requests using body_stream https://bugs.ruby-lang.org/issues/5605#change-25725 Author: normalperson (Eric Wong) Status: Assigned Priority: Low Assignee: naruse (Yui NARUSE) Category: lib Target version: 2.0.0 This significantly reduces both user and system CPU usage in the client while making uploads. When using plain HTTP with a known Content-Length, IO.copy_stream may also use sendfile() to further reduce user CPU time. I tested using the following script to upload a 1 gigabyte file. ------------------- net-http-upload.rb ------------------------- require "net/http" require "benchmark" require "tempfile" tmp = Tempfile.new("sparse") tmp.sysseek(1024 * 1024 * 1024) # 1 gigabyte tmp.syswrite(".") host = "127.0.0.1" port = 1234 res = nil Net::HTTP.start(host, port) do |http| put = Net::HTTP::Put.new("/") tmp.sysseek(0) put.body_stream = tmp put.content_length = tmp.size puts "with Content-Length: #{tmp.size}" bm = Benchmark.measure { res = http.request(put) } p [ res, res.body ] printf("utime=%0.3f stime=%0.3f\n", bm.utime, bm.stime) end Net::HTTP.start(host, port) do |http| put = Net::HTTP::Put.new("/") tmp.sysseek(0) put.body_stream = tmp put["Transfer-Encoding"] = "chunked" puts "with Transfer-Encoding: chunked" bm = Benchmark.measure { res = http.request(put) } p [ res, res.body ] printf("utime=%0.3f stime=%0.3f\n", bm.utime, bm.stime) end ------------------------ Results ------------------------------- before ------ with Content-Length: 1073741825 [#, "dcdd67a8f07b73796c0485890d48fa697966d09f\n"] utime=2.660 stime=1.680 with Transfer-Encoding: chunked [#, "dcdd67a8f07b73796c0485890d48fa697966d09f\n"] utime=6.230 stime=1.900 after ----- with Content-Length: 1073741825 [#, "dcdd67a8f07b73796c0485890d48fa697966d09f\n"] utime=0.010 stime=0.410 with Transfer-Encoding: chunked [#, "dcdd67a8f07b73796c0485890d48fa697966d09f\n"] utime=0.320 stime=0.620 ----------------------- Server setup -------------------------- Any server that'll accept chunked uploads works, I'm most familiar with unicorn. I used unicorn 4.1.1 with rewindable_input disabled, running t/sha1.ru from the Rainbows! source tree. t/sha1.ru returns the SHA1 of the uploaded body to verify the upload is sending correct data. $ git clone git://bogomips.org/rainbows $ unicorn -E none -p 1234 -c unicorn.conf rainbows/t/sha1.ru unicorn.conf.rb only had one line to disable rewindable input: rewindable_input false This was to prevent disk I/O from slowing the overall runtime of the test, it doesn't have a significant effect on measured CPU time in the client process even on the same box. -- http://bugs.ruby-lang.org/