Net/HTTP in 1.9.2dev is already working as you descri= bed with two exceptions:

1 - It uses nonblocking I/O

2 - It uses a 16KB buffer

The 1MB buffer you = are using is too large for several reasons:

1 - You use blocking I/O calls, this works great if y= ou are downloading from the local machine. OTOH if the server you are conne= cting to is slow, attempting to collect such a large buffer in a blocking m= anner is guaranteed to cause serious delays.

2 - If multiple threads are downloading data together (whic= h is normal for crawlers) you will be allocating a lot of memory at once th= is way (mutliple strings up to 1MB each). This will propably tax the GC.

You might want = to upgrade to 1.8.7 (which implements read_nonblock) and modify your patch = for a non blocking behavior

oldmoe

On = Sat, Jun 27, 2009 at 9:47 PM, Steven Hartland <redmine@ruby-lang.org> wrot= e:

Issue #1696 has been upda= ted by Steven Hartland.

File protocol.rb.patch added

Silly bug in original patch, fixed version here, results still consistent w= ith original.

It may be interesting to note that this also brings net/http speed inline w= ith that of libcurl ruby gems
open.core =A0 =A0 15.296875 =A0 1.437500 =A016.734375 ( 22.416853)
open.patched =A0 0.093750 =A0 0.218750 =A0 0.312500 ( =A00.340683)
taf2-curb =A0 =A0 =A00.171875 =A0 0.273438 =A0 0.445312 ( =A00.546568)

----------------------------------------

http://redmine.ruby-lang.org/issues/show/1696

----------------------------------------

http://redmine.ruby-lang.org

--001485f7c36cc7f0bd046d5c419d--