[ruby-core:38158] Proposal to merge net2-http for Ruby 1.9.4

I gave a talk at Ruby Kaigi about my work on Net::HTTP (
https://github.com/wycats/net-http) and Net::Reactor (
https://github.com/wycats/net2-reactor), and asked that they be considered
for inclusion into Ruby. Several people suggested that I send an email to
this list, so here I am! Since Ruby 1.9.3 is almost finished, I would
propose to include my improvements to Net::HTTP in Ruby 1.9.4.

When I started working on improving Net::HTTP, my main goal was to make it
possible to make a request without reading the response at the same time.
Since then, I have also worked on a number of other improvements.

   - All kinds of requests can support GZip and Inflate
   - It is possible to read_nonblock from Net::HTTP response, even if the
   response has Transfer-Encoding: Chunked, keepalive or compression. This
   makes it possible to use Net::HTTP in reactor libraries, instead of writing
   whole new HTTP libraries for each kind of reactor
   - Make decompression (Gzip and Inflate) use a general-purpose response
   middleware stack that can be set up per-request on the response body. This
   is still experimental.
   - Make the body parsers (regular body and chunked encoding) separate
   objects that can support either blocking read or non-blocking read.
   - A number of structural improvements to make the code easier to work
   with (breaking it up into multiple files, etc.)
   - Improving the tests to test non-blocking read across all kinds of
   requests, and add tests for decompression
   - A number of smaller improvements, such as making Net::HTTP.get(string)
   work

My intent was to retain full backwards compatibility across the API with one
small change:

When using the block form of the request *instance *method (
http.request(...) { ... } ), the behavior remains the same: the body is read
synchronously. However, when a block is not passed, the body is not read
until it is requested. This makes read_nonblock possible. This means that
when a block is not passed, it is up to the consumer of the API to close the
response when they are done with it.

This should not result in any major issues for several reasons:

   - The convenience class methods (Net::HTTP.get, etc.) all use the block
   form, which eagerly reads the body
   - The convenience instance methods (Net::HTTP#get, Net::HTTP#post) use
   the block form, which eagerly reads the body
   - The block form of Net::HTTP.start (as well as Net::HTTP#finish) will
   close the socket
   - The low-level method Net::HTTP#request, when used without a block, will
   not eagerly read the body, but clients that make low-level HTTP requests
   that can receive a body will usually read it
   - In the case of keepalive connections, the previous request body is read
   before a new request can begin

In short, the change, which is required in order to support non-blocking
reads, is extremely constrained, limited to the low-level API, and matches
the normal API used by Ruby I/O.

Thank you for your consideration,

Yehuda Katz
Chief Technologist | Strobe
(ph) 718.877.1325