[ruby-core:45532] Re: [ruby-trunk - Feature #6492][Open] Inflate all HTTP Content-Encoding: deflate, gzip, x-gzip responses by default

Eric Hodel <[e-mail]> wrote:
> On Jun 8, 2012, at 5:28 PM, Eric Wong <[e-mail]> wrote:
> 
> > I like Net::HTTP being able to inflate compressed responses.
> > 
> > However, I think doing this by default is exploitable by an evil server.
> > A server could compress a huge file of zeroes to trigger an
> > out-of-memory conditions in existing code that uses Net::HTTP.

> Net::HTTP#get does this by default already, this patch (and #6494)
> make this the default for all requests.

I've always considered Net::HTTP#get (or anything where slurping is done
blindly) dangerous when talking to untrusted servers regardless of gzip.

> If you aren't using the API to handle a compressed 100MB request
> (Net::HTTPResponse#read_body with a block) you probably can't handle
> an raw 100MB response, so what is the difference besides bandwidth
> cost of the server?

With your patch, I'm getting 16M chunks from read_body.  Maybe on newer
systems, 16M is "safe" to slurp in memory.  I think it's too big, but I
may also be a dinosaur :)

Also, HTTP servers may blindly send Content-Encoding:gzip data
regardless of whether the client requested with Accept-Encoding:gzip or
not.  I seem to recall reading of a major website that forces gzip on
visitors regardless of their Accept-Encoding:.

------------------------------ 8< -----------------------------
require 'uri'
require 'net/http'

# feel free to use this URL for testing
uri = URI('http://yhbt.net/x')

Net::HTTP.start(uri.host, uri.port) do |http|
  request = Net::HTTP::Get.new(uri.request_uri)
  request["Accept-Encoding"] = "gzip"
  http.request request do |response|
    response.read_body do |chunk|
      p [ chunk.bytesize ]
    end
  end
end
------------------------------ 8< -----------------------------

I only used "gzip -9" to generate this test example, I'm not sure if
there are ways to use zlib to compress even more aggressively.

Achieving bzip2-level compression ratios would be very scary :)
  dd if=/dev/zero bs=1M count=1000 | bzip2 -c -9 | wc -c
  => 753