Re: Zlib::Inflate changes to net/http.rb

On Fri, 28 Dec 2007, Filipe Lautert wrote:

> 
> Hi,
> 
> from ruby 1.8 to 1.9, NET::HTTP.get method improved a lot, and now it has
> support to compression, that is really good.
> 
> There is just one small "problem" with content-encoding = deflate
> (lib/net/http.rb:817): it works as defined by the specification.
> 
> But some browsers/servers have problem with this, as stated in zib site [1]:
> 'While the HTTP 1.1 RFC 2616 correctly points to the zlib specification in RFC
> 1950 for the "deflate" transfer encoding, there have been reports of servers
> and browsers that incorrectly produce or expect raw deflate data per the
> deflate specficiation in RFC 1951, most notably Microsoft. '
> 
> And mongrel too :) And apache mod_deflate default configuration too (2) -
> those are just some examples.

At the time of implementing this I pointed out that I was unable to
find any servers supporting deflate at all -- including W3C servers.
Are there any on line that I can actually test against?

> mongrel uses raw deflate instead of zlib because it had problems with IE and
> safari browsers when sendind correct zlib data. And as mongrel, many other
> guys (apache, for instance) are using raw deflate by default, because it is
> wrong but "just works".
> 
> The patch that follow change NET::HTTP.get to first try the correct Zlib
> format. If it fails, then try the raw deflate format:

I don't have final say on this of course but I'd be happy about this
patch.  I'd suggest you include the URL's in the comments as well,
and the remark about windowsize being negative.

> Index: lib/net/http.rb
> ===================================================================
> --- lib/net/http.rb     (revision 14746)
> +++ lib/net/http.rb     (working copy)
> @@ -815,7 +815,13 @@
>              r.body= Zlib::GzipReader.new(StringIO.new(the_body)).read
>              r.delete("content-encoding")
>            when "deflate"
> -            r.body= Zlib::Inflate.inflate(the_body);
> +           begin
> +              r.body= Zlib::Inflate.inflate(the_body);
> +           rescue Zlib::DataError
> +             #no luck with Zlib decompression. Let's try with raw deflate,
> +             #like some broken browsers do.
> +              r.body= Zlib::Inflate.new(-Zlib::MAX_WBITS).inflate(the_body);
> +           end
>              r.delete("content-encoding")
>            when "identity"
>              ; # nothing needed
> 
> 
> 
> 1. http://www.gzip.org/zlib/zlib_faq.html#faq38
> 2.
> https://svn.apache.org/viewvc/httpd/httpd/trunk/modules/filters/mod_deflate.c?revision=581198&view=markup
> (search for "windowsize is negative to suppress Zlib header")
> 
> Cheers,
> 
> filipe
> 
        Thank you,
        Hugh