From: Yves-Eric Martin <yem_ruby-talk@...>
Date: 2009-04-13T12:40:32+09:00
Subject: Potential bug in Net::HTTP, and tentative patch

--------------000200000704060404040306
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi all,


I am building a nice little system using Ruby (on Rails), one part of
which uses Net::HTTP to retrieve some data over HTTP. Everything seems
to work fine, but on some requests, I get an EOFError.

As I found out, this problem has already been reported, but without any
answer. See:
http://rubyforge.org/forum/forum.php?thread_id=28826&forum_id=6052


I think I may have traced the problem back to a bug in Net::HTTP. Here
is a two-liner to reproduce the error:

$ irb
 >> require 'net/http'
=> true
 >> res = 
Net::HTTP.get_response(URI.parse('http://snapcasa.com/get.aspx?code=1000&size=m&url=www.google.com'))
EOFError: end of file reached
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:133:in 
`sysread'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:133:in 
`rbuf_fill'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:56:in 
`timeout'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:76:in 
`timeout'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:132:in 
`rbuf_fill'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:116:in 
`readuntil'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:126:in 
`readline'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2236:in 
`read_chunked'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2216:in 
`read_body_0'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2182:in 
`read_body'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2207:in 
`body'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2146:in 
`reading_body'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1061:in 
`request'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:957:in 
`request_get'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:380:in 
`get_response'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:547:in 
`start'
    from 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:379:in 
`get_response'
    from (irb):2>>


My quick analysis:

In http.rb, line 2236 the "read_chunked" function calls
@socket.readline. This "readline" function, in protocol.rb line 126,
calls readuntil("\n"). This works fine if the data chunk is
"\n-terminated", but throws an EOFError if it is not.

I am not sure about the underlying standards of chunked http, maybe the
data chunk is supposed to always been \n-terminated, and it may be a
mis-behaving server, but the fact is: I can get the example image fine
with any browser, but not with Net::HTTP.

As a tentative fix, I wrote a patch that catches the EOFError in
read_chunked. You will the patch file attached. With the patch, things
work fine:

$ irb
 >> require 'net/http'
=> true
 >> res = 
Net::HTTP.get_response(URI.parse('http://snapcasa.com/get.aspx?code=1000&size=m&url=www.google.com'))
=> #<Net::HTTPOK 200 OK readbody=true>
 >> File.open('test.jpg','w').write res.body
=> 2881
 >>

With the patch, the above gives me a perfectly fine JPEG file. However,
I am afraid my current patch, with a big begin...rescue around most of
the body of the read_chunked function, catches the EOFError at level
higher than necessary, which is not good practice...


Anyway, before continuing any further, could someone involved in the
development of Ruby take a look at this, confirm the existence of the
bug, and maybe even come up with a better fix?

PS: please let me know if I posted this in the wrong list, or if I
should open a bug report on some bug tracking system.


Thank you for your help.


-- 
Yves-Eric


--------------000200000704060404040306
Content-Type: text/plain; x-mac-type="0"; x-mac-creator="0";
 name="read_chunked_eoferror_tentative_fix.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="read_chunked_eoferror_tentative_fix.patch"

--- http.rb_ORIG	2009-04-13 12:13:15.000000000 +0900
+++ http.rb	2009-04-13 12:08:58.000000000 +0900
@@ -2232,17 +2232,21 @@
     def read_chunked(dest)
       len = nil
       total = 0
-      while true
-        line = @socket.readline
-        hexlen = line.slice(/[0-9a-fA-F]+/) or
-            raise HTTPBadResponse, "wrong chunk size line: #{line}"
-        len = hexlen.hex
-        break if len == 0
-        @socket.read len, dest; total += len
-        @socket.read 2   # \r\n
-      end
-      until @socket.readline.empty?
-        # none
+      begin
+        while true
+          line = @socket.readline
+          hexlen = line.slice(/[0-9a-fA-F]+/) or
+              raise HTTPBadResponse, "wrong chunk size line: #{line}"
+          len = hexlen.hex
+          break if len == 0
+          @socket.read len, dest; total += len
+          @socket.read 2   # \r\n
+        end
+        until @socket.readline.empty?
+          # none
+        end
+      rescue EOFError
+        # Tentative fix to EOFError bug on non-\n-terminated chunks
       end
     end
 

--------------000200000704060404040306--