From: Horacio Sanson <horacio.sanson@...>
Date: 2006-08-28T12:44:53+09:00
Subject: Rescue http Bad File Descriptor error (EBADF)

------=_Part_32882_23577560.1156736690632
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

I have a little web spider that scraps several web pages. Sometimes the
script gets a Bad File Descriptor error and the script bails out.

As far as I can understand this error is an OS (Windows XP) error and there
is nothing ruby can do to avoid it (Maybe XP cannot handle so much http
connections so rapidly). But I cannot find a way to recover from the
error.... I don't want the script to bail out, but simply continue with the
next page.

I have tried to rescue and try/catch the error with all imaginable exception
classes, still the script always bails out when this error ocurrs. I know
this error is in the ruby net/http library since I have used Mechanize,
Http-access2, http-access and all of them suffer from this error.

Here are the details of the error

c:/ruby/lib/ruby/1.8/net/http.rb:562:in `initialize': Bad file descriptor -
connect(2) (Errno::EBADF)
        from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `connect'
        from c:/ruby/lib/ruby/1.8/timeout.rb:48:in `timeout'
        from c:/ruby/lib/ruby/1.8/timeout.rb:76:in `timeout'
        from c:/ruby/lib/ruby/1.8/net/http.rb:562:in `connect'
        from c:/ruby/lib/ruby/1.8/net/http.rb:555:in `do_start'
        from c:/ruby/lib/ruby/1.8/net/http.rb:544:in `start'
        from c:/ruby/lib/ruby/gems/1.8/gems/mechanize-0.4.7/lib/mechanize.rb:279:in
`fetch_page'
        from c:/ruby/lib/ruby/gems/1.8/gems/mechanize-0.4.7/lib/mechanize.rb:138:in
`get'
        from c:/ruby/lib/ruby/gems/1.8/gems/mechanize-0.4.7/lib/mechanize.rb:193:in
`submit'
        from ./urls.rb:1001
        from ./urls.rb:998
        from ./spider.rb:540:in `get_pages'
        from ./spider.rb:470:in `start'
        from ./spider.rb:733:in `process_link'
        from ./spider.rb:677:in `start'
        from ./spider.rb:670:in `start'
        from main2.rb:4

I repeat that this error is very sporadic... it occurs at different times,
different pages and using either Mechanize or http-access2 or simple
open-uri.

The code I use (Mechanize case) is simply:

def get_page
     @agent = WWW::Mechanize.new
     @page = @agent.get @url
end

This code is executed for each page (100~). Adding rescue or try/catch
anywhere inside the get_page method or around it when called does not catch
the error... the script always stops when the error ocurrs.  Also since this
error is very sporadic I cannot reproduce it making it very difficult to
debug.

Any tips are very appreciated.

Horacio

------=_Part_32882_23577560.1156736690632--