From: nvp <ruby-lists@...>
Date: 2009-12-31T08:50:08+09:00
Subject: Parallel::ForkManager, Net::HTTP and catching Timeout::Error

Hello,

I'm about to release Parallel::ForkManager 1.1 and while generating
new examples (that use Net::HTTP) for PFM 1.1 features, I catch the
following error *sometimes* (maybe 10-15% of the time) when I try to
connect to a URL where the host part doesn't exist or isn't reachable:

/usr/lib/ruby/1.8/timeout.rb:60:in `rbuf_fill': execution expired
(Timeout::Error)
        from /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
        from /usr/lib/ruby/1.8/net/protocol.rb:86:in `read'
        from /usr/lib/ruby/1.8/net/http.rb:2212:in `read_body_0'
        from /usr/lib/ruby/1.8/net/http.rb:2173:in `read_body'
        from /usr/lib/ruby/1.8/net/http.rb:773:in `get'
        from /usr/lib/ruby/1.8/net/http.rb:1053:in `request'
        from /usr/lib/ruby/1.8/net/http.rb:2136:in `reading_body'
        from /usr/lib/ruby/1.8/net/http.rb:1052:in `request'
        from /usr/lib/ruby/1.8/net/http.rb:1037:in `request'
        from /usr/lib/ruby/1.8/net/http.rb:543:in `start'
        from /usr/lib/ruby/1.8/net/http.rb:1035:in `request'
        from /usr/lib/ruby/1.8/net/http.rb:772:in `get'
        from ./parallel_http_get2.rb:36
        from ./lib/parallel/forkmanager.rb:232:in `call'
        from ./lib/parallel/forkmanager.rb:232:in `start'
        from ./lib/parallel/forkmanager.rb:232:in `fork'
        from ./lib/parallel/forkmanager.rb:232:in `start'
        from ./parallel_http_get2.rb:30
        from ./parallel_http_get2.rb:26:in `each'
        from ./parallel_http_get2.rb:26

Based on my review of the error message, it would seem to be a simple
case that I should catch (rescue) 'Exception' because Timeout::Error
is sending to stderr or such, but when I try to catch 'Exception' in
my test program, nothing happens!  I never catch the error!

Here's my test code:

#!/usr/bin/env ruby

require 'net/http'
require 'lib/parallel/forkmanager'

save_dir = '/tmp'

my_urls = [
    'http://www.cnn.com/index.html',
    'http://oreilly.com/index.html',
    'http://www.cakewalk.com/index.html',
    'http://www.asdfsemicolonl.kj/index.htm'
]
my_timeout = 5 # seconds

max_proc = 20
pfm = Parallel::ForkManager.new(max_proc)

pfm.run_on_finish(
    lambda {
        |pid,exit_code,ident|
        print "** PID (#{pid}) for #{ident} exited with code #{exit_code}!\n"
    }
)

my_urls.each {
    |my_url|

    begin
        pfm.start(my_url)  {
            url = URI.parse(my_url)
            out_file = save_dir + '/' + url.host + '.txt';

            http = Net::HTTP.new(url.host, url.port)
            http.open_timeout = http.read_timeout = my_timeout
            res = http.get(url.path)
            status = res.code

            if status.to_i == 200
                f = File.open(out_file, 'w')
                f.print res.body
                f.close()
                exit 0
            else
                exit 255
            end

            http = Net::HTTP.new(url.host, url.port)
            http.open_timeout = http.read_timeout = my_timeout
            res = http.get(url.path)
            status = res.code

            if status.to_i == 200
                f = File.open(out_file, 'w')
                f.print res.body
                f.close()
                exit 0 # start() with a block means that we exit with
status or else it's 0 all the time.
            else
                exit 255
            end
        } # end pfm.start { ... }
    rescue Exception => e
        print "Arggh, exception: ", e, "\n"
        exit 255
    end
}

pfm.wait_all_children()

# end

Would somebody be able to shed some light on why I'm unable to handle
the exceptions that Net::HTTP is throwing?  Am I just trying to catch
the wrong exception or is there something else?  Is it because I'm
trying to handle the exception in the child?

Note that this is: ruby 1.8.7 (2009-06-12 patchlevel 174) [i486-linux]
under Ubuntu 9.10 (Karmic).

-- 
NP