Re: handle http request timed out

On Mon, Apr 15, 2013 at 3:56 PM, Fosiul Alam <[e-mail]> wrote:
> Hi
> i have written a script which connects to a website and read from their
> but some time that website is soo slow that it gets time out .
> so i have done this
>
>
> http = Net::HTTP.new(uri.host, uri.port)
> http.read_timeout = 500
> request = Net::HTTP::Get.new(uri.request_uri)
>
> begin
> response = http.request(request)
> code=response.code
> log_info("Response Code - #{code}")
>
> end until response.code == "200"
>
>
> But it gets time out .. now i can increase the http time out but thats
> not a solution
>
> is not there any way that i can do ..
>
> As long as response.code == 200, it will keep try to get request without
> producing time out or any error...
> i thought the code i wrote it will do that job , but it does not ...
> where i am doing wrong ??
> or how can i get what i am looking for ??

With this loop, you are making a new request each time, which starts
the whole retrieval process over again. AFAIK, the only way to make
this work is by setting the timeout long enough to retrieve all the
data. What you can do is set the get request inside a begin-rescue-end
block and use a progressive timeout backoff and a retry. Limit the
number of retries so you don't end up with an infinite loop. Robert
Klemme showed me how to do this with another sort of request in an
earlier message by passing a block to a method. I implemented it
something like this:


def retry_http_call(http,limit=3,backoff=500,&block)

  tries = 0

  begin
    yield http
  rescue Exception => e
    tries += 1
    http.read_timeout = backoff * tries
    retry if tries < limit
  end

end

Then you can use it like so:

Net::HTTP.start(uri.host, uri.port) do |http|

  request = Net::HTTP::Get.new(uri.request_uri)
  response = retry_http_call(http, 5) { |http| http.request(request) }

end

You may want to fiddle with the number of retries and the progressive
back off increment to see what is optimal for your implementation.


(Robert, if you would kindly check me on this?)