[ruby-talk:12748] Re: http page download question

Your explanation and help are very nice. However, the page in question is
available via other means (browser, scripting language Rebol using
command print read http://[url]. When getting a page, does one need to
follow any protocol between different page reads like closing the
connection, creating an new instance of the http connection object 
(Net::HTTP.new('radio.cbc.ca')), or anything else. My concern is that if
I am not making any errors, Ruby may be causing the server connected to 
to shut down or something else. It should be fairly simple to get a page,
given what looks like an excellent HTTP library.

In article <[e-mail]>, "Minero Aoki"
<[e-mail]> wrote:

> 
>   In mail "[ruby-talk:12601] http page download question"
>     "Ian Marsman" <[e-mail]> wrote:
> 
>> I am writing a script to download webpages from a favourite radio
>> program in order to grab the URLs ot MP3s of recent shows that I want
>> to then pass on to XMMS to play. I am able to get the pages, but often
>> I get an error that the connection has been reset by the peer,
>> specifically:
>> 
>> /usr/lib/ruby/1.6/net/protocol.rb:582:in `sysread': Connection reset by
>> peer (Errno::ECONNRESET)
> 
> In this case what ruby is saying is all. connection is reset by
> (http) server. This error is thrown on, e.g. server down.
> 
> Catching error and retrying is only solution, I think. You can also
> fetch rest of entity instead of whole body like this:
> 
>   http.get path, { 'Range' => 'bytes=1024-2047' }
> 
> this code get body[1024..2047]. Then whole of "resume" get is (not
> tested yet):
> 
>   # File.unlink 'data' if File.file? 'data'   # if you need
>   param = nil read = 0 begin
>     Net::HTTP.start(host) do |http|
>       File.open( 'data', 'a' ) do |f|
> 	http.get( path, param ) do |str|
> 	  f.write str read += str.size
> 	end
>       end
>     end
>   rescue Errno::ECONNRESET
>     param = { 'Range' => "bytes=#{read}-" } retry
>   end
> 
> NOTE: Don't retry in start{} block because Net::HTTP object does not
> clear internal state (it's a bug).
> 
> 
> Minero Aoki