From: Robert Klemme <shortcutter@...>
Date: 2006-12-24T00:30:05+09:00
Subject: Re: Question: Downloading files with open(-uri)?

On 23.12.2006 15:40, Mariano Kamp wrote:
> 
> On Dec 23, 2006, at 3:20 PM, William James wrote:
> 
>> Mariano Kamp wrote:
>>> Hi,
>>>
>>>    I could need a quick hand here.
>>>
>>>    I want to watch the RailsConf 2006 videos and want to download
>>> them with a script.
>>>
>>>    Unfortunately open("http:/xx") never comes back? Any idea what I
>>> am doing wrong here?
>>>
>>>    I tested it with an URL that returns plain html and that worked
>>> fine. See the first line, ibm.com.
>>>
>>> require 'open-uri'
>>>
>>> urls = %w{
>>> http://ibm.com
>>> http://downloads.scribemedia.net/rails2006/03_martin_fowler_full.m4v
>>> http://downloads.scribemedia.net/rails2006/02_dave_thomas_full.m4v
>>> http://downloads.scribemedia.net/rails2006/01_dh_hansson.m4v
>>> http://downloads.scribemedia.net/rails2006/04_paul_graham_full.m4v
>>> http://downloads.scribemedia.net/rails2006/06_railsCorePanel_full.m4v
>>> http://downloads.scribemedia.net/rails2006/07_why_lucky_stiff.m4v
>>> }
>>> BUFFER_SIZE = 1_024*1_024*1
>>>
>>> urls.each do |url|
>>>    puts "downloading #{url}"
>>>    open(url) do |input|
>>>      puts "opened connection."
>>>      output = open(url.split(/\//).last, "wb")
>>>      while (buffer = input.read(BUFFER_SIZE))
>>>        print "."
>>>        $stdout.flush
>>>        output.write(buffer)
>>>      end
>>>      output.close
>>>    end
>>>    puts "done."
>>> end
>>> puts "All downloads done."
>>
>> There's nothing wrong with your program; I tested it by
>> downloading a picture.  If you have a dial-up connection, maybe
>> the transfer is progressing very slowly.
> 
> Hey Bill,
> 
>   hmm, not sure. If I change the BUFFER_SIZE to 1KB I still don't see 
> anything and the "puts 'opened connection'" should at least be visible, 
> shouldn't it?
> 
>   Anyways I have a 6 MBit/s downstream so even a 1MB buffer shouldn't be 
> a problem.
> 
>   I also suspected that the server is checking for deep links and would 
> evaluate the referer in the process, but when I enter one of the urls 
> directly into my browser it works.
> 
>   Very strange.

I observe the same behavior that you see.  I have no knowledge of 
openuri internals but here's my theory: the page is probably loaded 
completely before open returns.  This would explain why you see the dots 
from ibm.com in one go.  I would test the same with net/http and see 
whether there is any difference.  Make sure to use the stream form.

Kind regards

	robert