From: Mariano Kamp <mariano.kamp@...>
Date: 2006-12-23T23:40:39+09:00
Subject: Re: Question: Downloading files with open(-uri)?


On Dec 23, 2006, at 3:20 PM, William James wrote:

> Mariano Kamp wrote:
>> Hi,
>>
>>    I could need a quick hand here.
>>
>>    I want to watch the RailsConf 2006 videos and want to download
>> them with a script.
>>
>>    Unfortunately open("http:/xx") never comes back? Any idea what I
>> am doing wrong here?
>>
>>    I tested it with an URL that returns plain html and that worked
>> fine. See the first line, ibm.com.
>>
>> require 'open-uri'
>>
>> urls = %w{
>> http://ibm.com
>> http://downloads.scribemedia.net/rails2006/03_martin_fowler_full.m4v
>> http://downloads.scribemedia.net/rails2006/02_dave_thomas_full.m4v
>> http://downloads.scribemedia.net/rails2006/01_dh_hansson.m4v
>> http://downloads.scribemedia.net/rails2006/04_paul_graham_full.m4v
>> http://downloads.scribemedia.net/rails2006/06_railsCorePanel_full.m4v
>> http://downloads.scribemedia.net/rails2006/07_why_lucky_stiff.m4v
>> }
>> BUFFER_SIZE = 1_024*1_024*1
>>
>> urls.each do |url|
>>    puts "downloading #{url}"
>>    open(url) do |input|
>>      puts "opened connection."
>>      output = open(url.split(/\//).last, "wb")
>>      while (buffer = input.read(BUFFER_SIZE))
>>        print "."
>>        $stdout.flush
>>        output.write(buffer)
>>      end
>>      output.close
>>    end
>>    puts "done."
>> end
>> puts "All downloads done."
>
> There's nothing wrong with your program; I tested it by
> downloading a picture.  If you have a dial-up connection, maybe
> the transfer is progressing very slowly.

Hey Bill,

   hmm, not sure. If I change the BUFFER_SIZE to 1KB I still don't  
see anything and the "puts 'opened connection'" should at least be  
visible, shouldn't it?

   Anyways I have a 6 MBit/s downstream so even a 1MB buffer  
shouldn't be a problem.

   I also suspected that the server is checking for deep links and  
would evaluate the referer in the process, but when I enter one of  
the urls directly into my browser it works.

   Very strange.

Cheers,
Mariano