From: Robert Klemme <shortcutter@...>
Date: 2006-12-24T01:10:06+09:00
Subject: Re: Question: Downloading files with open(-uri)?

On 23.12.2006 16:29, Robert Klemme wrote:
> On 23.12.2006 15:40, Mariano Kamp wrote:
>>
>> On Dec 23, 2006, at 3:20 PM, William James wrote:
>>
>>> Mariano Kamp wrote:
>>>> Hi,
>>>>
>>>>    I could need a quick hand here.
>>>>
>>>>    I want to watch the RailsConf 2006 videos and want to download
>>>> them with a script.
>>>>
>>>>    Unfortunately open("http:/xx") never comes back? Any idea what I
>>>> am doing wrong here?
>>>>
>>>>    I tested it with an URL that returns plain html and that worked
>>>> fine. See the first line, ibm.com.
>>>>
>>>> require 'open-uri'
>>>>
>>>> urls = %w{
>>>> http://ibm.com
>>>> http://downloads.scribemedia.net/rails2006/03_martin_fowler_full.m4v
>>>> http://downloads.scribemedia.net/rails2006/02_dave_thomas_full.m4v
>>>> http://downloads.scribemedia.net/rails2006/01_dh_hansson.m4v
>>>> http://downloads.scribemedia.net/rails2006/04_paul_graham_full.m4v
>>>> http://downloads.scribemedia.net/rails2006/06_railsCorePanel_full.m4v
>>>> http://downloads.scribemedia.net/rails2006/07_why_lucky_stiff.m4v
>>>> }
>>>> BUFFER_SIZE = 1_024*1_024*1
>>>>
>>>> urls.each do |url|
>>>>    puts "downloading #{url}"
>>>>    open(url) do |input|
>>>>      puts "opened connection."
>>>>      output = open(url.split(/\//).last, "wb")
>>>>      while (buffer = input.read(BUFFER_SIZE))
>>>>        print "."
>>>>        $stdout.flush
>>>>        output.write(buffer)
>>>>      end
>>>>      output.close
>>>>    end
>>>>    puts "done."
>>>> end
>>>> puts "All downloads done."
>>>
>>> There's nothing wrong with your program; I tested it by
>>> downloading a picture.  If you have a dial-up connection, maybe
>>> the transfer is progressing very slowly.
>>
>> Hey Bill,
>>
>>   hmm, not sure. If I change the BUFFER_SIZE to 1KB I still don't see 
>> anything and the "puts 'opened connection'" should at least be 
>> visible, shouldn't it?
>>
>>   Anyways I have a 6 MBit/s downstream so even a 1MB buffer shouldn't 
>> be a problem.
>>
>>   I also suspected that the server is checking for deep links and 
>> would evaluate the referer in the process, but when I enter one of the 
>> urls directly into my browser it works.
>>
>>   Very strange.
> 
> I observe the same behavior that you see.  I have no knowledge of 
> openuri internals but here's my theory: the page is probably loaded 
> completely before open returns.  This would explain why you see the dots 
> from ibm.com in one go.  I would test the same with net/http and see 
> whether there is any difference.  Make sure to use the stream form.

Try this (note, this will not follow redirects):

	robert


require 'net/http'
require 'uri'

urls = %w{
http://ibm.com
http://downloads.scribemedia.net/rails2006/03_martin_fowler_full.m4v
http://downloads.scribemedia.net/rails2006/02_dave_thomas_full.m4v
}

$stdout.sync=true

urls.each do |url|
   puts "downloading #{url}"

   Net::HTTP.get_response(URI.parse(url)) do |res|
     puts "opened connection."
     target = url.split(/\//).last
     puts "writing to #{target}"

     File.open(target, "wb") do |output|
       # next line will read in chunks but not provide option for dots...
       # res.read_body(output)
       res.read_body do |chunk|
	output.write(chunk)
	print "."
       end
     end
   end

   puts "done."
end

puts "All downloads done."