From: James Tucker <jftucker@...>
Date: 2008-02-22T11:24:59+09:00
Subject: Re: Efficient file downloading


On 22 Feb 2008, at 01:54, Kyle Hunter wrote:

> Hello,
>
> I'm using open-uri to download files using a buffer. It seems very
> inefficient in terms of resource usage (CPU is ~10-20% in usage).
>
> If possible, I'd like some suggestions for downloading a file which
> names the outputted file the same as the URL, and does not actually
> write if the file comes out to a 404 (or some other exception hits).
>
> Current code:
> BUFFER_SIZE=4096

Try making that a lot lot bigger.

>
> def download(url)
>  from = open(url)
>  if (buffer = from.read(BUFFER_SIZE))
>    puts "Downloading #{url}"
>    File.open(url.split('/').last, 'wb') do |file|
>    begin
>      file.write(buffer)
>    end while (buffer = from.read(BUFFER_SIZE))
>    end
>  end
> end
> -- 
> Posted via http://www.ruby-forum.com/.
>