From: Lars Haugseth <njus@...>
Date: 2008-10-02T17:03:49+09:00
Subject: Re: Mirroring large files over HTTP

* Lars Haugseth <njus@larshaugseth.com> wrote:
> 
> I'm working on a script where I want to download large files off a remote
> web server and store on a local filesystem.
> 
> At the moment I'm using code like this:
> 
>   require 'open-uri'
> 
>   open(filename, 'w') do |file|
>     file.write(open(remote_url).read)
>   end
> 
> I assume this will read the complete content of the remote file into
> memory before writing it to the local file. If that assumption is
> correct, what is the best/easiest way to do a buffered piecemeal
> fetch/store? I've looked at the net/http library but haven't found
> anything in there that looks relevant to this.

Turns out the OpenURI module is indeed fetching the remote resource
in segments and storing to a temporary file. However, my code above
will read the complete contents of that file into memory before
writing it back out to another file.

By inspecting the OpenURI source code I've learned that this is how
it's done (sans proxy handling, error handling etc.):

  uri = URI.parse(url)
  http = Net::HTTP.new(uri.host, uri.port)
  http.request_get(uri.path) do |response|
    open(filename, 'w') do |file|
      response.read_body do |segment|
        file.write(segment)
      end
    end
  end

I'm a little surprised not to find any convenience method in the standard
libraries doing all this for me, though.

-- 
Lars Haugseth

"If anyone disagrees with anything I say, I am quite prepared not only to
 retract it, but also to deny under oath that I ever said it." -Tom Lehrer