From: Chad Burt <chad@...>
Date: 2006-11-02T09:43:30+09:00
Subject: Net:HTTP performance downloading large files

Hi folks,
I'm working on my first ruby/rails project and have run into my first 
major problem.
My application uses a web api for a scientific database and I am trying 
to download a very large file(150mb) using Net:HTTP.post_form. This 
operation is turning out to be very slow.

Net:HTTP is at least 10x slower than using my browser to download this 
file on my local network. It also pegs one of my processors while doing 
it.

It seems to have something to do with a "green thread" issue explained 
here :
http://headius.blogspot.com/2006_06_01_archive.html#114996043877111235

Are there any alternatives to using Net:HTTP to download files off the 
web with ruby?


---------- Code in question -----------
  def Metacat.read(docid)
    #load uri set in environment.rb
    uri = URI.parse(Path_to_metacat)
    #uri.query = "action=read&qformat=xml&docid=#{docid}"
    response = Net::HTTP.post_form(uri, {
      'action'  =>  'read',
      'qformat' =>  'xml',
      'docid'   =>  docid
    })
    #this line will raise an exception if post failed
    response.value
    if(response.content_type == "text/xml")
      doc = REXML::Document.new(response.body)
      #check to see if Metacat is sending an error message or EML
      if(doc.root.name == 'error')
        nil
      else
        Eml.new(response.body)
      end
    elsif(response.content_type == "text/plain")
      DataTable.new(docid, response.body)
    end
  end
--------------------------------------

File I'm trying to download :
http://data.piscoweb.org/catalog/metacat?action=read&qformat=xml&docid=HMS001_020ADCP019R00_20060612.40.1

-- 
Posted via http://www.ruby-forum.com/.