From: tedmilkey@...
Date: 2008-02-19T14:29:57+09:00
Subject: REXML document creation speed

Hi,

Please excuse my ignorance, but I'm new to this.

I've written a script that downloads historical stock quotes as .csv,
parses it, then writes out an XML doc that I can then use elsewhere.

It works as designed, but it's *dog slow* and I don't see why.  It can
take over 5 minutes to run, and nearly all of that time is writing the
XML docs (I've tested it running the script with the XML document
creation lines commented out and it takes only seconds).

The script is below:

require 'rubygems'
require 'net/http'
require 'FasterCSV'
require 'rexml/document'
include REXML
puts "Start #{Time.now()}"
symbols = Array.new
xml_symbols_doc = Document.new(File.new("symbols.xml"))
for i in 1..xml_symbols_doc.root.elements.size
   symbols[i] = xml_symbols_doc.root.elements[i].get_text.value
end
for i in 1..(symbols.length - 1)
   quote_source_url = "http://ichart.finance.yahoo.com/table.csv?
s=#{URI.encode(symbols[i])}"
   quote_response =
Net::HTTP.get_response(URI.parse(quote_source_url))
   csv_quotes = FasterCSV.parse(quote_response.body, {:headers =>
true, :header_converters => :symbol})
   xml_quotes = Document.new
   xml_quotes << XMLDecl.new
   xml_quotes.add_element("quotes", {"symbol" => "#{symbols[i]}"})
   for j in 0..(csv_quotes.length - 1)
      quote = Element.new("quote")
      for k in 0..(csv_quotes.headers.length - 1)
         quote.add_element("#{csv_quotes.headers()[k]}").text =
"#{csv_quotes[j][k]}"
      end
      xml_quotes.root << quote
   end
   xml_quotes_output_file = File.new("#{symbols[i]}.xml", "w+")
   xml_quotes.write(xml_quotes_output_file, 3)
   puts "#{symbols[i]} OK. File here: #{xml_quotes_output_file.path}"
end
puts "End #{Time.now()}"

Any suggestions as to how I can make this script run (much) faster are
greatly appreciated!

Thanks for your help!
Ted