From: Phlip <phlip2005@...>
Date: 2009-04-25T09:15:08+09:00
Subject: Re: Q: Architecting large Web service download app in Ruby

s.ross wrote:

> Assuming we can't improve the request/response rate of the Web service  
> calls or the granularity of the return data, is there a way to  
> implement some parallelism?

Run the entire downloader from a cron task.

Your question ass-umes that you must run out of one controller action. Wrong 
mindset!

And BTW 10,000 XML records should be trivial, so you might look for a bottleneck 
there. I would not read them all as a huge Ruby string and then convert them 
into a huge DOM model in memory. That would thrash. I would use what I think is 
called the "SAX" model of reading, where you register a callback for each node 
type, then let your reader stream them in...