From: Chad Perrin <perrin@...>
Date: 2006-07-27T05:14:36+09:00
Subject: Re: For performance, write it in C

On Thu, Jul 27, 2006 at 04:59:13AM +0900, Francis Cianfrocca wrote:
> Ashley Moran wrote:
> > performance lately.  I just had to write a script to process about
> > 1-1.5GB of CSV data  (No major calculations, but it involves about 20
> > million rows, or something in that region).
> 
> I've had tremendous results optimizing Ruby programs that process huge 
> piles of text. There is a range of "tricks" you can use to keep Ruby 
> from wasting memory, which is its real downfall. If it's possible, given 
> your application, to process your CSV text in such a way that you don't 
> store any transformations of the whole set in memory at once, you'll go 
> orders of magnitude faster. You can even try to break your computation 
> up into multiple stages, and stream the intermediate results out to 
> temporary files. As ugly as that sounds, it will be far faster.

One of these days, I'll actually know enough Ruby to be sure of what
language constructs work for what purposes in terms of performance.  I
rather suspect there are prettier AND better-performing options than
using temporary files to store data during computation, however.

-- 
CCD CopyWrite Chad Perrin [ http://ccd.apotheon.org ]
Ben Franklin: "As we enjoy great Advantages from the Inventions of
others we should be glad of an Opportunity to serve others by any
Invention of ours, and this we should do freely and generously."