From: Francis Cianfrocca <garbagecat10@...>
Date: 2006-07-27T13:23:47+09:00
Subject: Re: For performance, write it in C

Ashley Moran wrote:
 > I think the total data size is about 1.5GB, but the individual files
> are smaller, the largest being a few hundred GB.  The most rows in a
> file is ~15,000,000 I think.  The server I run it on has 2GB RAM (an
> Athlon 3500+ running FreeBSD/amd64, so the hardware is not really an
> issue)... it can get all the way through without swapping (just!)

The problem isn't the raw size of the dataset. It's the number of 
objects you create and the amount of garbage that has to be cleaned up. 
If you're clever about how you write, you can help Ruby by not creating 
so much garbage. That can give a huge benefit.


> 
> The processing is pretty trivial, and mainly involves incrementing
> some ID columns so we can merge datasets together, adding a text
> column to the start of every row, and eliminating a few duplicates.

Eliminating the dupes is the only scary thing I've seen here. What's the 
absolute smallest piece of data that you need to look at in order to 
distinguish a dupe? (If it's the whole line, then the answer is 16 
bytes- the length of the MD5 hash ;-)) That's the critical working set. 
If you can't get the Ruby version fast enough, it's cheap and easy to 
sort through 15,000,000 of them in C. Then one pass through the sorted 
set finds your dupes. I've never found a consistently-fastest performer 
among Ruby's several different ways of storing sorted sets.

Make sure that your inner loop doesn't allocate any new variables, 
especially arrays- declare them outside your inner loop and re-use them 
with Array#clear.

> doesn't improve things, there's always the option of going dual-core
> and forking to do independent files.

Obviously I haven't seen your code or your data, but if the Ruby app is 
memory-bus-bound, then this approach may make your problem worse, not 
better.

Good luck. I recently got a Ruby program that aggregates several LDAP 
directory-pulls with about a million entries down from a few hours to a 
few seconds, without having to drop into C. It can be done, and it's 
kindof fun too.


-- 
Posted via http://www.ruby-forum.com/.