From: James Gray <james@...>
Date: 2009-05-15T00:05:36+09:00
Subject: Re: Performance issues with large files -- ruby vs. python :)

On May 13, 2009, at 3:49 AM, Monika Moser wrote:

> 2008/11/27 sa 125 <s_ayalon@hotmail.com>
>
>> Hi all -
>>
>> I'm new to ruby after working with python for a while. My work is
>> performing data mining and doing some web dev for my company. We
>> recently started looking at rails, and I wanted to see if it's worth
>> migrating some of my code from python to ruby. So much for the intro.
>>
>> I've re-written a script that extracts data from a very large csv  
>> file
>> (~8 million rows or so, almost 1Gb in size). It does so by iterating
>> through the rows and building a tree-like hash (dict in python) in
>> memory that will later be written to our DB. The structure is  
>> something
>> like this:
>>
>> {date =>
>> { company =>
>>    { product => [array of relevant info] } } }
>>
>> This way I only get the data I need, sort it on the fly, and speed up
>> the process. The idea is to propagate down the keys if some field has
>> duplicate values -- I hope that makes some sense.. Anyway, I copied  
>> my
>> python code, and basically translated it to ruby.
>>
>> This is where it got interesting -- after solving all the quirks and
>> getting it to run, it appeared to be super slow compared to python. I
>> mean nearly an 30 min in ruby vs under 2 minutes in python code. It's
>> worth mentioning that I used the psyco module in python, and that I  
>> did
>> my testing on win-xp.
>>
>> Since I'm a newbie and **really** don't want to spark a python/ruby
>> talk-back war (I actually like ruby a lot from what I've seen so  
>> far), I
>> was just wondering if there's something I might have missed, like a
>> psyco equivalent module for ruby or something else to narrow the gap.
>>
>> I'd appreciate feedback - thanks!
>> --
>> Posted via http://www.ruby-forum.com/.
>>
>>
> Hi,
> I do have a very similar problem: large data files and csv-import with
> fastercsv which is much slower than an implementation in c++.
> I was wondering if you have some interesting insights about that  
> meanwhile
> which you would like to share :).

Well, I take it to be pretty obvious that FasterCSV (written in Ruby),  
isn't going to be as fast as a C/C++ parser.  I believe Ruby has a C  
based parse though, if you want to go that way:

http://rubyforge.org/projects/simplecsv

You might also try FasterCSV's latest code, not yet released but in  
version control.  It has a new parser that can be faster for some  
things.

James Edward Gray II