From: Monika Moser <monika.m.moser@...>
Date: 2009-05-13T17:49:18+09:00
Subject: Re: Performance issues with large files -- ruby vs. python :)

--000e0cd245ea4433600469c7493a
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

2008/11/27 sa 125 <s_ayalon@hotmail.com>

> Hi all -
>
> I'm new to ruby after working with python for a while. My work is
> performing data mining and doing some web dev for my company. We
> recently started looking at rails, and I wanted to see if it's worth
> migrating some of my code from python to ruby. So much for the intro.
>
> I've re-written a script that extracts data from a very large csv file
> (~8 million rows or so, almost 1Gb in size). It does so by iterating
> through the rows and building a tree-like hash (dict in python) in
> memory that will later be written to our DB. The structure is something
> like this:
>
> {date =>
>  { company =>
>     { product => [array of relevant info] } } }
>
> This way I only get the data I need, sort it on the fly, and speed up
> the process. The idea is to propagate down the keys if some field has
> duplicate values -- I hope that makes some sense.. Anyway, I copied my
> python code, and basically translated it to ruby.
>
> This is where it got interesting -- after solving all the quirks and
> getting it to run, it appeared to be super slow compared to python. I
> mean nearly an 30 min in ruby vs under 2 minutes in python code. It's
> worth mentioning that I used the psyco module in python, and that I did
> my testing on win-xp.
>
> Since I'm a newbie and **really** don't want to spark a python/ruby
> talk-back war (I actually like ruby a lot from what I've seen so far), I
> was just wondering if there's something I might have missed, like a
> psyco equivalent module for ruby or something else to narrow the gap.
>
> I'd appreciate feedback - thanks!
> --
> Posted via http://www.ruby-forum.com/.
>
>
Hi,
I do have a very similar problem: large data files and csv-import with
fastercsv which is much slower than an implementation in c++.
I was wondering if you have some interesting insights about that meanwhile
which you would like to share :).
Thanks,
Monika

--000e0cd245ea4433600469c7493a--