From: Mike Woodhouse <mikewoodhouse@...>
Date: 2008-04-02T21:30:28+09:00
Subject: Re: FasterCSV heavy loads?

On Apr 2, 1:26 pm, Mike Woodhouse <mikewoodho...@gmail.com> wrote:
> On Apr 2, 9:08 am, Michael Linfield <globyy3...@hotmail.com> wrote:
>
> > Recently I've attempted to push a huge csv into arrays via code that
> > looks along the lines of this:
>
> > csvFile = FasterCSV.read('data.csv', :headers => true)
>
> > array = []
>
> > csvFile.each do |row|
> > array << row['column_name']
> > end
>
> > The problem arises when the csv file is someodd 2 million lines or more.
>
> How many fields in a row? You're appending that many times (2 million
> or more) values to an array, which I suspect is where your performance
> problem lies.
>
> You could probably check by
>
> csvFile = FasterCSV.read('data.csv', :headers => true)
> count = 0
> csvFile.each do |row|
>
> end

Hmph. I must have hit some unknown "send" key combination...

I meant to say, before I interrupted myself:

csvFile = FasterCSV.read('data.csv', :headers => true)
count = 0
csvFile.each do |row|
  count += 1
end

...which replaces the array append with a lightweight operation. (I
don't know if Ruby is "smart" and likely to skip the iteration with an
empty block - probably not, but adding 1 shouldn't impose a heavy
load)

Mike