From: Mike Woodhouse <mikewoodhouse@...>
Date: 2008-04-02T21:30:08+09:00
Subject: Re: FasterCSV heavy loads?

On Apr 2, 9:08 am, Michael Linfield <globyy3...@hotmail.com> wrote:
> Recently I've attempted to push a huge csv into arrays via code that
> looks along the lines of this:
>
> csvFile = FasterCSV.read('data.csv', :headers => true)
>
> array = []
>
> csvFile.each do |row|
> array << row['column_name']
> end
>
> The problem arises when the csv file is someodd 2 million lines or more.

How many fields in a row? You're appending that many times (2 million
or more) values to an array, which I suspect is where your performance
problem lies.

You could probably check by

csvFile = FasterCSV.read('data.csv', :headers => true)
count = 0
csvFile.each do |row|

end