From: Michael Linfield <globyy3000@...>
Date: 2008-04-03T02:54:27+09:00
Subject: Re: FasterCSV heavy loads?

Mike Woodhouse wrote:
> On Apr 2, 1:26 pm, Mike Woodhouse <mikewoodho...@gmail.com> wrote:
>> > array << row['column_name']
>> csvFile = FasterCSV.read('data.csv', :headers => true)
>> count = 0
>> csvFile.each do |row|
>>
>> end
> 
> Hmph. I must have hit some unknown "send" key combination...
> 
> I meant to say, before I interrupted myself:
> 
> csvFile = FasterCSV.read('data.csv', :headers => true)
> count = 0
> csvFile.each do |row|
>   count += 1
> end
> 
> ...which replaces the array append with a lightweight operation. (I
> don't know if Ruby is "smart" and likely to skip the iteration with an
> empty block - probably not, but adding 1 shouldn't impose a heavy
> load)
> 
> Mike

> James
>Let's just read it row by
>row, instead.
>
>   column = [ ]
>   FCSV.foreach('data.csv', :headers => true) do |row|
>     column << row['column_name']
>   end

Well firstly, I did count the rows already in the csv via

file = File.readlines('data.csv')
file.length

output is about  ~2,500,300

To answer your other question Mike the amount of columns is 3. All 
integers.

Thanks James for that snippet, though it might be more efficient it 
likely cuts the time in half (really nice) however, being that after 9 
hours I gave up, I don't know what half even is! The data from each 
column is being written into its own array. IE: column1Array = [] 
column2Array = [] ect.

Usually if the numbers were a defined length in each column I could just 
use regexp's to pull them out, however the numbers are mostly random. 
I'll give your snippet a shot James and let you know how the results 
turn out. Till then any additional thoughts are much appreciated.

Thanks,

Mac
-- 
Posted via http://www.ruby-forum.com/.