From: Drew Olson <olsonas@...>
Date: 2006-11-30T03:56:52+09:00
Subject: Re: Splitting a CSV file into 40,000 line chunks

James -

I'll give FasterCSV a try when I get home from work and out from behind 
this proxy. Here's another question: in some cases I need to sort the 
file before splitting it (in this case sorting by the 4th cell in each 
row). However, the current file I'm trying to sort and split is around 
76 MB and ruby fails when trying to store the CSV as an array. The code 
and output are below. How else can I go about this?

Code:

require 'csv'

infile = ""

if ARGV[0] != nil
  infile = ARGV[0]
else
  puts "Please enter a file to split"
  exit
end

puts "loading file"

rows = CSV.read(infile)

puts "file loaded"

rows.sort!{|x,y| x[3] <=> y[3]}

counter = 1
header = ""
writer = CSV.open(infile.gsub(/\./,"_part_"+counter.to_s+"."),'w')

rows.each_index do |i|
    if(i != 0 && i%40000 == 0)
    writer.close
    counter+=1
    writer = CSV.open(infile.gsub(/\./,"_part_"+counter.to_s+"."),'w')
    writer << header
  end
  if (i == 0)
    header = row
  end
  writer << row
end

writer.close()

Output:

loading file
The exception unknown software exception (0xc00000fd) occured in the 
application at location.....etc

-- 
Posted via http://www.ruby-forum.com/.