From: Paul Lutus <nospam@...>
Date: 2006-11-30T02:29:18+09:00
Subject: Re: Splitting a CSV file into 40,000 line chunks

Will Jessop wrote:

> Paul Lutus wrote:
>>> CSV.open(infile, 'r') do |row|
>> 
>> Why are you using CSV for this? You aren't parsing the lines into fields,
>> so the fact that they contain CSV content has no bearing on the present
>> task. Your goal is to split the input file into groups of lines delimited
>> by linefeeds, not fields delimited by commas.
>> 
>> Why not simply read lines from the input file and write them to a series
>> of output files, until the input file is exhausted?
> 
> Because CSV understands csv data with embedded newlines:

A plain-text CSV file uses linefeeds as record delimiters. A program that
uses "readline" or "gets" splits the records just as a sane CSV parser
would. And IMHO a CSV file should never, ever have linefeeds embedded in
fields.

> [will@localhost ~]$ cat csvfile.csv
> some, field,"new
> line"
> other,field,here

Ah, yes, another terrific "improvement" in database protocols. Sometimes I
wonder what these standards committees are smoking.

-- 
Paul Lutus
http://www.arachnoid.com