From: Hugh Sasse <hgs@...>
Date: 2005-10-28T23:15:34+09:00
Subject: Re: csv.rb a start on refactoring.

On Fri, 28 Oct 2005, Ara.T.Howard wrote:

> 
> wow - your mailer sent everything as attachments, including your message....
> strange.  maybe it's on my end...  so sorry for not quoting context here.

Oh, sorry about that.  I'll have to have a poke about in the config.
It looked ok... Oh, mailscanner claimed it had a Cyrus Mime Boundary
problem, and added some more Mime info to it I think.  I can't see
anything in Pine's config for overcoming that -- I'm using Pine 4.64
as you can probably tell, which is the latest.
> 
> anyhow - thanks for doing this.  fyi, i've used the following approach many
> times for loading huge csv files in an attempt to squeeze out speed - it
> works.  the approach is simple:
> 
>   - parse the first line __only__ using the built-in csv class, note the
>     number of columns as n_columns.
> 
>   - for each subsequent line
> 
>       fields = row.strip.split(%r/\s*,\s*/)
> 
>       if fields.size == n_columns
>         yield fields
>       else
>         fields = CSV::parse row
>         yield fields
>       end
> 

I have found for the CSV cases I have that split suffices.

> this apprach would need to be expanded slightly to deal with awful csv lines
> like
> 
>     foo, "bar
>   and more and more and more bar"

CSV (std and modified) choke on that in the default case.   I've not
figured out how to do the complex tests or I would have.
> 
> but it can still be done.  essentially the parser must remain optimistic at
> all times - assuming a simple split and stripping of all fields is sufficient.
> __only__ upon finding it not so should it degrade to the slow, but very
> complete, built-in csv parser.  tweaking my appoach to be buffer-of-lines
> based rather than lines based should do the trick.
> 
> food for thought.
> 
> regards.
> 
> -a

        Thank you,
        Hugh