Re: csv.rb a start on refactoring.

wow - your mailer sent everything as attachments, including your message....
strange.  maybe it's on my end...  so sorry for not quoting context here.

anyhow - thanks for doing this.  fyi, i've used the following approach many
times for loading huge csv files in an attempt to squeeze out speed - it
works.  the approach is simple:

   - parse the first line __only__ using the built-in csv class, note the
     number of columns as n_columns.

   - for each subsequent line

       fields = row.strip.split(%r/\s*,\s*/)

       if fields.size == n_columns
         yield fields
       else
         fields = CSV::parse row
         yield fields
       end

this apprach would need to be expanded slightly to deal with awful csv lines
like

     foo, "bar
   and more 
and more and more bar"

but it can still be done.  essentially the parser must remain optimistic at
all times - assuming a simple split and stripping of all fields is sufficient.
__only__ upon finding it not so should it degrade to the slow, but very
complete, built-in csv parser.  tweaking my appoach to be buffer-of-lines
based rather than lines based should do the trick.

food for thought.

regards.

-a
-- 
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: [phone]
| anything that contradicts experience and logic should be abandoned.
| -- h.h. the 14th dalai lama
===============================================================================
For a database application I found using CSV to be rather slow.
I have made an attempt to speed it up, which, frankly has not been
very successuful.  However I think I have improved the code a
little.

I've added an error for if the state machine should get into an
unknown state, rather than assuming that "it must be in the other
one" after several tests.  This slows things down, but is probably
safer, given how hard state machines can be to debug.

I've changed if block to if block_given?  -- more idiomatic,
possibly faster??

I've tried to simplify 
  if something.is_a?(Klass)
    something.method
  end
to 
  if something.respond_to?(:method)
    something.method
  end

I think its faster, and its more duck-typing style.

I've tried to handle state transitions with a hash, as it is
probably clearer than if...elsif...else...end.  Likely to be faster,
too.

I've changed 
  if expr
    singlestatement
  end

to 
   singlestatement if expr

to shorten the code.  Don't know how that impacts speed.

I've changed if...elsif...else... to case statements where I can.  I
think that ought to be faster, but it is clearer code, I think.

There's a while statement with an if immediately inside it.  Since
thre is no else for that if, and the loop condition doesn't change
if the if-condition is false, then the condition must always apply.
For, if it did not apply, the while loop would never exit.  I have
commented out the condition and corresponding end.

idx_is_eos? is private, and thus not really part of the external
interface.  It takes a parameter that it doesn't use. I have removed
that and made it parameterless.  


I have tested the code and it still works.  I could not find
exhaustive test cases for CSV (looked in Rubycon and on the web) so
hope I haven't broken anything.  It does seem to be a tiny bit
faster, for the given case.  My tests are attached.  The test
program takes the file to test (csv or nova_csv) as an argument, to
allow comparison.  The name?  nova_csv because new_csv was already
being used for doc patches.

        HTH
        Hugh