Re: csv.rb a start on refactoring.

On Sun, 30 Oct 2005, James Edward Gray II wrote:

> On Oct 28, 2005, at 10:06 PM, James Edward Gray II wrote:
>
>> Here's the hyper-optimized version with an example of how he uses it 
>> (translated from Perl to Ruby by me):
>
> This certainly seems promising, at least as a starting point:
>
> Neo:~/Desktop$ cat bm_csv.rb
> #!/usr/local/bin/ruby -w
>
> require "csv"
> require "benchmark"
>
> def parse_csv( line )
>  results = Array.new
>  line.scan(/\G(?:^|,)(?:"((?>[^"]*)(?>""[^"]*)*)"|([^",]*))/) do
>    if $1.nil?
>      results << $2
>    else
>      results << $1.gsub('""', '"')
>    end
>  end
>  results
> end
>
> DATA  = %Q{Ten Thousand,10000, 2710 ,,"10,000","It's ""10 Grand"", baby",10K}
> TESTS = 50000
>
> Benchmark.bm do |timings|
>  timings.report("CSV") { TESTS.times { CSV.parse_line(DATA) } }
>  timings.report("Regexp") { TESTS.times { parse_csv(DATA) } }
> end
> Neo:~/Desktop$ ruby bm_csv.rb
>      user     system      total        real
> CSV 18.570000   0.060000  18.630000 ( 18.675331)
> Regexp  2.700000   0.010000   2.710000 (  2.726666)


it __is__ promising!

it may or may not be tricky to get these failing cases working though:

harp:~ > ruby a.rb
==========================================
CSV2[7] => FAILED (RuntimeError)
==========================================
input:
"a,\"\"\"\nb\n\"\"\",\nc"
csv:
["a", "\"\nb\n\"", "\nc"]
expected:
["a", "\"\nb\n\"", nil]
==========================================

==========================================
CSV2[8] => FAILED (RuntimeError)
==========================================
input:
"a,,,"
csv:
["a", "", "", ""]
expected:
["a", nil, nil, nil]
==========================================

==========================================
CSV2[9] => FAILED (RuntimeError)
==========================================
input:
","
csv:
[""]
expected:
[nil, nil]
==========================================

==========================================
CSV2[13] => FAILED (RuntimeError)
==========================================
input:
",\"\""
csv:
[""]
expected:
[nil, ""]
==========================================

==========================================
CSV2[14] => FAILED (RuntimeError)
==========================================
input:
",\"\r\""
csv:
[""]
expected:
[nil, "\r"]
==========================================

==========================================
CSV2[16] => FAILED (RuntimeError)
==========================================
input:
"\"\r\n,\","
csv:
["\r\n,", ""]
expected:
["\r\n,", nil]
==========================================


mode of the erros stem from problems dealing with leading/trailing commas.  i
think it could be fixed.

code inlined at the end of this message.  btw - don't get me wrong - i'd
__love__ to see csv be faster, i just happen to load tons of mega escaped
doccuments so it has to handle anything we can throw at it too.

cheers.

-a
-- 
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: [phone]
| anything that contradicts experience and logic should be abandoned.
| -- h.h. the 14th dalai lama
===============================================================================


require 'pp'
require 'csv'

module CSV2
   def self::parse_line line
     csv = Array.new
     line.scan(/\G(?:^|,)(?:"((?>[^"]*)(?>""[^"]*)*)"|([^",]*))/){ csv << ($1 ? $1.gsub('""', '"') : $2)}
     csv
   end
end

tests = [
   [
     %( a,b ),
     ["a", "b"]
   ],
   [
     %( a,"""b""" ),
     ["a", "\"b\""]
   ],
   [
     %( a,"""b" ),
     ["a", "\"b"]
   ],
   [
     %( a,"b""" ),
     ["a", "b\""]
   ],
   [
     %( a,"
b""" ),
     ["a", "\nb\""]
   ],
   [
     %( a,"""
b" ),
     ["a", "\"\nb"]
   ],
   [
     %( a,"""
b
""" ),
     ["a", "\"\nb\n\""]
   ],
   [
     %( a,"""
b
""",
c ),
     ["a", "\"\nb\n\"", nil]
   ],
   [
     %( a,,, ),
     ["a", nil, nil, nil]
   ],
   [
     %( , ),
     [nil, nil]
   ],
   [
     %( "","" ),
     ["", ""]
   ],
   [
     %( """" ),
     ["\""]
   ],
   [
     %( """","" ),
     ["\"",""]
   ],
   [
     %( ,"" ),
     [nil,""]
   ],
   [
     %( \r,"\r" ),
     [nil,"\r"]
   ],
   [
     %( "\r\n," ),
     ["\r\n,"]
   ],
   [
     %( "\r\n,", ),
     ["\r\n,", nil]
   ],
]

impls = CSV, CSV2

tests.each_with_index do |test, idx|
   input, expected = test
   csv = []
   impls.each do |impl|
     begin
       csv = impl::parse_line input.strip
       raise "FAILED" unless csv == expected
     rescue => e
       puts "=" * 42
       puts "#{ impl }[#{ idx }] => #{ e.message } (#{ e.class })"
       puts "=" * 42
       puts "input:\n#{ PP::pp input.strip, '' }"
       puts "csv:\n#{ PP::pp csv, '' }"
       puts "expected:\n#{ PP::pp expected, '' }"
       puts "=" * 42
       puts
     end
   end
end

__END__

http://www.ietf.org/rfc/rfc4180.txt