From: James Edward Gray II <james@...>
Date: 2007-02-27T02:48:15+09:00
Subject: Re: Parsing CSV

On Feb 26, 2007, at 8:45 AM, Rafael George wrote:

> passvalues = []
> i = 0
> IO.foreach(fsource) do |line|
> cols = []
> cols=CSV::parse_line line.chomp
> sourceval = cols[scomp_args[0]] + " " + cols[scomp_args[1]]
>  IO.foreach(tdest) do |line|
>    tcols = []
>    tcols=CSV::parse_line line.chomp
>    testval = tcols[tcomp_args[0]] + " " + tcols[tcomp_args[1]]
>    if sourceval == testval
>      passvalues[i] = sourceval
>      i += 1
>   end
>  end
> end

The direct translation of this code to FasterCSV is:

passvalues = Array.new
FCSV.foreach(fsource) |s_row|
   source = s_row[scomp_args[0]..scomp_args[1]].join(" ")
   FCSV.foreach(tdest) |t_row|
     if source == t_row[scomp_args[0]..scomp_args[1]].join(" ")
       passvalues << source
     end
   end
end

If you can afford to read one of the files into memory because it's  
not too large, you can probably speed that up quite a bit:

require "set"

allowed = Set.new
FCSV.foreach(tdest) do |row|
   allowed.add(row[scomp_args[0]..scomp_args[1]].join(" "))
end

passvalues = FCSV.open(fsource) do |source|
   source.select do |row|
     allowed.include? row[scomp_args[0]..scomp_args[1]].join(" ")
   end
end

Hope that gives you some fresh ideas.

James Edward Gray II