From: Jeremy Woertink <jeremywoertink@...>
Date: 2010-12-02T03:16:51+09:00
Subject: Re: FasterCSV parsing issues

Nathaniel Smith wrote in post #965441:
> I've had similar issues recently, and they are due to character
> encodings. Something like Iconv will probably be necessary to convert
> the files to a standard encoding
>
> On Wed, Dec 1, 2010 at 11:52 AM, Jeremy Woertink

I've never actually used Iconv before, but I was just reading 
http://blog.grayproductions.net/articles/encoding_conversion_with_iconv 
and I did a test. I converted from ISO8859-1 to UTF8, and that actually 
changes the characters, so it changes the meaning of the words. Now, 
this is assuming that the CSV files I'm getting are all ISO8859-1 
encoded (which I think they are).

I tried a test to just tell FasterCSV to read it as 'ISO8859-1'using the 
first 3 lines of this CSV file:


Universal,ID,Kir,"Commonly, white wine with Cassis. Traditionally, the 
cocktail kir (also known as vin blanc cassis in French) is made with 
AligotÈ. Kir Royal is made with Champagne instead of AligotÈ."
Universal,GRAPE,MourvËdre / Monastrell / Mataro,"Grape: MourvËdre, 
MatarÛ, or Monastrell is variety of grape used to make both strong, dark 
red wines and rosÈs. It is grown in many regions around the world.
Universal,Tasting,Leafy,Specific aroma/taste descriptor: Having the 
smell or taste sensation of Leaves.


ruby-1.8.7-p302 > file = File.open(File.join(Rails.root, 'public', 
'sample.csv'))
 => #<File:/Users/jeremywoertink/Sites/winovations/public/sample.csv>
ruby-1.8.7-p302 > csv = FasterCSV.new(file, :encoding => 'ISO8859-1')
 => <#FasterCSV io_type:File 
io_path:"/Users/jeremywoertink/Sites/winovations/public/sample.csv" 
lineno:0 col_sep:"," row_sep:"\n" quote_char:"\"" encoding:"ISO8859-1">
ruby-1.8.7-p302 > csv.each { |row| puts row }
Universal
ID
Kir
Commonly, white wine with Cassis. Traditionally, the cocktail kir (also 
known as vin blanc cassis in French) is made with AligotÈ. Kir Royal is 
made with Champagne instead of AligotÈ.
FasterCSV::MalformedCSVError: Unclosed quoted field on line 2.
  from 
/Users/jeremywoertink/.rvm/gems/ruby-1.8.7-p302/gems/fastercsv-1.5.3/lib/faster_csv.rb:1663:in 
`shift'
  from 
/Users/jeremywoertink/.rvm/gems/ruby-1.8.7-p302/gems/fastercsv-1.5.3/lib/faster_csv.rb:1581:in 
`loop'
  from 
/Users/jeremywoertink/.rvm/gems/ruby-1.8.7-p302/gems/fastercsv-1.5.3/lib/faster_csv.rb:1581:in 
`shift'
  from 
/Users/jeremywoertink/.rvm/gems/ruby-1.8.7-p302/gems/fastercsv-1.5.3/lib/faster_csv.rb:1526:in 
`each'
  from (irb):28

I'm not seeing any unclosed quotes... Also, I thought that when you 
iterate through the returned csv file, it gives you rows, but this one 
seems to be giving my columns on the first row, then dies when it hits 
the second row.

-- 
Posted via http://www.ruby-forum.com/.