From: James Edward Gray II <james@...>
Date: 2010-12-02T04:04:48+09:00
Subject: Re: FasterCSV parsing issues

On Dec 1, 2010, at 12:16 PM, Jeremy Woertink wrote:

> I've never actually used Iconv before, but I was just reading 
> http://blog.grayproductions.net/articles/encoding_conversion_with_iconv 
> and I did a test. I converted from ISO8859-1 to UTF8, and that actually 
> changes the characters, so it changes the meaning of the words. Now, 
> this is assuming that the CSV files I'm getting are all ISO8859-1 
> encoded (which I think they are).

You probably want to hit the files with some encoding guessing script to be sure.

> I tried a test to just tell FasterCSV to read it as 'ISO8859-1'using the 
> first 3 lines of this CSV file:

> ruby-1.8.7-p302 >

On Ruby 1.8.7, FasterCSV supports only four encodings (the same four Ruby does) and Latin-1 (ISO-8859-1) isn't one of them.  You need to transcode the data to UTF-8 on the way in or use the standard CSV library in Ruby 1.9 (which can parse Latin-1 directly).

James Edward Gray II