From: James Edward Gray II <james@...>
Date: 2011-01-14T05:11:40+09:00
Subject: Re: Reading a CSV file with UTF-16LE encoding

On Jan 13, 2011, at 1:58 PM, Daniel de Angelis Cordeiro wrote:

> I'm using ruby 1.9.2p0 and I'm trying to read a CSV file encoded in
> UTF-16LE using the following script:
> 
> # encoding: utf-8
> require 'csv'
> CSV.foreach("file_path", {:col_sep => ";", :encoding =>
> "UTF-16LE:UTF-8"}) { |row|
>  p row
> }
> 
> When I run this, I got the following exception:
> 
> /usr/lib/ruby/1.9.1/csv.rb:2020:in `=~': invalid byte sequence in
> UTF-8 (ArgumentError)
> 	from /usr/lib/ruby/1.9.1/csv.rb:2020:in `init_separators'
> 	from /usr/lib/ruby/1.9.1/csv.rb:1570:in `initialize'
> 	from /usr/lib/ruby/1.9.1/csv.rb:1335:in `new'
> 	from /usr/lib/ruby/1.9.1/csv.rb:1335:in `open'
> 	from /usr/lib/ruby/1.9.1/csv.rb:1201:in `foreach'
> 	from test.rb:3:in `<main>'
> 
> The csv module reads a sample from the file (using IO.read(),
> csv.rb:2309) and tries to match it against a Regexp of possible line
> endings. This sample have its encoding forced to the encoding I've
> chose (UTF-8), but the result is sample.valid_encoding? == false. When
> the regexp match takes place, the result is this exception I showed.
> 
> 
> Am I missing something here or this is a bug on csv module?

It does look like it's probably a bug.  I think it only affects the line ending guessing though, so set :row_sep manually to avoid it for now.  Sorry!

James Edward Gray II