From: Daniel de Angelis Cordeiro <dcordeiro@...>
Date: 2011-01-14T04:58:07+09:00
Subject: Reading a CSV file with UTF-16LE encoding

Hy all,


I'm using ruby 1.9.2p0 and I'm trying to read a CSV file encoded in
UTF-16LE using the following script:

# encoding: utf-8
require 'csv'
CSV.foreach("file_path", {:col_sep => ";", :encoding =>
"UTF-16LE:UTF-8"}) { |row|
  p row
}

When I run this, I got the following exception:

/usr/lib/ruby/1.9.1/csv.rb:2020:in `=~': invalid byte sequence in
UTF-8 (ArgumentError)
	from /usr/lib/ruby/1.9.1/csv.rb:2020:in `init_separators'
	from /usr/lib/ruby/1.9.1/csv.rb:1570:in `initialize'
	from /usr/lib/ruby/1.9.1/csv.rb:1335:in `new'
	from /usr/lib/ruby/1.9.1/csv.rb:1335:in `open'
	from /usr/lib/ruby/1.9.1/csv.rb:1201:in `foreach'
	from test.rb:3:in `<main>'

The csv module reads a sample from the file (using IO.read(),
csv.rb:2309) and tries to match it against a Regexp of possible line
endings. This sample have its encoding forced to the encoding I've
chose (UTF-8), but the result is sample.valid_encoding? == false. When
the regexp match takes place, the result is this exception I showed.


Am I missing something here or this is a bug on csv module?


Thanks in advance,
Daniel