From: James Gray <james@...>
Date: 2008-09-28T08:53:09+09:00
Subject: [ruby-core:18992] Re: CSV Upgrade Complete

On Sep 27, 2008, at 2:14 PM, Nobuyoshi Nakada wrote:

> Hi,

Hello.  Thanks for the feedback Nobu.

> At Mon, 22 Sep 2008 01:49:27 +0900,
> James Gray wrote in [ruby-core:18773]:
>> I finished my first pass at reworking the standard CSV library to  
>> have
>> an m17n aware parser and generator.  I committed the changes  
>> yesterday.
>
> /home/nobu/src/ruby/trunk-debug/src/test/csv/tc_encodings.rb:214:  
> warning: shadowing outer local variable - csv
> Loaded suite test/csv/ts_all
> Started
> .........................................................................................F 
> ............................................
> Finished in 0.621960 seconds.
>
>  1) Failure:
> test_inspect_is_us_ascii_encoded(TestCSVTable) [/home/nobu/src/ruby/ 
> trunk-debug/src/test/csv/tc_table.rb:404]:
> Expected "US-ASCII", not "UTF-8".
>
> 134 tests, 1784 assertions, 1 failures, 0 errors

Interesting, this is a recent change in Ruby:

$ ruby_dev -ve 'p "a String: #{123} #{:abc}".encoding'
ruby 1.9.0 (2008-09-19 revision 0) [i386-darwin9.4.0]
#<Encoding:US-ASCII>

$ ruby_dev -ve 'p "a String: #{123} #{:abc}".encoding'
ruby 1.9.0 (2008-09-27 revision 0) [i386-darwin9.5.0]
#<Encoding:UTF-8>

I guess Strings no longer default to US-ASCII when that's all they  
contain?

> Index: lib/csv.rb
> ===================================================================
> --- lib/csv.rb	(revision 19599)
> +++ lib/csv.rb	(working copy)
> @@ -1,4 +1,4 @@
> #!/usr/bin/env ruby -w
> -# encoding: UTF-8
> +# -*- encoding: us-ascii -*-
> # = csv.rb -- CSV Reading and Writing
> #

I would rather not switch my source encoding out of UTF-8, just in  
case I forget and use a multibyte character somewhere (maybe in an  
error message).  I'll transcode the problematic inspect() message  
though since I know it will only contain ASCII.

> Index: test/csv/tc_encodings.rb
> ===================================================================
> --- test/csv/tc_encodings.rb	(revision 19599)
> +++ test/csv/tc_encodings.rb	(working copy)
> @@ -198,6 +198,10 @@ class TestEncodings < Test::Unit::TestCa
>     each_encoding do |encoding|
>       # test generate_line with encoding hint
> -      csv = %w[abc d|ef].map { |f| f.encode(encoding) }.
> -                         to_csv(:col_sep => "|", :encoding =>  
> encoding.name)
> +      begin
> +        csv = %w[abc d|ef].map { |f| f.encode(encoding) }.
> +                           to_csv(:col_sep => "|", :encoding =>  
> encoding.name)
> +      rescue Encoding::ConverterNotFoundError
> +        next
> +      end
>       assert_equal(encoding, csv.encoding)

It looks like you had to do this because of a new bug in Ruby:

$ ruby_dev -e 'p "abc".encode("Big5"); p "".encode("Big5")'
"abc"
-e:1:in `encode': code converter not found (UTF-8 to Big5)  
(Encoding::ConverterNotFoundError)
	from -e:1:in `<main>'

> @@ -208,6 +212,6 @@ class TestEncodings < Test::Unit::TestCa
>       # writing to files
>       data = encode_ary([%w[abc d,ef], %w[123 456 ]], encoding)
> -      CSV.open(@temp_csv_path, "wb:#{encoding.name}") do |csv|
> -        data.each { |row| csv << row }
> +      CSV.open(@temp_csv_path, "wb:#{encoding.name}") do |c|
> +        data.each { |row| c << row }
>       end
>       assert_equal(data, CSV.read(@temp_csv_path, :encoding =>  
> encoding.name))

Good catch.  Thank you.

>
> @@ -222,6 +226,7 @@ class TestEncodings < Test::Unit::TestCa
>     parsed   = CSV.parse(ary_to_data(fields, options), options)
>     assert_equal(fields, parsed)
> -    assert( parsed.flatten.all? { |field| field.encoding ==  
> encoding },
> -            "Fields were transcoded." )
> +    parsed.flatten.each_with_index { |field, i|
> +      assert_equal(encoding, field.encoding, "Field\##{i+1} were  
> transcoded")
> +    }
>   end

Yeah, that's better.

James Edward Gray II