From: Martin Duerst Date: 2008-12-25T15:02:45+09:00 Subject: [ruby-core:20862] Re: [ruby-dev:37591] [BUG:trunk] [m17n] TestCSVFeatures fails because of r20905 [Redirected to ruby-core so that James can also read this.] Hello James, This is an error report from Yugui about a csv test failing on a Mac. The reason for the failure is line 498 in lib/csv.rb, in method CSV#inspect. This line reads: str.map { |s| s.encode("ASCII-8BIT") }.join The reason for the failure is that currently, filenames on a Mac are labeled as being in an "encoding" of UTF8-MAC. The label UTF8-MAC is used to mark the assumption that this string is in a character normalization form particular to the Mac (mostly NFD, but not for Korean, and not for CJKV compatibility ideographs, as far as I understand). There is in general no knowledge about character normalization with respect to strings labeled UTF-8 (and even for UTF8-MAC, there is no guarantee about character normalization at all). In my personal view, the value of UTF8-MAC is questionable at least at the current point in time where we do not handle character normalization in any particular way. But for the current bug, that's actually a side issue. We might be able to fix this by introducing a (dummy) conversion from UTF8-MAC to UTF-8, but that won't actually fix the real problem. The real problem is that the line above ignores that conversion to ASCII-8BIT only works for US-ASCII characters, but not for all the other characters that might appear e.g. in a filename. The easiest fix for this, which is probably what was intended, is to change the above line to str.map { |s| s.force_encoding("ASCII-8BIT") }.join A slightly more "user-friendly" fix is to change this to something like: begin str.join rescue str.map { |s| s.force_encoding("ASCII-8BIT") }.join end This will only do a force_encoding if the encodings can't be joined as is. [The code above hasn't been tested; I don't have access to a Mac.] Hope this helps. Regards, Martin. At 12:27 08/12/25, Yugui (Yuki Sonoda) wrote: >Yuguiです。 > >r20905で、csvのテストが失敗するようになりました。pathのエンコーディング >がUTF8-MACに変わったため、CSV#inspect内でString#encodeに失敗しているよう >です。 >これはどう扱ったらよいものでしょうか。 > >% make test-all TESTS=csv >./miniruby -I../../lib -I.ext/common -I./- -r../../ext/purelib.rb >../../runruby.rb --extout=.ext -- "../../test/runner.rb" csv >nil >Loaded suite ../../test/runner >Started >..........E........................................................................................................................... >Finished in 0.768623 seconds. > > 1) Error: >test_inspect_is_smart_about_io_types(TestCSVFeatures): >Encoding::ConverterNotFoundError: code converter not found (UTF8-MAC to >ASCII-8BIT) > /Users/yugui/src/ruby/mri/test/csv/test_features.rb:233:in `block in >test_inspect_is_smart_about_io_types' > /Users/yugui/src/ruby/mri/test/csv/test_features.rb:233:in >`test_inspect_is_smart_about_io_types' > >134 tests, 1886 assertions, 0 failures, 1 errors, 0 skips >make: *** [test-all] Error 1 > >-- >Yugui >http://yugui.jp >私は私をDumpする #-#-# Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University #-#-# http://www.sw.it.aoyama.ac.jp mailto:duerst@it.aoyama.ac.jp