From: "jonforums (Jon Forums)" <redmine@...>
Date: 2012-04-07T01:10:19+09:00
Subject: [ruby-core:44163] [ruby-trunk - Bug #6266][Open] encoding related exception with recent integrated psych


Issue #6266 has been reported by jonforums (Jon Forums).

----------------------------------------
Bug #6266: encoding related exception with recent integrated psych
https://bugs.ruby-lang.org/issues/6266

Author: jonforums (Jon Forums)
Status: Open
Priority: Normal
Assignee: tenderlovemaking (Aaron Patterson)
Category: ext
Target version: 1.9.3
ruby -v: ruby 1.9.3p176 (2012-04-04 revision 35238) [i386-mingw32]


Newer versions of Psych integrated into both ruby_1_9_3 and trunk create exceptions if the yaml file isn't perceived to be correctly encoded.

While I discovered this in my code that uses the 'logging' gem on Win7 32bit, this test snippet shows the same failure on both ruby_1_9_3@r35238 and trunk@r35229:

  # hello.yml uses LF-only line endings
  C:\temp>type hello.yml
  :greeting: hello yaml

  # yaml_test.rb uses LF-only
  C:\temp>type yaml_test.rb
  require 'yaml'
  puts "YAML engine: #{YAML::ENGINE.yamler} #{'v'+Psych::VERSION if defined?(Psych)}"

  # fix: 'rb'
  # fix: 'r:utf-8'
  # fix: Encoding.default_external = Encoding::UTF_8 if defined?(Encoding)
  # fix: chcp 65001 in cmd.exe before running script
  # fix: run script with ruby -Eutf-8
  File.open('hello.yml', 'r') do |f|
    puts YAML.load_stream(f)
  end

  # IBM437 codepage is a typical default for the cmd.exe shell
  C:\temp>chcp
  Active code page: 437

  C:\temp>ruby -v yaml_test.rb
  ruby 1.9.3p176 (2012-04-04 revision 35238) [i386-mingw32]
  YAML engine: psych v1.3.1
  C:/ruby193/lib/ruby/1.9.1/psych.rb:206:in `parse': YAML file must be UTF-8, UTF-16LE, or UTF-16BE, not IBM437 (ArgumentError)
        from C:/ruby193/lib/ruby/1.9.1/psych.rb:206:in `parse_stream'
        from C:/ruby193/lib/ruby/1.9.1/psych.rb:289:in `load_stream'
        from yaml_test.rb:10:in `block in <main>'
        from yaml_test.rb:9:in `open'
        from yaml_test.rb:9:in `<main>'

  # older versions of integrated psych do not cause exceptions
  C:\temp>chcp
  Active code page: 437

  C:\temp>ruby -v yaml_test.rb
  tcs-ruby 1.9.3p134 (2012-02-19, TCS patched 2012-02-25) [i386-mingw32]
  YAML engine: psych v1.2.2
  {:greeting=>"hello yaml"}

I've not tried on Arch with bash.

From the `test_yaml.rb` comments you can see some of the workarounds. None of these are good options because they require changes to either a users code/yaml, gem code that uses the psych library, or the ruby invocation command line.

For both trunk and ruby_1_9_3, I'd prefer psych implement a fallback strategy that Just Works combined with an informative warning rather than a hard exception.


-- 
http://bugs.ruby-lang.org/