From: Hiroyuki DOI <hiroyuki.doi495@...>
Date: 2013-08-04T16:45:20+09:00
Subject: [ruby-dev:47581] 特定のYAMLデータのloadでエラーになる場合がある

はじめまして、土居と申します。
RubyでYAMLの処理をする際に、不具合と思われる挙動を見つけたのでご報告します。
もし報告先や内容が適切でなければ、ご指摘ください。

【概要】
Ruby 1.9.3-p448でdumpしたYAMLデータをRuby 2.0.0-p247でloadすると
エラーになる場合がある。

【環境】
Ruby 1.9.3-p448 (RubyInstaller版)
Ruby 2.0.0-p247 (RubyInstaller版)
OS: WindowsXP Pro SP3

【手順】
(1) 以下のようにdump.rbとload.rbを用意する。

[dump.rb]
# -*- coding: UTF-8 -*-
require 'yaml'
File.open("test.yml", "w") { |file|
  YAML.dump("ねこ\n1", file)
}

[load.rb]
# -*- coding: UTF-8 -*-
require 'yaml'
File.open("test.yml", "r") { |file|
  print YAML.load(file)
}

(2) Ruby 1.9.3-p448でdump.rbを実行する。test.ymlができる。

D:\Work\Ruby\bug_Psych>ruby -v
ruby 1.9.3p448 (2013-06-27) [i386-mingw32]

D:\Work\Ruby\bug_Psych>ruby dump.rb

[test.yml]
--- ! 'ねこ

  1'

(3) Ruby 2.0.0-p247でload.rbを実行すると、エラーになる。

D:\Work\Ruby\bug_Psych>ruby -v
ruby 2.0.0p247 (2013-06-27) [i386-mingw32]

D:\Work\Ruby\bug_Psych>ruby load.rb
C:/Ruby200/lib/ruby/2.0.0/psych/scalar_scanner.rb:114:in `Integer': invalid value for Integer(): "\xE3\x81\xAD\xE3\x81\x93\n1" (ArgumentError)
        from C:/Ruby200/lib/ruby/2.0.0/psych/scalar_scanner.rb:114:in `parse_int'
        from C:/Ruby200/lib/ruby/2.0.0/psych/scalar_scanner.rb:102:in `tokenize'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:98:in `deserialize'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:104:in `visit_Psych_Nodes_Scalar'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/visitor.rb:15:in `visit'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/visitor.rb:5:in `accept'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:20:in `accept'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:240:in `visit_Psych_Nodes_Document'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/visitor.rb:15:in `visit'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/visitor.rb:5:in `accept'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:20:in `accept'
        from C:/Ruby200/lib/ruby/2.0.0/psych/nodes/node.rb:35:in `to_ruby'
        from C:/Ruby200/lib/ruby/2.0.0/psych.rb:130:in `load'
        from load.rb:5:in `block in <main>'
        from load.rb:4:in `open'
        from load.rb:4:in `<main>'

(4) dump.rb中のYAML.dumpに渡す文字列によって、以下のように結果が変わる。

"ねこ\n1.0" → エラー
D:\Work\Ruby\bug_Psych>ruby load.rb
C:/Ruby200/lib/ruby/2.0.0/psych/scalar_scanner.rb:99:in `Float': invalid value for Float(): "\xE3\x81\xAD\xE3\x81\x93\n1.0" (ArgumentError)
        from C:/Ruby200/lib/ruby/2.0.0/psych/scalar_scanner.rb:99:in `tokenize'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:98:in `deserialize'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:104:in `visit_Psych_Nodes_Scalar'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/visitor.rb:15:in `visit'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/visitor.rb:5:in `accept'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:20:in `accept'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:240:in `visit_Psych_Nodes_Document'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/visitor.rb:15:in `visit'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/visitor.rb:5:in `accept'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:20:in `accept'
        from C:/Ruby200/lib/ruby/2.0.0/psych/nodes/node.rb:35:in `to_ruby'
        from C:/Ruby200/lib/ruby/2.0.0/psych.rb:130:in `load'
        from load.rb:4:in `block in <main>'
        from load.rb:3:in `open'
        from load.rb:3:in `<main>'

"ねこa\n1" → エラー
D:\Work\Ruby\bug_Psych>ruby load.rb
C:/Ruby200/lib/ruby/2.0.0/psych/scalar_scanner.rb:114:in `Integer': invalid value for Integer(): "\xE3\x81\xAD\xE3\x81\x93a\n1" (ArgumentError)
        from C:/Ruby200/lib/ruby/2.0.0/psych/scalar_scanner.rb:114:in `parse_int'
        from C:/Ruby200/lib/ruby/2.0.0/psych/scalar_scanner.rb:102:in `tokenize'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:98:in `deserialize'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:104:in `visit_Psych_Nodes_Scalar'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/visitor.rb:15:in `visit'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/visitor.rb:5:in `accept'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:20:in `accept'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:240:in `visit_Psych_Nodes_Document'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/visitor.rb:15:in `visit'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/visitor.rb:5:in `accept'
        from C:/Ruby200/lib/ruby/2.0.0/psych/visitors/to_ruby.rb:20:in `accept'
        from C:/Ruby200/lib/ruby/2.0.0/psych/nodes/node.rb:35:in `to_ruby'
        from C:/Ruby200/lib/ruby/2.0.0/psych.rb:130:in `load'
        from load.rb:4:in `block in <main>'
        from load.rb:3:in `open'
        from load.rb:3:in `<main>'

"aねこ\n1" → 正常終了
"ね\n1" → 正常終了
"ab\n1" → 正常終了
"ねこ 1" → 正常終了
"ねこ\na1" → 正常終了
"ねこ\n1a" → 正常終了

(5) 元のdump.rbをRuby 1.9.3-p448で実行し、load.rbも同じ環境で実行した場合、
    正常終了する。

(6) 元のdump.rbをRuby 2.0.0-p247で実行した場合はtest.ymlが以下のように
    なり、これはRuby 1.9.3-p448でもRuby 2.0.0-p247でも正常にloadできる。

[test.yml]
--- |-
  ねこ
  1

(7) 元のdump.rbをRuby 1.9.3-p448で実行し、その環境にpsych 2.0.0をインストール
    した後、以下のように書き換えたload.rbを実行するとエラーになる。

[load.rb]
# -*- coding: UTF-8 -*-
gem 'psych', '=2.0.0'  # ←追加
require 'yaml'
File.open("test.yml", "r") { |file|
  print YAML.load(file)
}

D:\Work\Ruby\bug_Psych>ruby load.rb
C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/scalar_scanner.rb:117:in `Integer': invalid value for Integer(): "\xE3\x81\xAD\xE3\x81\x93\n1" (ArgumentError)
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/scalar_scanner.rb:117:in `parse_int'
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/scalar_scanner.rb:105:in `tokenize'
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/visitors/to_ruby.rb:114:in `deserialize'
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/visitors/to_ruby.rb:120:in `visit_Psych_Nodes_Scalar'
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/visitors/visitor.rb:15:in `visit'
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/visitors/visitor.rb:5:in `accept'
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/visitors/to_ruby.rb:31:in `accept'
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/visitors/to_ruby.rb:274:in `visit_Psych_Nodes_Document'
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/visitors/visitor.rb:15:in `visit'
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/visitors/visitor.rb:5:in `accept'
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/visitors/to_ruby.rb:31:in `accept'
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych/nodes/node.rb:37:in `to_ruby'
        from C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-2.0.0/lib/psych.rb:245:in `load'
        from load.rb:5:in `block in <main>'
        from load.rb:4:in `open'
        from load.rb:4:in `<main>'

(8) psychの最新版(2.0.0)の1つ前のバージョン(1.3.4)でどうなるか試そうとしたが、
    Ruby 1.9.3-p448環境、Ruby 2.0.0-p247環境ともにインストールに失敗したため、
    試せなかった。

(Ruby 1.9.3-p448環境でのエラー)
D:\Work\Ruby\bug_Psych>gem install psych -v "1.3.4"
Fetching: psych-1.3.4.gem (100%)
Temporarily enhancing PATH to include DevKit...
Building native extensions.  This could take a while...
ERROR:  Error installing psych:
        ERROR: Failed to build gem native extension.

    C:/Ruby193/bin/ruby.exe extconf.rb
checking for yaml.h... no
yaml.h is missing. Try 'port install libyaml +universal' or 'yum install libyaml-devel'
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=C:/Ruby193/bin/ruby
        --with-libyaml-dir
        --without-libyaml-dir
        --with-libyaml-include
        --without-libyaml-include=${libyaml-dir}/include
        --with-libyaml-lib
        --without-libyaml-lib=${libyaml-dir}/lib


Gem files will remain installed in C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-1.3.4 for inspection.
Results logged to C:/Ruby193/lib/ruby/gems/1.9.1/gems/psych-1.3.4/ext/psych/gem_make.out

(Ruby 2.0.0-p247環境でのエラー)
D:\Work\Ruby\bug_Psych>gem install psych -v "1.3.4"
Temporarily enhancing PATH to include DevKit...
Building native extensions.  This could take a while...
ERROR:  Error installing psych:
        ERROR: Failed to build gem native extension.

    C:/Ruby200/bin/ruby.exe extconf.rb
checking for yaml.h... no
yaml.h is missing. Try 'port install libyaml +universal' or 'yum install libyaml-devel'
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=C:/Ruby200/bin/ruby
        --with-libyaml-dir
        --without-libyaml-dir
        --with-libyaml-include
        --without-libyaml-include=${libyaml-dir}/include
        --with-libyaml-lib
        --without-libyaml-lib=${libyaml-dir}/


Gem files will remain installed in C:/Ruby200/lib/ruby/gems/2.0.0/gems/psych-1.3.4 for inspection.
Results logged to C:/Ruby200/lib/ruby/gems/2.0.0/gems/psych-1.3.4/ext/psych/gem_make.out

【備考】
Ruby 1.9.3-p448環境でもpsych 2.0.0を使うようにしてYAML.loadするとエラーに
なるため、psych 2.0.0に問題があるのではないかと考えました。
しかし、Ruby 2.0.0-p247環境で古いpsychを使った場合どうなるかを試せなかったので、
Ruby自体のバージョンに依存しないのかどうかは分かりませんでした。


以上です。

-- 
---------------------------------------
  土居 弘幸
  hiroyuki.doi495@gmail.com