From: Yui NARUSE Date: 2012-01-15T08:46:25+09:00 Subject: [ruby-core:42130] [ruby-trunk - Bug #5888] JSON unittest fails Issue #5888 has been updated by Yui NARUSE. Bohuslav Kabrda wrote: > After few hours of debugging, it seems that the flaw may actually be in the json_string_unescape function (in the same file, deeper in the callstack). > When compiled with -O0, it shows nothing suspicious, but with -O2, it shows some possible problems: > > 1344 unescape_len = convert_UTF32_to_UTF8(buf, ch); > convert_UTF32_to_UTF8 (ch=0, buf=0x7fffffffdd00 "\270", ) at parser.c:46 > > -- The incomplete sequence is the thing that is weird. The "ch" variable is created using some byte shifting in the json_string_unescape function, which may be the cause (if the optimization does something with the byte shifting). Incomplete sequence in buf is not the root of this issue because its content and the length are explicitly specified in convert_UTF32_to_UTF8(). But yes, the bug is here. The root of this issue is optimization for convert_UTF32_to_UTF8(). With -O2 convert_UTF32_to_UTF8() is inlined to json_string_unescape. If ch < 0x7F, it runs as 0x00000008034017a0 <+1120>: cmp $0x7f,%rsi # if (ch <= 0x7F) { 0x00000008034017a4 <+1124>: jbe 0x80340180c 0x000000080340180c <+1228>: mov $0x1,%edx # int len = 1; 0x0000000803401811 <+1233>: jmp 0x8034017b4 0x00000008034017b4 <+1140>: lea 0x20(%rsp),%rsi 0x00000008034017b9 <+1145>: jmpq 0x8034016c0 # return So it doesn't change buf. I concluded this issue is caused by gcc 4.7's optimization; it wrongly optimizes out assignment to buf. Its workaround can be following: diff --git a/ext/json/parser/parser.c b/ext/json/parser/parser.c index d1d14c7..f3bf50c 100644 --- a/ext/json/parser/parser.c +++ b/ext/json/parser/parser.c @@ -40,8 +40,9 @@ static UTF32 unescape_unicode(const unsigned char *p) return result; } -static int convert_UTF32_to_UTF8(char *buf, UTF32 ch) +static int convert_UTF32_to_UTF8(char *out, UTF32 ch) { + volatile char *buf = out; int len = 1; if (ch <= 0x7F) { buf[0] = (char) ch; ---------------------------------------- Bug #5888: JSON unittest fails https://bugs.ruby-lang.org/issues/5888 Author: Vit Ondruch Status: Open Priority: Normal Assignee: Category: Target version: ruby -v: ruby 1.9.3p0 (2011-10-30) [x86_64-linux] Hello, When building Ruby 1.9.3 (as well as 2.0.0), the make check spits following errors: 1) Failure: test_parse_values(TC_JSON) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json.rb:174]: <["\"\b\n\r\t\u0000\u001F"]> expected but was <["\"\b\n\r\t\xA8\xA8"]>. 2) Failure: test_parser_reset(TC_JSON) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json.rb:291]: <{"a"=>2, "b"=>3.141, "c"=>"c", "d"=>[1, "b", 3.14], "e"=>{"foo"=>"bar"}, "g"=>"\"\u0000\u001F", "h"=>1000.0, "i"=>0.001}> expected but was <{"a"=>2, "b"=>3.141, "c"=>"c", "d"=>[1, "b", 3.14], "e"=>{"foo"=>"bar"}, "g"=>"\"\xA0\xA0", "h"=>1000.0, "i"=>0.001}>. 3) Failure: test_fast_generate(TC_JSONGenerate) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json_generate.rb:78]: <{"a"=>2, "b"=>3.141, "c"=>"c", "d"=>[1, "b", 3.14], "e"=>{"foo"=>"bar"}, "g"=>"\"\u0000\u001F", "h"=>1000.0, "i"=>0.001}> expected but was <{"a"=>2, "b"=>3.141, "c"=>"c", "d"=>[1, "b", 3.14], "e"=>{"foo"=>"bar"}, "g"=>"\"\xA0\xA0", "h"=>1000.0, "i"=>0.001}>. 4) Failure: test_generate(TC_JSONGenerate) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json_generate.rb:47]: <{"a"=>2, "b"=>3.141, "c"=>"c", "d"=>[1, "b", 3.14], "e"=>{"foo"=>"bar"}, "g"=>"\"\u0000\u001F", "h"=>1000.0, "i"=>0.001}> expected but was <{"a"=>2, "b"=>3.141, "c"=>"c", "d"=>[1, "b", 3.14], "e"=>{"foo"=>"bar"}, "g"=>"\"\xA0\xA0", "h"=>1000.0, "i"=>0.001}>. 5) Failure: test_generate_pretty(TC_JSONGenerate) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json_generate.rb:61]: <{"a"=>2, "b"=>3.141, "c"=>"c", "d"=>[1, "b", 3.14], "e"=>{"foo"=>"bar"}, "g"=>"\"\u0000\u001F", "h"=>1000.0, "i"=>0.001}> expected but was <{"a"=>2, "b"=>3.141, "c"=>"c", "d"=>[1, "b", 3.14], "e"=>{"foo"=>"bar"}, "g"=>"\"\xA0\xA0", "h"=>1000.0, "i"=>0.001}>. 6) Failure: test_own_state(TC_JSONGenerate) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json_generate.rb:92]: <{"a"=>2, "b"=>3.141, "c"=>"c", "d"=>[1, "b", 3.14], "e"=>{"foo"=>"bar"}, "g"=>"\"\u0000\u001F", "h"=>1000.0, "i"=>0.001}> expected but was <{"a"=>2, "b"=>3.141, "c"=>"c", "d"=>[1, "b", 3.14], "e"=>{"foo"=>"bar"}, "g"=>"\"\xA0\xA0", "h"=>1000.0, "i"=>0.001}>. 7) Failure: test_chars(TC_JSONUnicode) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json_unicode.rb:57]: <"\x00"> expected but was <"\xA0">. 8) Failure: test_unicode(TC_JSONUnicode) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json_unicode.rb:20]: <["\u00A9 \u2260 \u20AC! \u0001"]> expected but was <["\u00A9 \u2260 \u20AC! \xD8"]>. Please note that I am building Ruby using GCC 4.7 on Fedora Rawhide. I have not seen this errors before with GCC 4.6 -- http://bugs.ruby-lang.org/