[ruby-core:42130] [ruby-trunk - Bug #5888] JSON unittest fails

Issue #5888 has been updated by Yui NARUSE.


Bohuslav Kabrda wrote:
> After few hours of debugging, it seems that the flaw may actually be in the json_string_unescape function (in the same file, deeper in the callstack).
> When compiled with -O0, it shows nothing suspicious, but with -O2, it shows some possible problems:
> 
> 1344	                        unescape_len = convert_UTF32_to_UTF8(buf, ch);
> convert_UTF32_to_UTF8 (ch=0, buf=0x7fffffffdd00 "\270", <incomplete sequence \343\225>) at parser.c:46
> 
> -- The incomplete sequence is the thing that is weird. The "ch" variable is created using some byte shifting in the json_string_unescape function, which may be the cause (if the optimization does something with the byte shifting).

Incomplete sequence in buf is not the root of this issue because its content and the length are explicitly
specified in convert_UTF32_to_UTF8().
But yes, the bug is here.

The root of this issue is optimization for convert_UTF32_to_UTF8().
With -O2 convert_UTF32_to_UTF8() is inlined to json_string_unescape.
If ch < 0x7F, it runs as

   0x00000008034017a0 <+1120>:  cmp    $0x7f,%rsi                            # if (ch <= 0x7F) {
   0x00000008034017a4 <+1124>:  jbe    0x80340180c <JSON_parse_string+1228>
   0x000000080340180c <+1228>:  mov    $0x1,%edx                             # int len = 1;
   0x0000000803401811 <+1233>:  jmp    0x8034017b4 <JSON_parse_string+1140>
   0x00000008034017b4 <+1140>:  lea    0x20(%rsp),%rsi
   0x00000008034017b9 <+1145>:  jmpq   0x8034016c0 <JSON_parse_string+896>   # return

So it doesn't change buf.
I concluded this issue is caused by gcc 4.7's optimization; it wrongly optimizes out assignment to buf.

Its workaround can be following:

diff --git a/ext/json/parser/parser.c b/ext/json/parser/parser.c
index d1d14c7..f3bf50c 100644
--- a/ext/json/parser/parser.c
+++ b/ext/json/parser/parser.c
@@ -40,8 +40,9 @@ static UTF32 unescape_unicode(const unsigned char *p)
     return result;
 }

-static int convert_UTF32_to_UTF8(char *buf, UTF32 ch)
+static int convert_UTF32_to_UTF8(char *out, UTF32 ch)
 {
+    volatile char *buf = out;
     int len = 1;
     if (ch <= 0x7F) {
         buf[0] = (char) ch;
----------------------------------------
Bug #5888: JSON unittest fails
https://bugs.ruby-lang.org/issues/5888

Author: Vit Ondruch
Status: Open
Priority: Normal
Assignee: 
Category: 
Target version: 
ruby -v: ruby 1.9.3p0 (2011-10-30) [x86_64-linux]


Hello,

When building Ruby 1.9.3 (as well as 2.0.0), the make check spits following errors:

  1) Failure:
test_parse_values(TC_JSON) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json.rb:174]:
<["\"\b\n\r\t\u0000\u001F"]> expected but was
<["\"\b\n\r\t\xA8\xA8"]>.
  2) Failure:
test_parser_reset(TC_JSON) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json.rb:291]:
<{"a"=>2,
 "b"=>3.141,
 "c"=>"c",
 "d"=>[1, "b", 3.14],
 "e"=>{"foo"=>"bar"},
 "g"=>"\"\u0000\u001F",
 "h"=>1000.0,
 "i"=>0.001}> expected but was
<{"a"=>2,
 "b"=>3.141,
 "c"=>"c",
 "d"=>[1, "b", 3.14],
 "e"=>{"foo"=>"bar"},
 "g"=>"\"\xA0\xA0",
 "h"=>1000.0,
 "i"=>0.001}>.
  3) Failure:
test_fast_generate(TC_JSONGenerate) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json_generate.rb:78]:
<{"a"=>2,
 "b"=>3.141,
 "c"=>"c",
 "d"=>[1, "b", 3.14],
 "e"=>{"foo"=>"bar"},
 "g"=>"\"\u0000\u001F",
 "h"=>1000.0,
 "i"=>0.001}> expected but was
<{"a"=>2,
 "b"=>3.141,
 "c"=>"c",
 "d"=>[1, "b", 3.14],
 "e"=>{"foo"=>"bar"},
 "g"=>"\"\xA0\xA0",
 "h"=>1000.0,
 "i"=>0.001}>.
  4) Failure:
test_generate(TC_JSONGenerate) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json_generate.rb:47]:
<{"a"=>2,
 "b"=>3.141,
 "c"=>"c",
 "d"=>[1, "b", 3.14],
 "e"=>{"foo"=>"bar"},
 "g"=>"\"\u0000\u001F",
 "h"=>1000.0,
 "i"=>0.001}> expected but was
<{"a"=>2,
 "b"=>3.141,
 "c"=>"c",
 "d"=>[1, "b", 3.14],
 "e"=>{"foo"=>"bar"},
 "g"=>"\"\xA0\xA0",
 "h"=>1000.0,
 "i"=>0.001}>.
  5) Failure:
test_generate_pretty(TC_JSONGenerate) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json_generate.rb:61]:
<{"a"=>2,
 "b"=>3.141,
 "c"=>"c",
 "d"=>[1, "b", 3.14],
 "e"=>{"foo"=>"bar"},
 "g"=>"\"\u0000\u001F",
 "h"=>1000.0,
 "i"=>0.001}> expected but was
<{"a"=>2,
 "b"=>3.141,
 "c"=>"c",
 "d"=>[1, "b", 3.14],
 "e"=>{"foo"=>"bar"},
 "g"=>"\"\xA0\xA0",
 "h"=>1000.0,
 "i"=>0.001}>.
  6) Failure:
test_own_state(TC_JSONGenerate) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json_generate.rb:92]:
<{"a"=>2,
 "b"=>3.141,
 "c"=>"c",
 "d"=>[1, "b", 3.14],
 "e"=>{"foo"=>"bar"},
 "g"=>"\"\u0000\u001F",
 "h"=>1000.0,
 "i"=>0.001}> expected but was
<{"a"=>2,
 "b"=>3.141,
 "c"=>"c",
 "d"=>[1, "b", 3.14],
 "e"=>{"foo"=>"bar"},
 "g"=>"\"\xA0\xA0",
 "h"=>1000.0,
 "i"=>0.001}>.
  7) Failure:
test_chars(TC_JSONUnicode) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json_unicode.rb:57]:
<"\x00"> expected but was
<"\xA0">.
  8) Failure:
test_unicode(TC_JSONUnicode) [/builddir/build/BUILD/ruby-1.9.3-p0/test/json/test_json_unicode.rb:20]:
<["\u00A9 \u2260 \u20AC! \u0001"]> expected but was
<["\u00A9 \u2260 \u20AC! \xD8"]>.


Please note that I am building Ruby using GCC 4.7 on Fedora Rawhide. I have not seen this errors before with GCC 4.6


-- 
http://bugs.ruby-lang.org/