[ruby-core:52202] [ruby-trunk - Bug #7758] Ruby on Windows crashes when active codepage is codepage 65001 and outputting unicode character

Issue #7758 has been updated by mame (Yusuke Endoh).

Assignee changed from mame (Yusuke Endoh) to luislavena (Luis Lavena)

Usa-san, thank you for the information!

Luis, could you please commit it to both trunk and ruby_2_0_0 branch?

-- 
Yusuke Endoh <[e-mail]>

----------------------------------------
Bug #7758: Ruby on Windows crashes when active codepage is codepage 65001 and outputting unicode character
https://bugs.ruby-lang.org/issues/7758#change-36226

Author: joshc (Josh C)
Status: Assigned
Priority: Normal
Assignee: luislavena (Luis Lavena)
Category: M17N
Target version: 
ruby -v: ruby 1.9.3p374


=begin
I have a script that contains the unicode BOX_DRAWINGS_LIGHT_VERTICAL_AND_RIGHT character (http://www.fileformat.info/info/unicode/char/251c/index.htm). When the console code page is set to 65001 (utf-8), both ruby 1.9.3 p125 and p374 crash, but ruby 1.8.7 p357 does not crash (nor does it output the character correctly). When the code page is set to 437, ruby 1.9.3 (both versions) correctly output the character, while ruby 1.8.7 outputs the wrong character again.

I don't care about ruby 1.8.7 so much, but I would expect ruby 1.9.3 to output the correct character, especially since the console codepage is set to 65001 (utf-8).

Here is my script:

 # -*- coding: utf-8 -*-
 puts "a"
 puts "├"

I verified that the file is correctly encoded:

 C:\test>od.exe -t x1 test65001.rb
 0000000 23 20 2d 2a 2d 20 63 6f 64 69 6e 67 3a 20 75 74
 0000020 66 2d 38 20 2d 2a 2d 0a 70 75 74 73 20 22 61 22
 0000040 0a 70 75 74 73 20 22 e2 94 9c 22 0a 0a

Note the UTF-8 sequence `e2 94 9c`

Here are the codepage settings in the registry:
 C:\test>reg query hklm\SYSTEM\CurrentControlSet\Control\Nls\CodePage /v acp
 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage
    acp    REG_SZ    1252
 C:\test>reg query hklm\SYSTEM\CurrentControlSet\Control\Nls\CodePage /v oemcp
 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage
    oemcp    REG_SZ    437
 
On ruby 1.8.7:

 C:\test>ruby --version
 ruby 1.8.7 (2011-12-28 patchlevel 357) [i386-mingw32]
 
 C:\test>chcp 437
 Active code page: 437
 
 C:\test>ruby test65001.rb
 a
 ├
 
 C:\test>chcp 65001
 Active code page: 65001
 
 C:\test>ruby test65001.rb
 a
 ├

On ruby 1.9.3p125:

 C:\test>ruby --version
 ruby 1.9.3p125 (2012-02-16) [i386-mingw32]
 
 C:\test>chcp 437
 Active code page: 437
 
 C:\test>ruby test65001.rb
 a
 Ã
 
 C:\test>chcp 65001
 Active code page: 65001
 
 C:\test>ruby test65001.rb
 a
 ├test65001.rb:3: [BUG] rb_sys_fail(<STDOUT>) - errno == 0
 ruby 1.9.3p125 (2012-02-16) [i386-mingw32]
 
 -- Control frame information -----------------------------------------------
 c:0006 p:---- s:0014 b:0014 l:000013 d:000013 CFUNC  :write
 c:0005 p:---- s:0012 b:0012 l:000011 d:000011 CFUNC  :puts
 c:0004 p:---- s:0010 b:0010 l:000009 d:000009 CFUNC  :puts
 c:0003 p:0023 s:0006 b:0006 l:00215c d:0015b4 EVAL   test65001.rb:3
 c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
 c:0001 p:0000 s:0002 b:0002 l:00215c d:00215c TOP
 
 -- Ruby level backtrace information ----------------------------------------
 test65001.rb:3:in `<main>'
 test65001.rb:3:in `puts'
 test65001.rb:3:in `puts'
 test65001.rb:3:in `write'
 
 -- C level backtrace information -------------------------------------------
 C:\Windows\SysWOW64\ntdll.dll(ZwWaitForSingleObject+0x15) [0x77bbf8b1]
 C:\Windows\syswow64\kernel32.dll(WaitForSingleObjectEx+0x43) [0x772d1194]
 C:\Windows\syswow64\kernel32.dll(WaitForSingleObject+0x12) [0x772d1148]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_vm_bugreport+0xf9) [0x62e5c61d]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_name_err_mesg_new+0x17a) [0x62d3a6fa]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_bug+0x2f) [0x62d3b45b]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_sys_fail+0x163) [0x62d3cfb7]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_io_close+0x171f) [0x62d651cb]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_vm_invoke_proc+0x2fe) [0x62e54586]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_funcall+0x59) [0x62e55035]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_io_puts+0x7e) [0x62d5fb7e]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_thread_mark+0x741) [0x62e49ca1]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_vm_invoke_proc+0x2fe) [0x62e54586]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_io_puts+0x200) [0x62d5fd00]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_thread_mark+0x741) [0x62e49ca1]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_vm_call+0x2b2) [0x62e5855a]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_vm_localjump_error+0x1f66) [0x62e4ed8a]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_vm_localjump_error+0x650e) [0x62e53332]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_iseq_eval_main+0x98) [0x62e596f0]
 c:\ruby193\bin\msvcrt-ruby191.dll(rb_check_frozen+0x2a89) [0x62d3fed9]
 c:\ruby193\bin\msvcrt-ruby191.dll(ruby_run_node+0x48) [0x62d41ff4]
 [0x0040136f]
 [0x004010b9]
 [0x00401284]
 C:\Windows\syswow64\kernel32.dll(BaseThreadInitThunk+0x12) [0x772d339a]
 C:\Windows\SysWOW64\ntdll.dll(RtlInitializeExceptionChain+0x63) [0x77bd9ef2]
 
 -- Other runtime information -----------------------------------------------
 
 * Loaded script: test65001.rb
 
 * Loaded features:
 
    0 enumerator.so
    1 c:/ruby193/lib/ruby/1.9.1/i386-mingw32/enc/encdb.so
    2 c:/ruby193/lib/ruby/1.9.1/i386-mingw32/enc/iso_8859_1.so
    3 c:/ruby193/lib/ruby/1.9.1/i386-mingw32/enc/trans/transdb.so
    4 c:/ruby193/lib/ruby/site_ruby/1.9.1/rubygems/defaults.rb
    5 c:/ruby193/lib/ruby/1.9.1/i386-mingw32/rbconfig.rb
    6 c:/ruby193/lib/ruby/site_ruby/1.9.1/rubygems/deprecate.rb
    7 c:/ruby193/lib/ruby/site_ruby/1.9.1/rubygems/exceptions.rb
    8 c:/ruby193/lib/ruby/site_ruby/1.9.1/rubygems/defaults/operating_system.rb
    9 c:/ruby193/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb
   10 c:/ruby193/lib/ruby/site_ruby/1.9.1/rubygems.rb
   11 c:/ruby193/lib/ruby/1.9.1/i386-mingw32/enc/utf_16le.so
   12 c:/ruby193/lib/ruby/1.9.1/i386-mingw32/enc/trans/utf_16_32.so
   13 c:/ruby193/lib/ruby/1.9.1/i386-mingw32/enc/trans/single_byte.so

 [NOTE]
 You may have encountered a bug in the Ruby interpreter or extension libraries.
 Bug reports are welcome.
 For details: http://www.ruby-lang.org/bugreport.html
 
 This application has requested the Runtime to terminate it in an unusual way.
 Please contact the application's support team for more information.

On ruby 1.9.3p374:

 c:\test>ruby --version
 ruby 1.9.3p374 (2013-01-15) [i386-mingw32]
 
 c:\test>chcp 437
 Active code page: 437
 
 c:\test>ruby test65001.rb
 a
 Ã
 
 c:\test>chcp 65001
 Active code page: 65001
 
 c:\test>ruby test65001.rb
 a
 ├test65001.rb:3: [BUG] rb_sys_fail_str(<STDOUT>) - errno == 0
 ruby 1.9.3p374 (2013-01-15) [i386-mingw32]
 
 -- Control frame information -----------------------------------------------
 c:0006 p:---- s:0014 b:0014 l:000013 d:000013 CFUNC  :write
 c:0005 p:---- s:0012 b:0012 l:000011 d:000011 CFUNC  :puts
 c:0004 p:---- s:0010 b:0010 l:000009 d:000009 CFUNC  :puts
 c:0003 p:0023 s:0006 b:0006 l:001afc d:00059c EVAL   test65001.rb:3
 c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
 c:0001 p:0000 s:0002 b:0002 l:001afc d:001afc TOP
 
 -- Ruby level backtrace information ----------------------------------------
 test65001.rb:3:in `<main>'
 test65001.rb:3:in `puts'
 test65001.rb:3:in `puts'
 test65001.rb:3:in `write'
 
 -- C level backtrace information -------------------------------------------
 C:\Windows\SysWOW64\ntdll.dll(ZwWaitForSingleObject+0x15) [0x77bbf8b1]
 C:\Windows\syswow64\kernel32.dll(WaitForSingleObjectEx+0x43) [0x772d1194]
 C:\Windows\syswow64\kernel32.dll(WaitForSingleObject+0x12) [0x772d1148]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_vm_bugreport+0xf9) [0x62e5c661]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_name_err_mesg_new+0x17a) [0x62d3a81a]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_bug+0x2f) [0x62d3b4f7]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_sys_fail_str+0x15f) [0x62d3d3a3]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_io_flush+0x1336) [0x62d64e22]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_vm_invoke_proc+0x2fe) [0x62e5498a]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_funcall+0x59) [0x62e553e9]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_io_puts+0x78) [0x62d60484]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_vm_mark+0x54d) [0x62e49b85]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_vm_invoke_proc+0x2fe) [0x62e5498a]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_io_puts+0x248) [0x62d60654]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_vm_mark+0x54d) [0x62e49b85]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_vm_call+0x2b2) [0x62e5882e]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_vm_localjump_error+0x1f84) [0x62e4f1b0]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_vm_localjump_error+0x6496) [0x62e536c2]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_iseq_eval_main+0xd7) [0x62e597ef]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(rb_check_frozen+0x2a85) [0x62d4055d]
 c:\Ruby193p374\bin\msvcrt-ruby191.dll(ruby_run_node+0x48) [0x62d42688]
 [0x0040136f]
 [0x004010b9]
 [0x00401284]
 C:\Windows\syswow64\kernel32.dll(BaseThreadInitThunk+0x12) [0x772d339a]
 C:\Windows\SysWOW64\ntdll.dll(RtlInitializeExceptionChain+0x63) [0x77bd9ef2]
 
 -- Other runtime information -----------------------------------------------
 
 * Loaded script: test65001.rb
 
 * Loaded features:

    0 enumerator.so
    1 c:/Ruby193p374/lib/ruby/1.9.1/i386-mingw32/enc/encdb.so
    2 c:/Ruby193p374/lib/ruby/1.9.1/i386-mingw32/enc/iso_8859_1.so
    3 c:/Ruby193p374/lib/ruby/1.9.1/i386-mingw32/enc/trans/transdb.so
    4 c:/Ruby193p374/lib/ruby/site_ruby/1.9.1/rubygems/defaults.rb
    5 c:/Ruby193p374/lib/ruby/1.9.1/i386-mingw32/rbconfig.rb
    6 c:/Ruby193p374/lib/ruby/site_ruby/1.9.1/rubygems/deprecate.rb
    7 c:/Ruby193p374/lib/ruby/site_ruby/1.9.1/rubygems/exceptions.rb
    8 c:/Ruby193p374/lib/ruby/site_ruby/1.9.1/rubygems/defaults/operating_system.rb
    9 c:/Ruby193p374/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb
   10 c:/Ruby193p374/lib/ruby/site_ruby/1.9.1/rubygems.rb
   11 c:/Ruby193p374/lib/ruby/1.9.1/i386-mingw32/enc/utf_16le.so
   12 c:/Ruby193p374/lib/ruby/1.9.1/i386-mingw32/enc/trans/utf_16_32.so
   13 c:/Ruby193p374/lib/ruby/1.9.1/i386-mingw32/enc/trans/single_byte.so

 [NOTE]
 You may have encountered a bug in the Ruby interpreter or extension libraries.
 Bug reports are welcome.
 For details: http://www.ruby-lang.org/bugreport.html

 This application has requested the Runtime to terminate it in an unusual way.
 Please contact the application's support team for more information.

Based on the line `c:/Ruby193p374/lib/ruby/1.9.1/i386-mingw32/enc/iso_8859_1.so` it makes me think ruby is looking at the registry for the active code page (cp1252) instead of calling GetConsoleOutputCP to see which codepage it should be using, but that is just a guess.

=end



-- 
http://bugs.ruby-lang.org/