From: Jonathan del Strother Date: 2011-07-15T01:24:59+09:00 Subject: [ruby-core:38067] [Ruby 1.9 - Bug #5028][Open] Solaris encoding problems with rdiscount & redcarpet Issue #5028 has been reported by Jonathan del Strother. ---------------------------------------- Bug #5028: Solaris encoding problems with rdiscount & redcarpet http://redmine.ruby-lang.org/issues/5028 Author: Jonathan del Strother Status: Open Priority: Normal Assignee: Category: Target version: ruby -v: ruby 1.9.2p180 (2011-02-18 revision 30909) [i386-solaris2.10] =begin Hi, I've been having encoding problems under 1.9.2 and Solaris, which I've been unable to explain. Certain strings produce invalid encodings when passed through Redcarpet/RDiscount - for example, the tamil character ��� - (U+0BB4). s = "\u0BB4\n" '\x%X\x%X\x%X' % s.each_byte.to_a # => "\xE0\xAE\xB4" Redcarpet.new(s).to_html # => "

\xE0\xAE

\n" Redcarpet.new(s).to_html.valid_encoding? # => false So in the original string, that codepoint is represented with the bytes 0xE0,0xAE,0xB4, but after redcarpeting we end up with just 0xAE,0xB4. Running it through RDiscount results in the bytes 0xE0,0xAE. On this Solaris box, I get the same result on rubies : ruby 1.9.2p136 (2010-12-25 revision 30365) [i386-solaris2.10] ruby 1.9.2p180 (2011-02-18 revision 30909) [i386-solaris2.10] but I can't reproduce it with the same ruby version on OS X. I've reported it against the rdiscount & redcarpet gems here - https://github.com/rtomayko/rdiscount/issues/46 & https://github.com/tanoku/redcarpet/issues/32. I've been unable to reproduce the problem just by taking the ruby string and performing operations on it like gsub(), split(), encoding(), each_byte(), and so on. How can I narrow it down any further? -- http://redmine.ruby-lang.org