[ruby-core:38067] [Ruby 1.9 - Bug #5028][Open] Solaris encoding problems with rdiscount & redcarpet

Issue #5028 has been reported by Jonathan del Strother.

----------------------------------------
Bug #5028: Solaris encoding problems with rdiscount & redcarpet
http://redmine.ruby-lang.org/issues/5028

Author: Jonathan del Strother
Status: Open
Priority: Normal
Assignee: 
Category: 
Target version: 
ruby -v: ruby 1.9.2p180 (2011-02-18 revision 30909) [i386-solaris2.10]


=begin
Hi,
I've been having encoding problems under 1.9.2 and Solaris, which I've been unable to explain.
Certain strings produce invalid encodings when passed through Redcarpet/RDiscount - for example, the tamil character 牋エ - (U+0BB4).

 s = "\u0BB4\n"
 '\x%X\x%X\x%X' % s.each_byte.to_a             # => "\xE0\xAE\xB4"
 
 Redcarpet.new(s).to_html                      # => "<p>\xE0\xAE</p>\n"
 Redcarpet.new(s).to_html.valid_encoding?      # => false
 

So in the original string, that codepoint is represented with the bytes 0xE0,0xAE,0xB4, but after redcarpeting we end up with just 0xAE,0xB4.  Running it through RDiscount results in the bytes 0xE0,0xAE.

On this Solaris box, I get the same result on rubies :
ruby 1.9.2p136 (2010-12-25 revision 30365) [i386-solaris2.10]
ruby 1.9.2p180 (2011-02-18 revision 30909) [i386-solaris2.10]

but I can't reproduce it with the same ruby version on OS X.  

I've reported it against the rdiscount & redcarpet gems here - https://github.com/rtomayko/rdiscount/issues/46 & https://github.com/tanoku/redcarpet/issues/32.

I've been unable to reproduce the problem just by taking the ruby string and performing operations on it like gsub(), split(), encoding(), each_byte(), and so on.  How can I narrow it down any further?



-- 
http://redmine.ruby-lang.org