From: Yui NARUSE <naruse@...>
Date: 2011-07-15T02:27:30+09:00
Subject: [ruby-core:38068] [Ruby 1.9 - Bug #5028][Third Party's Issue] Solaris encoding problems with rdiscount & redcarpet


Issue #5028 has been updated by Yui NARUSE.

Status changed from Open to Third Party's Issue

It is the issue of markdown.c in redcarpet.
In ext/redcarpet/redcarpet.c it passes the data from Ruby's String to markdown.c's input_buffer without Ruby's encoding:

  input_buf.data = RSTRING_PTR(text);
  input_buf.size = RSTRING_LEN(text);
  ...
  ups_markdown(output_buf, &input_buf, &renderer, enabled_extensions);

After this, it is up to markdown.c.
You can inspect the process of it with gdb or printf or something throught ext/redcarpet/markdown.c.
----------------------------------------
Bug #5028: Solaris encoding problems with rdiscount & redcarpet
http://redmine.ruby-lang.org/issues/5028

Author: Jonathan del Strother
Status: Third Party's Issue
Priority: Normal
Assignee: 
Category: 
Target version: 
ruby -v: ruby 1.9.2p180 (2011-02-18 revision 30909) [i386-solaris2.10]


=begin
Hi,
I've been having encoding problems under 1.9.2 and Solaris, which I've been unable to explain.
Certain strings produce invalid encodings when passed through Redcarpet/RDiscount - for example, the tamil character ��� - (U+0BB4).

 s = "\u0BB4\n"
 '\x%X\x%X\x%X' % s.each_byte.to_a             # => "\xE0\xAE\xB4"
 
 Redcarpet.new(s).to_html                      # => "<p>\xE0\xAE</p>\n"
 Redcarpet.new(s).to_html.valid_encoding?      # => false
 

So in the original string, that codepoint is represented with the bytes 0xE0,0xAE,0xB4, but after redcarpeting we end up with just 0xAE,0xB4.  Running it through RDiscount results in the bytes 0xE0,0xAE.

On this Solaris box, I get the same result on rubies :
ruby 1.9.2p136 (2010-12-25 revision 30365) [i386-solaris2.10]
ruby 1.9.2p180 (2011-02-18 revision 30909) [i386-solaris2.10]

but I can't reproduce it with the same ruby version on OS X.  

I've reported it against the rdiscount & redcarpet gems here - https://github.com/rtomayko/rdiscount/issues/46 & https://github.com/tanoku/redcarpet/issues/32.

I've been unable to reproduce the problem just by taking the ruby string and performing operations on it like gsub(), split(), encoding(), each_byte(), and so on.  How can I narrow it down any further?



-- 
http://redmine.ruby-lang.org