[ruby-core:40769] [Backport93 - Backport #5564] 1.9.3 regression with encoding conversion

Issue #5564 has been updated by Martin D端rst.


Hello Jon,

What I would do now is to try and get a C-level backtrace that shows what happens exactly between the call to encode! and the actual errro.

As for "we don't know what converter we might need at runtime", one solution would be to include all converters. When we designed the whole transcoding stuff, we were careful to save runtime memory by dynamically loading just the stuff that's really needed, but there may be some cases where it's easier to load everything in advance. On a clever OS, even if you run many instances of your application, all the data for the transcodings should be shared because it's all static. (I'm not saying that's the solution to your bug, but it's something that may come in handy in some situations.)

Another solution might be to introduce a way to 'bless' certain libraries in advance so that they can be loaded later under $SAFE >= 3 if they are needed. That would leave a security risk of a library being changed between being blessed and being loaded, but there should be enough ways to prevent that.
----------------------------------------
Backport #5564: 1.9.3 regression with encoding conversion
http://redmine.ruby-lang.org/issues/5564

Author: Jon Leighton
Status: Assigned
Priority: Normal
Assignee: Yuki Sonoda
Category: 
Target version: 


Hello,

We are experiencing a bug in the Rails test suite against ruby 1.9.3.

The bug looks like this:

  1) Error:
test_default_external_works(TestERBTemplate):
ActionView::Template::Error: code converter not found (ISO-8859-1 to UTF-8)
    /home/turnip/Code/rails/actionpack/lib/action_view/template/handlers/erb.rb:83:in `encode!'
    /home/turnip/Code/rails/actionpack/lib/action_view/template/handlers/erb.rb:83:in `call'
    /home/turnip/Code/rails/actionpack/lib/action_view/template.rb:252:in `compile'
    /home/turnip/Code/rails/actionpack/lib/action_view/template.rb:189:in `compile!'
    /home/turnip/Code/rails/actionpack/lib/action_view/template.rb:142:in `block in render'
    /home/turnip/Code/rails/activesupport/lib/active_support/notifications.rb:55:in `instrument'
    /home/turnip/Code/rails/actionpack/lib/action_view/template.rb:141:in `render'
    test/template/template_test.rb:53:in `render'
    test/template/template_test.rb:137:in `block in test_default_external_works'
    test/template/template_test.rb:176:in `with_external_encoding'
    test/template/template_test.rb:135:in `test_default_external_works'
    /home/turnip/.rvm/gems/ruby-1.9.3-p0/gems/mocha-0.10.0/lib/mocha/integration/mini_test/version_230_to_251.rb:28:in `run'

A simplified view of what that code is doing is the following:

 s = "hello \u{fc}mlat"
 s.force_encoding "ISO-8859-1"
 s.encode! 'UTF-8'

However, I have been unable to create any simplified test case for this bug.

What I have been able to find out is the following:

1. Via git-bisect, I tracked down the changed which introduced the bug (it was not present in 1.9.3-preview1). That change was https://github.com/ruby/ruby/commit/05c2cc90895750255f64254f2491ff0ad77cc13a (r32791)
2. If I do some sort of conversion at *load time* in the file action_view/template/handlers/erb.rb, the bug does not occur. For example, if I insert "Encoding::Converter.new('ISO-8859-1', 'UTF-8')" (or the 3 lines above) at the top of the file, I do not experience the bug.

Let me know if I can provide any more information. Also any information about what exactly is triggering this would be very helpful so that we can put a workaround in Rails (if possible).

Thanks


-- 
http://redmine.ruby-lang.org