From: Brian Candler <b.candler@...>
Date: 2009-08-07T17:07:12+09:00
Subject: Re: 1.9 String and M17N documentation

James Gray wrote:
>> 2. Working with other people's libraries.
>>
>> Take REXML as an example. Suppose I decide I want to do this:
>>
>>  doc = REXML::Document.new(src)
>>
>> Under 1.8, I could do this without worrying.
> 
> Really?
> 
> What did it do under Ruby 1.8 when fed an XML document that was UTF-16
> encoded?  Will it read it?  When I do searches for content, will it
> hand me UTF-16 or UTF-8?  These are just some questions that jump to
> my mind.

OK, I didn't write my statement clearly enough.

In ruby 1.8, the question is, "will it parse this document?"

In ruby 1.9, the question is, "will it parse this document, *and* does 
the correct parsing depend on which encoding I set the 'src' string to, 
and if so, what do I need to set it to?"

Then take a method which returns a string, say REXML::Element.text(). 
This is a bit simpler.

In ruby 1.8, the question is, "does this return the content of my 
element, and has it been transcoded?"

In ruby 1.9, the question is the same, *plus* "what encoding does it set 
on that value?"

>> require 'rexml/document'
=> true
>> d = REXML::Document.new("<?xml encoding='iso-8859-1'?><root>\xfcber</root>")
=> <UNDEFINED> ... </>
>> d.elements.first
=> <root> ... </>
>> d.elements.first.text
=> "über"
>> d.elements.first.text.encoding
=> #<Encoding:UTF-8>
>> 

OK, so it looks like REXML has transcoded to UTF-8, and tagged the 
result as such. I'm not really helping my case because you have to do 
the same test with 1.8:

>> require 'rexml/document'
=> true
>> d = REXML::Document.new("<?xml encoding='iso-8859-1'?><root>\xfcber</root>")
=> <UNDEFINED> ... </>
>> d.elements[1]
=> <root> ... </>
>> d.elements[1].text
=> "\303\274ber"

So it's been transcoded here too. But I don't have to worry about what 
encoding 'tag' it has been given.

Maybe all this would be much simpler if Ruby didn't crash when given 
incompatible encodings, but transcoded the right-hand-side 
automatically. For example:

  a << b
  # a keeps its original encoding, b is transcoded to a's encoding

  a.tr("ü","Ü")
  # the ü and Ü are transcoded to a's encoding first

- with transcoding to BINARY being a null operation.
-- 
Posted via http://www.ruby-forum.com/.