From: "Carlo E. Prelz" <fluido@...>
Date: 2012-10-31T17:32:07+09:00
Subject: Re: REXML & HTMLentities incorrectly map to UTF-8

	Subject: REXML & HTMLentities incorrectly map to UTF-8
	Date: Wed 31 Oct 12 05:30:50AM +0900

Quoting Mark S. (lists@ruby-forum.com):

> I have some XML data (UTF 8) that I'm trying to convert into another XML
> set which will be eventually UTF 16. The data contains encoded html/xml
> entities.
> 
> The problem is that when I try to parse the data, html/xml entities
> inside of CDATA text are converted into 2-byte codes that don't match
> their original usage.
> 
> For instance, &#146: (should be right single quote) is translated into
> bytes C292 when parsed and exported and examined in a hex editor.

I am not really sure about what happens within rexml there, but when
you get your CDATA string, if you are sure that the stuff inside is
UTF-8, you can force the encoding. By

p string.encoding

you can see the current encoding, and by

string.force_encoding('utf-8')

you can, *without changing the byte content of the string*, change
Ruby's idea of how to interpret it. 

Then, you should be able to obtain a utf-16 version of the same string
by

string_16=string.encode('utf-16')

I cannot assure this works. In the past, I poured bucketfuls of sweat
and tears on character encodings - I was able to reach what I wanted
to reach only by (much) trial and (much) error.

Carlo

-- 
  *         Se la Strada e la sua Virtu' non fossero state messe da parte,
* K * Carlo E. Prelz - fluido@fluido.as             che bisogno ci sarebbe
  *               di parlare tanto di amore e di rettitudine? (Chuang-Tzu)