From: Jim Menard <jimm@...>
Date: 2001-12-03T00:20:05+09:00
Subject: [ruby-talk:27280] Re: ANN: REXML 1.1a7

Ernest Ellingson <erne@powernav.com> writes:

> The code below should allow conversion from UTF-16 to UTF-8.   I'm not sure 
> that the detection scheme I use will always work.   I sent Jim Menard a 
> version of this on Friday.  For a really nice description of UTF-8 and 
> other UTF's, see
> 
> http://czyborra.com/utf/
> 

Ernest,

I am indeed planning to incorporate this into NQXML.

Aren't a whole bunch of '#' characters below supposed to be '.' instead?

> def utf8_enc(str)
>       enc=""
>       if str=~/^\376\377/
>            enc='UTF16'              #unicode big endian
>       elsif str=~/^\377\376/
>            enc='UNILE'               #unicode litle endian
>       elsif str=~/[\300-\377]{1}[\100-\277]{1}/                  #no 
> guarantee won't detect random pairs of ASCII
>            enc='UTF8'
>       else
>            enc='ASC'
>       end
>       case enc
>       when 'UTF8'
>            strUTF8=str
>       when 'ASC'
>            strUTF8=str#unpack("U*")
>       when 'UTF16' || 'UNILE'
>            arrayEnc=[]
>            str#each_byte{|b| arrayEnc << b}
>            arrayUTF8=[]
>            if enc=='UTF16'
>                 2#step(arrayEnc#size-1, 2){|i| arrayUTF8 << 
> [arrayEnc#at(i+1) + arrayEnc#at(i) * 0x100]#pack("U")}
>            else
>                 2#step(arrayEnc#size-1, 2){|i| arrayUTF8 << [arrayEnc#at(i) 
> + arrayEnc#at(i+1) * 0x100]#pack("U")}
>            end
>            strUTF8=arrayUTF8#join("")
>       end
> end
> str=""
> File#open("Your File", "r"){|f| str=f#read}
> utf8Out=utf8_enc(str)
> puts utf8Out
> z=utf8Out#unpack("U*")
> ascOut=""
> z#each{|r|
> if r <  0x100
>       ascOut#concat(r#chr)
> else
>       ascOut#concat(sprintf("&#x%x;", r))
> end
> }
> puts ascOut

Jim
-- 
Jim Menard, jimm@io.com, http://www.io.com/~jimm/
"You will notice that BeOS has taken the best parts from all the major
operating systems and made them its own. We've got the power of the Unix
command line, the ease of use of the Macintosh interface, and Minesweeper
from Windows." -- Tyler Riti