From: Jeremy Hinegardner <jeremy@...>
Date: 2008-09-18T01:47:22+09:00
Subject: Re: How to clean an xml files from non-utf-8 chars?

On Wed, Sep 17, 2008 at 09:44:23PM +0900, James Gray wrote:
> On Sep 17, 2008, at 4:07 AM, Krzysieq wrote:
>
>> I have a problem. I'm trying to parse with ruby some test results from
>> jmeter, that are stored in xml files. Unfortunately, while they should be 
>> utf-8, some of them aren't. Probably because some db data isn't. In any 
>> case, this makes other toys break down, like xslt transformation and
>> anything else that relies on the xml files being utf-8.
>>
>> Does anyone know, how to get rid of such characters?
>
> If you can figure out the encoding they are actually in, I recommend using 
> Iconv's transliterate mode:
>
> require "iconv"
> Iconv.conv("UTF-8//TRANSLIT", old_encoding_name, data)

This is the approach we have take on some of our code, basically we wanted to
replicate the 'iconv -c' behavior.  Does TRANSLIT do this ?  I've never used
that mode before.

  module UTF8
    module Cleanable
      #   
      # Converts the string representation of this class to a utf8 clean
      # string.  This assumes that #to_s on the object will result in a utf8
      # string.  All chars that are not valid utf8 char sequences will be
      # silently dropped.
      #   
      def utf8_clean
        Iconv.open( "UTF-8", "UTF-8" ) do |iconv|
          output  = StringIO.new
          working = self.to_s
          loop do
            begin
              output.print iconv.iconv( working )
              break
            rescue Iconv::IllegalSequence => is
              output.print is.success
              working = is.failed[1..-1]
            end 
          end 
          return output.string
        end 
      end 
    end 
  end 

  class String
    include UTF8::Cleanable
  end

enjoy,

-jeremy

-- 
========================================================================
 Jeremy Hinegardner                              jeremy@hinegardner.org