From: Brian Candler <b.candler@...>
Date: 2010-04-27T19:19:15+09:00
Subject: Re: Forcing a string to valid UTF-8

Gavin Kistner wrote:
> How do I force it into a valid UTF-8 encoding, losing as little data
> as possible but happily throwing out the senseless bits?

AFAICS, the trouble with your rescue clause is that the string failed to 
be encoded into Windows-1252, so it remains with its existing UTF-8 tag, 
and so an attempt to "re-encode" as UTF-8 is silently ignored because 
it's already UTF-8, even though it contains invalid characters.

For example, this doesn't do anything:

>> a = "abc\xffdef".force_encoding("UTF-8")
=> "abc\xFFdef"
>> b = a.encode("UTF-8", :invalid=>:replace, :replace=>"?")
=> "abc\xFFdef"

but this does:

>> b = a.encode("UTF-16BE", :invalid=>:replace, :replace=>"?").encode("UTF-8")
=> "abc?def"

Proviso: ruby 1.9 string handling is undocumented and subject to 
continuous change. I tested the above with

>> RUBY_DESCRIPTION
=> "ruby 1.9.2dev (2009-07-18 trunk 24186) [i686-linux]"

so it may or may not work with your version, or with future versions of 
Ruby.
-- 
Posted via http://www.ruby-forum.com/.