From: Phrogz <phrogz@...>
Date: 2010-04-27T23:45:07+09:00
Subject: Re: Forcing a string to valid UTF-8

On Apr 27, 4:19 am, Brian Candler <b.cand...@pobox.com> wrote:
> Gavin Kistner wrote:
> > How do I force it into a valid UTF-8 encoding, losing as little data
> > as possible but happily throwing out the senseless bits?
>
> AFAICS, the trouble with your rescue clause is that the string failed to
> be encoded into Windows-1252, so it remains with its existing UTF-8 tag,
> and so an attempt to "re-encode" as UTF-8 is silently ignored because
> it's already UTF-8, even though it contains invalid characters.

Excellent point. Fixing that led me to a similar error earlier: I had
assumed that
  s2 = s1.force_encoding(...)
left s1 intact. In fact, it modifies and returns s1. Thank you very
much, Brian.

For those that care or stumble upon this via Google, here's a modified
version that works:

  # Converting ASCII-8BIT to UTF-8 based domain-specific guesses
  if new_value.is_a? String
    begin
      # Try it as UTF-8 directly
      cleaned = new_value.dup.force_encoding('UTF-8')
      unless cleaned.valid_encoding?
        # Some of it might be old Windows code page
        cleaned = new_value.encode( 'UTF-8', 'Windows-1252' )
      end
      new_value = cleaned
    rescue EncodingError
      # Force it to UTF-8, throwing out invalid bits
      new_value.encode!( 'UTF-8', invalid: :replace, undef: :replace )
    end
  end

> Proviso: ruby 1.9 string handling is undocumented and subject to
> continuous change. I tested the above with

FWIW my new code works on ruby 1.9.1p243 (2009-07-16 revision 24175)
[i386-mingw32]

Thanks again!