From: Paul Duncan <pabs@...>
Date: 2005-12-21T23:54:35+09:00
Subject: Re: How are people making use of Iconv?

--SLOnmNMLAcnry0J+
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

* Andreas S. (f@andreas-s.net) wrote:
[snipped]
> I dislike that Iconv raises an exception when it finds characters it can=
=20
> not convert. I would prefer if it could be made to ignore invalid=20
> characters and just try to make the best of the text.

Seconded, Thirded, and Quadrupled.

Iconv needs a "as close as I could get with transliteration and ignoring
invalid characters" mode.

We're doing something comparable in Raggle by trapping the exception and
stripping out the invalid character.  Obviously this doesn't work
properly for multibyte characters, and you won't be able to use a lookup
table for arbitrary source encodings, but it's a start.

    begin
      # convert element_text to native charset (note: in this case we're
      # converting from utf-8 to the native charset, but the only thing
      # about the code that's utf-8 specific is the assumption about
      # character width and the unicode lookup table below)
      ret =3D $iconv.iconv(element_text) << $iconv.iconv(nil)
    rescue Iconv::IllegalSequence =3D> e
      # save the portion of the string that was successful, the=20
      # invalid character, and the remaining (pending) string
      success_str =3D e.success
      ch, pending_str =3D e.failed.split(//, 2)
      ch_int =3D ch.to_i

      # see if we have a map for that characters
      if String::UNICODE_LUT.has_key?(ch_int)
        # we have a mapping for this character, so convert it and
        # re-process the string

        # log status
        err_str =3D _('converting unicode')
        $log.warn(meth) { "#{err_str} ##{ch_int}" }

        # create new string, with the bad character mapped
        element_text =3D success_str + UNICODE_LUT[ch_int] + pending_str
      else
        if $config['iconv_munge_illegal']
          # munge the illegal character with a safe string

          # log status
          err_str =3D _('munging unicode')
          $log.warn(meth) { "#{err_str} ##{ch_int}" }

          # create new string, with the bad character munged
          munge_str =3D $config['unicode_munge_str']
          element_text =3D success_str + munge_str + pending_str
        else
          # just drop the character altogether

          # log status
          err_str =3D _('dropping unicode')
          $log.warn(meth) { "#{err_str} ##{ch_int}" }

          # create new string, sans the bad character
          element_text =3D success_str + pending_str
        end
      end
      retry
    end                                                                =20

Not a perfect solution, but it helps a bit.

--=20
Paul Duncan <pabs@pablotron.org>        pabs in #ruby-lang (OPN IRC)
http://www.pablotron.org/               OpenPGP Key ID: 0x82C29562

--SLOnmNMLAcnry0J+
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFDqWyfzdlT34LClWIRArZXAJ9FPc/IvlgyPI5coH3mRdZyqBMCJgCgiydX
FZyP/peF9b1zmZq6RH2u5iw=
=zDrq
-----END PGP SIGNATURE-----

--SLOnmNMLAcnry0J+--