From: Paul Duncan <pabs@...>
Date: 2005-12-22T02:33:26+09:00
Subject: Re: How are people making use of Iconv?

--PDzrc2MStrmgSXgo
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

* Wilson Bilkovich (wilsonb@gmail.com) wrote:
> On 12/21/05, Paul Duncan <pabs@pablotron.org> wrote:
> > * Andreas S. (f@andreas-s.net) wrote:
> > [snipped]
> > > I dislike that Iconv raises an exception when it finds characters it =
can
> > > not convert. I would prefer if it could be made to ignore invalid
> > > characters and just try to make the best of the text.
> >
> > Seconded, Thirded, and Quadrupled.
> >
> > Iconv needs a "as close as I could get with transliteration and ignoring
> > invalid characters" mode.
> >
> > We're doing something comparable in Raggle by trapping the exception and
> > stripping out the invalid character.  Obviously this doesn't work
> > properly for multibyte characters, and you won't be able to use a lookup
> > table for arbitrary source encodings, but it's a start.
> >
> <snip interesting code>
>=20
> What if String just had a couple of new methods on it:
> String#transcode(from_encoding, to_encoding)
> ..and
> String#transcode!(from_encoding, to_encoding)
> ..and the "modifies receiver" version returned true or false,
> depending on whether it managed to convert every character?
> Then you could do:
> unless some_string.transcode!('Shift-JIS', 'UTF-8')
>   puts "Some characters got mangle-fied!"
> end
>=20
> Is that a mess? I kinda like it, at first glance.

I know a future version of Ruby (2.0?) will make a distinction between
strings as arrays of bytes and strings as sets of characters with an
encoding (with the former being an obvious superset of the latter), so
I'm not sure how well that method would work with the new way of
handling strings.

That said, I like the idea, although I'd like an optional block to
handle unknown characters.  I'd also add an hash as an optional third
argument which allows you to toggle transliteration, munging, and
exception behavior.

--=20
Paul Duncan <pabs@pablotron.org>        pabs in #ruby-lang (OPN IRC)
http://www.pablotron.org/               OpenPGP Key ID: 0x82C29562

--PDzrc2MStrmgSXgo
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFDqZHazdlT34LClWIRAjRbAJsEtvFNzaKZDtrxiZGacEl6jdEM0wCfWnFV
q9msx0Kq0120iTF8C9ILR14=
=npn0
-----END PGP SIGNATURE-----

--PDzrc2MStrmgSXgo--