From: Alex Fenton <alex@...>
Date: 2008-05-12T03:10:25+09:00
Subject: Re: testing stdin for bad encoding, ruby 1.9

Ben Crowell wrote:
> I have some existing ruby 1.9 code that broke recently with a new build
> of ruby. It looks like the problem was that my preexisting text input
> files, which I'd been reading from stdin, contained some characters that
> were not valid UTF-8 or US-ASCII. 

...

> I'm happy to change the input files, because it is an error that they
> aren't properly encoded. However, I'd also like to find some way to test
> for this type of error more gracefully, and I can't seem to figure out
> how to do it.

I use IConv in the standard library to convert from UTF8 to UTF8 to test 
whether files being imported by a user are in fact in the right 
encoding. This otherwise redundant recoding will raise a 
BadSequenceError if there's a problem. This can be caught and reported.

a