From: Caleb Clausen <vikkous@...>
Date: 2010-09-27T06:13:55+09:00
Subject: Re: Encoding problems .. ruby 1.9.2

On 9/26/10, Bhay Zone <bhayzone@yahoo.co.in> wrote:
> I am pretty new to ruby and am trying to read text data coming from a
> backend which can only be queried using proprietary Command Line
> Interface commands.
>
> The problem is that this text data contains non-ascii characters...I
> don't know what these characters are .. and nor do I know the encoding.
>
> Earlier, when we were using ruby 1.8.7 we had some code that handled
> these characters pretty well. Now after switching to ruby 1.9.2, the
> same code breaks with encoding errors like "invalid multibyte sequence"
> in gsub.
>
> Here is the code we were using to replace the non-ascii characters which
> is breaking now. The code it breaks at the first line.
>
> content.gsub!( "\221", '')
>   content.gsub!( "\222", '')
>   content.gsub!( "\223", '')
>   content.gsub!( "\224", '')
>   content.gsub!( "\246", '')
>   content.gsub!( "\247", '')
>   content.gsub!( "\237", '')
>   content.gsub!( "\377", '')
>   content.gsub!( "\226", '')
>   content.gsub!( "\227", '')
>   content.gsub!( "\\000", "?")
>   content.gsub!( "\\001", "?")
>   content.gsub!( "\FB01", "")
>   content.gsub!(/[\x80-\xFF]/,'')
>   content.gsub!(/[\x00-\x08]/,'')
>   content.gsub!(/[\x0B-\x0C]/,'')
>   content.gsub!(/[\x0E-\x1F]/,'')
>
> I just cannot figure how to fix this problem and any help would be
> greatly appreciated.

In 1.9, every string (and regular expression) has an encoding attached
to it. If there are any byte sequences in your string that don't match
the encoding, it causes errors. 1.8 was much more permissive about its
strings, allowing arbitrary binary data in any string, which is why it
worked better for you. You can get back the 1.8 behavior under 1.9 by
setting the encoding of your string objects to 'binary'.

My first suggestion would be to set the encoding of the string in the
variable content to binary before doing any of the gsub!s:
  content.force_encoding('binary')

However, a better way would be to set the encoding of the IO object
the strings are read from. That way you don't need to force_encoding
each string as it comes in.

Even better is to figure out what the encoding this external tool is
using and set the IO's encoding to that. Then perhaps a lot of this
hacky string manglich could go away.

But this is still only half the story. You also have to consider the
encoding of the strings and regexps which get passed as the first
argument to gsub. Those string (and regexp) literals default to the
same encoding as the source file they're contained in. If no explicit
encoding is declared for a specific source file, ruby guesses an
encoding based on your environment (using the LOCALE env var and some
others that I can't remember right now). Often, this means ruby
assumes your sources
are utf-8 encoded.

You can declare a specific encoding explicitly by putting something
like this as the very first line in your source:
#encoding: binary
(or the second line if the first line is a shebang line).

I used the binary encoding in the example line above because that's
probably the one which will work best for you under the circumstances.
Declaring the source encoding to be binary is a bit hackish, but
probably the easiest way to get you where you want to go. If you
figure out what encoding your data is in, you're probably better off
declaring the source encoding to be the same thing, but there may be
more work involved there.

PS: there is some redundancy in the sequence of gsub!s you posted. The
first 10 (for "\221" thru "\227") are special cases of the 14th (for
/[\x80-\xFF]/) and can safely be deleted. Also, "\FB01" is the same
thing as "FB01" in both ruby 1.8 and 1.9 and probably not what you
wanted. (Maybe "\xFB\x01" is what you actually meant?)

HTH