Migrating TMail to 1.9 (1) - Regexp encoding problem

Hello all,

This is a repost (and change) to an email I sent to Ruby-Talk.  I
think it is more appropriate here...

I am the new maintainer for TMail and am getting it working with 1.9.
I had it working (all test passing) in the January version of 1.9, but
it seems the March release of 1.9 has gotten a lot more strict in it's
encoding enforcement as now I have the following error comming up:

ArgumentError: incompatible encoding regexp match (US-ASCII regexp
with ISO-2022-JP string)

I don't really want to set the regexp to UTF-8 or something and then
transliterate the match strings as that just isn't going to scale I
think when you are talking about emails which can have almost anything
in them, and making a regexp for every encoding type also isn't the
solution.

Also, the string that is being matched may not always be an
iso-2022-jp string, so changing the encoding type on the regexp (which
I don't think is possible) wouldn't really work as I would have to do
it for all encodings.

The only other solution I can think of is going through TMail and
making all encodings internal to TMail one type (say UTF-8) and then
transliterating all input and output to match.  But I am not totally
sure what I will run into on that, as while I understand some of the
issues of encodings and charactersets, I am by no means an expert on
the subject.

Plus, I don't really feel like ripping out the insides of TMail right now :)

How are others handling this sort of thing?

The method that is failing is scanadd, at each of the matches.

    def scanadd( str, force = false )
      types = ''
      strs = []

      until str.empty?
        if m = /\A[^\e\t\r\n ]+/.match(str)
          types << (force ? 'j' : 'a')
          strs.push m[0]

        elsif m = /\A[\t\r\n ]+/.match(str)
          types << 's'
          strs.push m[0]

        elsif m = /\A\e../.match(str)
          esc = m[0]
          str = m.post_match
          if esc != "\e(B" and m = /\A[^\e]+/.match(str)
            types << 'j'
            strs.push m[0]
          end

        else
          raise 'TMail FATAL: encoder scan fail'
        end
        (str = m.post_match) unless m.nil?
      end

      do_encode types, strs
    end

And the failing test is in test_encode.rb (for anyone with TMail
installed) and looks like this:

 def test_s_encode
   SRCS.each_index do |i|
     assert_equal crlf(OK[i]),
                  TMail::Encoder.encode(NKF.nkf('-j', SRCS[i]))
   end
 end

 def crlf( str )
   str.gsub(/\n|\r\n|\r/) { "\r\n" }
 end

Which is using the string:

SRCS = ["a cde \343\201\202\343\201\204\343\201\206\343\201\210\343\201\212\343\201\202\343\201\204\343\201\206\343\201\210\343\201\212\343\201\202\343\201\204\343\201\206\343\201\210\343\201\212\343\201\202\343\201\204\343\201\206\343\201\210\343\201\212\343\201\202\343\201\204\343\201\206\343\201\210\343\201\212"]

To match against:

OK = [
 "a cde あいうえおあいうえおあいうえおあ\n\tいうえおあいうえお",
#1
 ]

Regards

Mikel