From: Brian Candler <B.Candler@...>
Date: 2007-02-27T18:21:11+09:00
Subject: MIME decoding confused by non-MIME characters

Could someone who has bleeding-edge Ruby installed please test the
following?

irb(main):001:0> RUBY_VERSION
=> "1.8.5"
irb(main):002:0> a = "b2s="
=> "b2s="
irb(main):003:0> b = "\xef\xbb\xbf" + a
=> "\357\273\277b2s="
irb(main):004:0> a.unpack("m")
=> ["ok"]
irb(main):005:0> b.unpack("m")
=> ["$\000\e\332"]

That is, non-printable characters (here a UTF8-encoded BOM) are causing MIME
unpack to return garbage.

According to RFC 2045 section 6.8,

  "In base64
   data, characters other than those in Table 1, line breaks, and other
   white space probably indicate a transmission error, about which a
   warning message or even a message rejection might be appropriate
   under some circumstances."

So I'd suggest reasonable behaviour might be:

   * skip these characters silently; or
   * skip these characters and warn if -w; or
   * raise an exception

But I don't think that returning garbage is good behaviour :-)

More info on [ruby-talk:238357] and the surrounding thread.

Regards,

Brian Candler.

P.S. I wondered if this was due to parity-stripping taking place, but that
isn't it:

irb(main):006:0> c = "\x6f\x3b\x3f" + a
=> "o;?b2s="
irb(main):007:0> c.unpack("m")
=> [""]