From: James Edward Gray II <james@...>
Date: 2007-10-29T06:20:48+09:00
Subject: Enhancing the Gateway (Help Needed)

Here's the short-story on the current situation with our mailing list  
to Usenet gateway:

* Our Usenet host rejects multipart/alternative messages
   because they are technically illegal Usenet posts
* This means that some emails do not reach comp.lang.ruby
   (several messages each day according to the logs)
* We don't like this

To solve this, we want to enhance the gateway to convert multipart/ 
alternative messages into something we can legally post to Usenet.  I  
have two thoughts on this strategy:

1.  If possible, we should gather all text/plain portions of an email  
and post those with a content-type of text/plain
2.  If that fails, we can just post the original body but force the  
content-type to text/plain for maximum compatibility

Now I need all of you email and Usenet experts to tell me if that's a  
sane strategy.  If another approach would be better, please clue me in.

I've pretty much made it this far.  The code at the bottom of this  
message is the mail_to_news.rb script used by the gateway rewritten  
using this strategy.

If you aren't familiar with the gateway code, you can get details  
from the articles at:

http://blog.grayproductions.net/categories/the_gateway

There's one problem left I know I haven't solved correctly.  Help me  
figure out a decent strategy for this last piece and we can deploy  
the new code.

The outstanding issue is how to handle character sets for the  
constructed message.  You'll see in the code below that I just pull  
the charset param from the original message, but after looking at a  
few messages, I realize that this doesn't make sense.  For example,  
here are the relevant portions of a recent post that wasn't gated  
correctly:

   Content-Type: multipart/alternative; boundary=Apple-Mail-18-445454026

   --Apple-Mail-18-445454026
   Content-Transfer-Encoding: 7bit
   Content-Type: text/plain;
   	charset=US-ASCII;
   	delsp=yes;
   	format=flowed

As you can see, the overall email doesn't have a charset but each  
text portion can.  If we are going to merge these parts, what's the  
best strategy for handling the charset?

I thought of trying to convert them all to UTF-8 with Iconv, but I'm  
not sure what to do if a type doesn't declare a charset or when Iconv  
chokes on what is declared?  Please share your opinions.

If you are feeling really adventurous, rewrite the relevant portion  
of the code below which I will bracket with a FIX ME comments.

Here's the script:

#!/usr/bin/env ruby

# written by James Edward Gray II <james@grayproductions.net>

$KCODE = "u"

GATEWAY_DIR = File.join(File.dirname(__FILE__), "..").freeze

$LOAD_PATH << File.join(GATEWAY_DIR, "config") << File.join 
(GATEWAY_DIR, "lib")

require "tmail"

require "servers_config"
require "nntp"

require "logger"
require "timeout"

# prepare log
log = Logger.new(ARGV.shift || $stdout)
log.datetime_format = "%Y-%m-%d %H:%M "

# build incoming and outgoing message object
incoming = TMail::Mail.parse($stdin.read)
outgoing = TMail::Mail.new

# skip any flagged messages
if incoming["X-Rubymirror"].to_s == "yes"
   log.info "Skipping message ##{incoming.message_id}, sent by  
news_to_mail"
   exit
elsif incoming["X-Spam-Status"].to_s =~ /\AYes/
   log.info "Ignoring Spam ##{incoming.message_id}:  " +
            "#{incoming.subject}–#{incoming.from}"
   exit
end

# only allow certain headers through
%w[from subject in_reply_to transfer_encoding date].each do |header|
   outgoing.send("#{header}=", incoming.send(header))
end
outgoing.message_id = incoming.message_id.sub(/\.+>$/, ">")
%w[X-ML-Name X-Mail-Count X-X-Sender].each do |header|
   outgoing[header] = incoming[header].to_s if incoming.key?header
end

# doctor headers for Ruby Talk
outgoing.references = if incoming.key? "References"
   incoming.references
else
	if incoming.key? "In-Reply-To"
	  incoming.reply_to
	else
		if incoming.subject =~ /^Re:/
		  outgoing.reply_to = "<this_is_a_dummy_message-id@rubygateway>"
		end
	end
end
outgoing["X-Ruby-Talk"]     = incoming.message_id
outgoing["X-Received-From"] = <<END_GATEWAY_DETAILS.gsub(/\s+/, " ")
This message has been automatically forwarded from the ruby-talk  
mailing list by
a gateway at #{ServersConfig::NEWSGROUP}. If it is SPAM, it did not  
originate at
#{ServersConfig::NEWSGROUP}. Please report the original sender, and  
not us.
Thanks! For more details about this gateway, please visit:
http://blog.grayproductions.net/categories/the_gateway
END_GATEWAY_DETAILS
outgoing["X-Rubymirror"]    = "Yes"

# translate the body of the message, if needed
if incoming.multipart? and incoming.sub_type == "alternative"
   ### FIX ME ###
   # handle multipart/alternative messages
   # extract body
   body         = ""
   extract_text = lambda do |message_or_part|
     if message_or_part.multipart?
       message_or_part.each_part { |part| extract_text[part] }
     elsif message_or_part.content_type == "text/plain"
       body += message_or_part.body
     end
   end
   extract_text[incoming]
   if body.empty?
     outgoing.body = "Note:  the content-type of this message was  
altered by " +
                     "the gateway.\n\n#{incoming.body}"
   else
     outgoing.body = "Note:  non-text portions of this message were  
stripped " +
                     "by the gateway.\n\n#{body}"
   end
   # set the content type of the new message
   outgoing.set_content_type( "text", "plain",
                              "charset" => incoming.type_param 
("charset") )
   ### END FIX ME ###
else
   %w[content_type body].each do |header|
     outgoing.send("#{header}=", incoming.send(header))
   end
end

log.info "Sending message ##{incoming.message_id}:  " +
          "#{incoming.subject}–#{incoming.from}…"
log.info "Message looks like:\n#{outgoing.encoded}"

# connect to NNTP host
begin
   nntp = nil
   Timeout.timeout(30) do
     nntp = Net::NNTP.new( ServersConfig::NEWS_SERVER,
           	              Net::NNTP::NNTP_PORT,
                   	      ServersConfig::NEWS_USER,
                         	ServersConfig::NEWS_PASS )
   end
rescue Timeout::Error
     log.error "The NNTP connection timed out"
     exit -1
rescue
	log.fatal "Unable to establish connection to NNTP host:  #{$!.message}"
	exit -1
end

# attempt to send newsgroup post
unless $DEBUG
   begin
     result = nil
     Timeout.timeout(30) { result = nntp.post(outgoing.encoded) }
   rescue Timeout::Error
       log.error "The NNTP post timed out"
       exit -1
   rescue
   	log.fatal "Unable to post to NNTP host:  #{$!.message}"
   	exit -1
   end
   log.info "…  Sent.  nntp.post() result:  #{result}"
end

__END__

Thanks for the help.

James Edward Gray II