Re: Mail filter (was RE: announce@ == less email (FAQ item?))

-----BEGIN PGP SIGNED MESSAGE-----

In article <anmco6$ffo3l$[e-mail]>,
Stefan Schmiedl  <[e-mail]> wrote:
>On Sat, 5 Oct 2002 16:14:53 +0900,
>Pe, Botp <[e-mail]> wrote:
>> 
>> 
>>> Stefan Schmiedl [mailto:[e-mail]] explains
>>>
>>>> >> Julian Fitzell [mailto:[e-mail]] enlightens
>>> >> 
>>> >> ...
>>> >> accept if message.subject =~ /Ruby Weekly News/i
>>> >> accept if message.subject =~ /ruby-dev summary/i
>>> >> ...
>>> >> 
>>> 
>>> It seems to be a handmade replacement of the
>>> fetchmail - procmail toolchain on linux.
>>> 
>> Yes. I've seen procmail scripts, and they don't come close...
>> 
>> I hope he'd share it though ;-)
>> 
>
>Well, it's not that difficult ... when I upgraded
>my box last week, the new version of fetchmail got
>confused by something in the first message of the
>mailbox, so I took a look at net/smtp.rb (IIRC),
>and fumbled something together to take a closer look.
>
>However, a simultaneously performed downgrade of
>fetchmail did the job, too, so I stopped tinkering.
>
>It is very easily doable, as long as your output
>is easy to manage, like appending to different files.
>
>Go on, do it yourself ... or investigate at RAA.
>
>s.


- - Here's mine. It's got a whole bunch of wierd stuff in it since
  I need to read my email out of NFS and write it into AFS. It
  uses ifile and the rmail and rfilter modules, but it might
  give you some ideas. I've found the rmail/rfilter stuff pretty
  useful. 

- - Booker C. Bense 

#!/var/local/bin/ruby
# Get the Mailspool and process it. 
#
#

require 'rmail/parser'
require 'rmail/parser/mbox' 
require 'tempfile'

require 'rfilter/deliver'

# Manage Ifile interactions

#
# $Id: ifile.rb,v 1.1 2002/09/25 20:02:13 bbense Exp $
#
# Ifile, a class for interacting with ifile program.
# Get ifile at http://www.ai.mit.edu/~jrennie/ifile/
#
# Booker C. Bense <[e-mail]> 
#

require 'open3'

module Ifile

# This is the wrong name, but I can't think of anything better.
class Process
  # Tell me where ifile lives.
  def initialize(path="/var/local/bin/ifile",args="--verbosity=0") 
    if FileTest.executable?(path) then 
      @ifile = path
      @args = args
    else
      raise ArgumentError 
    end 
  end

  # Given a message, query folders

  def query(msg)
    results = Array.new
    output = self.run_ifile(msg,"--query ")
    i = 0 
    output.each do |line|
      # Format of output is folder score
      folder , score = line.split
      if ( folder && score ) then 
	tmp = Hash.new
	tmp['folder'] = folder
	tmp['score'] = score.to_f
	tmp['position'] = i 
	results << tmp 
	i = i + 1 
      end
    end 
    return results
  end 

  # Add a message to a folder
  def add(msg,folder)
    output = self.run_ifile(msg,"--insert=#{folder}") 
  end
  # Delete a message from a folder
  def delete(msg,folder) 
    output = self.run_ifile(msg,"--delete=#{folder}") 
  end 

  # Refile
  def refile(msg,oldfolder,newfolder) 
    self.delete(msg,oldfolder) 
    self.add(msg,newfolder)
  end 
# internal methods
  
  
  def run_ifile(msg,args) 
    stdin, stdout, stderr = Open3.popen3("#{@ifile} #{@args} #{args}")
    #write msg to ifile
    msg.each { |line| stdin.puts line } 
    stdin.close
    #Read output 
    output = stdout.readlines
    stdout.close
    stderr.close
    return output
  end 


end


end # module Ifile
 

class MailSpool 
  # Set the spool name
  def initialize(user,path="/var/spool/mail/")
    if ( user == nil ) then
    	user = Etc.getpwuid(Process.uid).name	
    end
    @spool = File.open("#{path}#{user}","r+");   
    @tmpspool = Tempfile.new("mailspool") 
    @parser = RMail::Parser.new
    @mreader = RMail::Parser::MBoxReader.new(@tmpspool)
  end
  # Move the spool to tmpspool 

  def update
    begin
      if @spool.stat.size > 0 then
	# Lock the spool file	
	@spool.flock(File::LOCK_EX) 
	# Read it in 
	@spool.rewind
	new_msgs = @spool.read
	# Empty it.
	@spool.truncate(0)
        # Unlock it 
	@spool.flock(File::LOCK_UN)
	@tmpspool.truncate(0)
	# Write it out to tmpspool
	@tmpspool.write(new_msgs)
        @tmpspool.close
      return true
     else
      return nil
     end 
    rescue 
      return nil
    end
  end

  # Return an array of messages. 
  def messages
    @tmpspool.open
  
    messages = []
    loop do
      msg = @parser.parse(@mreader.next)
      if msg.header.mbox_from then
	messages << msg
      else
	break
      end
    end
    messages
  end  

end 

# These are really HeaderFilters. 
class Filter
  
  # remember the regexp test and the folder it maps to. 
  def initialize ( test, folder ) 
    @test = test
    @folder = folder
    @has_proc = nil
  end 

  def test
    @test
  end 

  def folder
    @folder
  end

  def has_proc?
    @has_proc
  end
  
# This proc is for after a successful match to do 
# dynamic folder setting. 
  def addProc(proc)
    if defined? proc.call then 
      @proc = proc
      @has_proc = true
    else
      raise ArguementError
    end
  end 

  def call(msg) 
    @proc.call(msg)
  end

end 

# Return a string that indicates a folder to file into. 
# Add auto month splitting ? 
class FilterProcess 

 def initialize(user, filterdir, default_folder="incoming" )
   @user = user
   @default_folder = "#{filterdir}/#{default_folder}"
   @filterdir = filterdir

#   Switch to using date folder for auto month rollover
#   @folderdir = "#{filterdir}/Now"
   tmp = Time.now.asctime
   tmpfolder = tmp.split[1] + tmp.split[4]
   @folderdir = "#{filterdir}/#{tmpfolder}"
   @logfile = File.open("#{filterdir}/log.#{tmpfolder}","a")

   @filters = Hash.new
   @filterKeys = Array.new
 end

 def default_folder
   @default_folder
 end

 def folderdir
   @folderdir
 end

 def addFilter(filter,type='envelope') 
   my_type = 'envelope' unless type 
   if ( filter.test.respond_to?(:match) ) && ( defined? filter.folder ) then
     if @filters.has_key? my_type then
       @filters[my_type] << filter
     else 
       @filters[my_type] = Array.new
       @filters[my_type] << filter
       @filterKeys << my_type
     end 
   else
     raise ArgumentError 
   end 
 end

 def process(msg)
   if msg.header then 
     @filterKeys.each do |key| 
       if msg.header.mbox_from && ( key == 'envelope')  then
	 test_string = msg.header.mbox_from
       else 
	 test_string = msg.header[key]
       end
       if test_string then 
	 activeFilter = @filters[key].detect { |filter| filter.test.match(test_string) }
	 if activeFilter then
	   if activeFilter.has_proc? then
	     folder = activeFilter.call(msg)
	     if folder then
	       return "#{@folderdir}/#{folder}"
	     end
	   else
	     # Wrong place to do logging.
	     # self.log("Filing #{msg.header['message-id']} in #{activeFilter.folder}")
	     return "#{@folderdir}/#{activeFilter.folder}"
	   end
	 end
       end
     end #filterKeys
   end #msg.header
   return @default_folder
 end 

 def log (string)
   print "#{string}\n" if ( string =~ /ERROR/ ) 
   @logfile.print "#{string}\n"
 end 

 def initFilters
   filterfile = "#{@filterdir}/filters"
   begin
     File.open(filterfile) do |file|
       while ( line = file.gets ) do 
	 unless ( line =~ /^#/ ) then
	   flag_string , mbox , type = line.split
	   if ( flag_string && mbox ) then
	     self.addFilter(Filter.new(Regexp.new(Regexp.quote(flag_string)),mbox),type) 
	   end
	 end
       end
     end
   rescue
     print "Error in reading #{filterfile}"
     exit
   end
 end 

end #FilterProcess

# For dealing with file
class IfileProcess

  def initialize(filterdir,default_folder)
    @filterdir = filterdir
    @default_folder = default_folder
    @ifile = Ifile::Process.new
    @same = Hash.new
    @same["unix-admin"] = "oldmail"
    @same["admin-log"]  = "oldmail"
    @same[default_folder] = "oldmail"
  end

# Return a full fledge path 
  def query(msg)
    results = @ifile.query(msg) 
    best = results[0]["folder"]
    # Where did oldmail score
    # oldmail = results.detect { |result| result["folder"] == "oldmail" }
    if ( best == "oldmail" ) then
      return @default_folder
    end 
    dest = "#{@filterdir}/#{best}"
    dest.sub!(/\s/,"")
    return dest
  end 

  def add(msg,destination)
    if ( @same[destination] ) then
      folder = @same[destination]
    else
      folder = destination.sub(@filterdir,"")
      folder.sub!(/[\/]/,"")
      # Deal with incoming/oldmail problem 
      if ( @same[folder]  ) 
	folder = @same[folder] 
      end
    end
#    print "Adding message to :#{folder}:\n"
    @ifile.add(msg,folder)
  end

end #IfileProcess

# This is for real.
begin 
  filterP = FilterProcess.new("foobar","/home/foobar/filtermail")

  begin 
    # Be more paranoid add a timestamp and pid
    backup = "/tmp/foobar.backupspool.#{Time.now.to_i}.#{Process.pid}"
    system("/bin/cp /var/spool/mail/foobar #{backup}")
  rescue
    print "Error making backup... /tmp/foobar.backsupspool"
    exit 
  end 
  emergency = "/tmp/foobar.emergency"

  spool = MailSpool.new("foobar")
  filterP.initFilters

#Ifile interface. 
  ifile = IfileProcess.new(filterP.folderdir,filterP.default_folder)

rescue => error
  print "Error in startup. #{error.inspect}\n"
  exit
end

report = Hash.new(0)

if ( spool.update) then 
  messages = spool.messages

  messages.each do |msg|
    # Where does it go ? 
    destination = filterP.process(msg)
    # Do some destination dependant checks. 
    if ( destination == filterP.default_folder ) then
      begin
	new_destination = ifile.query(msg)
        filterP.log("INFO: Ifile redirect #{destination} => #{new_destination}")
	destination = new_destination
      rescue
	filterP.log("ERROR: using ifile.query") 
      end
    end
    report[destination] = report[destination] + 1  
    # Write to destination
    filterP.log("INFO: Filing #{msg.header['message-id']} in #{destination}")
    # Check for error
    begin
      RFilter::Deliver.deliver_mbox(destination,msg)
    rescue
      RFilter::Deliver.deliver_mbox(emergency,msg)
      filterP.log("ERROR: Delivering to #{emergency}")
    end
    # Add to ifile database. 
    begin
      ifile.add(msg,destination)
    rescue => error
      filterP.log("ERROR: using ifile.add #{error.inspect}") 
    end 
  end
end

# print report
report.each do |folder, count|
  print "#{folder} : #{count} new messages\n"
end


-----BEGIN PGP SIGNATURE-----
Version: 2.6.2

iQCVAwUBPaLtDWTWTAjn5N/lAQHBvAP/Zg5+4Zh8qPNQuHg98naoktqzhovZh4ws
61wlZzsQ13SKStl40hF3TqS7x2BbvgYNJnl9r5iC6qXpraxByQVU59yntEY3HP14
22LPvL3r3rIxgzbbO0sQ7mCF2s/Gw4wVfk0xHxgM4zSjZ9uKqNz3KdS8B0eKtUR3
TTSGrodsN6g=
=AaZQ
-----END PGP SIGNATURE-----