From: Claus Spitzer <docboobenstein@...>
Date: 2005-03-08T04:08:56+09:00
Subject: Text chunking?

Greetings! I am working on a program to extract sentences from e-mails
for parsing, and was wondering if I could get some input before I
ended up taking a brute force approach. I got as far as recursing the
mail directories and using rmail to extract the body from each
e-mail... So the last part that I would need to do is split each body
into individual sentences. The naive approach would be to split at
every '.', but I'd like to know if there was something smarter
available - I took a quick glance at RAA and the gems package list and
couldn't find anything relevant, though it is not impossible that I
might have missed something. Comments?

For the curious ones - below is a snippet of my current program
(Disclaimer: It was a 5-minute job - I apologize for any horrors in
in):

------8<----------
require "rubygems"
require_gem "rmail"

class Chunker
    attr_accessor :path

    def initialize( path = "~/tmp/enron/maildir" )
        $path = File.expand_path( path )
    end

    def recurse
        # puts "Recursing #{$path}"
        Dir["#{$path}/**/*"].each do |entry|
            # skip spamassassin files
            chunk( entry ) unless entry =~ /.SA$/
        end
    end

    def strip_headers( source )
        file    =  File.open( source )
        message = RMail::Parser.read( file )
        file.close
        message
    end
    def strip_headers( source )
        file    =  File.open( source )
        message = RMail::Parser.read( file )
        file.close
        message
    end

    def chunk( source )
        message = strip_headers( source )
        puts message.body
    end
end

chunker = Chunker.new( "~/tmp/enron/maildir/allen-p/straw" )
chunker.recurse

------8<----------
 
$ls ~/tmp/enron/maildir/allen-p/straw 
produces the following:
.  ..  1.  2.  2..SA  3.  4.  5.  6.  7.  8.