From: Robert Klemme <shortcutter@...>
Date: 2009-06-28T21:20:07+09:00
Subject: Re: Need Advice Help On Parsing A File

On 28.06.2009 12:13, Ryan Davis wrote:
> On Jun 27, 2009, at 17:26 , Mrmaster Mrmaster wrote:
> 
>> Hello,
>>
>> I'm creating an application that will parse mbox files, extract the
>> data, and put it into a db. I have a couple of problems. For those of
>> you who are not familiar with mbox files, just think of one text file
>> that stores all of the emails in text format.
>>
>> 1) mbox files keep updating so how do notify my script that new data  
>> has
>> come in? Do I rerun the script with a placeholder where it last
>> finished? That would require me to rescan the whole mbox file to find
>> the placeholder which is pretty bad design.
> 
> Is it bad design? What happens when the user deletes the first email  
> in the mbox?
> 
>> 2) What is the most efficient way to read the emails into memory  
>> before
>> putting it into a db? Since there are multiple emails in each mbox  
>> file
>> will I just read one of the emails, store it into memory, dumb it into
>> db, then replace the current email in memory with the new one?
> 
> efficient? why do you care about efficiency already? Get something  
> working first, worry about efficiency AFTER you've measured stuff, not  
> blindly guessed. Oh... and measure only once you have efficiency  
> issues, until then it is Fast Enough(tm) (cousin of Just Works(tm)).

I don't fully agree.  It does not hurt to waste a quick thought about 
efficiency here since you know already that mbox files can grow large. 
Someone archiving his complete email history in and never deleting 
anything from a single mbox file will reach memory limits sooner or 
later when reading the whole file into memory at once.  It may prove 
more efficient (developer time wise) to start with a solution that 
assumed to be more efficient right from the start (in this case, process 
a single message at a time because otherwise you might be stuck with a 
working solution that needs heavy refactoring leading to much higher 
efforts (with high likelyhood!) than doing the right thing initially.

Luckily, it is not too difficult.  You could do something like

MBox = Struct.new :io
   include Enumerable
   include Enumerator

   def each
     msg = nil

     io.each do |line|
       if /^From/ =~ line
         yield msg if msg
         msg = line
       else
         msg << line
       end
     end

     self
   end
end

And then

File.open "mbox" do |io|
   MBox.new(io).each do |message|
     # deal with one message at a time
   end
end

> Luckily in this case, one of the most efficient (time wise) is also  
> the cleanest (code wise):
> 
>    File.read(path) #=> contents

How do you know without profiling the concrete application? :-)

Kind regards

	robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/