From: Robert Klemme <shortcutter@...>
Date: 2009-06-28T17:05:51+09:00
Subject: Re: Need Advice Help On Parsing A File

On 28.06.2009 02:26, Mrmaster Mrmaster wrote:
> I'm creating an application that will parse mbox files, extract the
> data, and put it into a db. I have a couple of problems. For those of
> you who are not familiar with mbox files, just think of one text file
> that stores all of the emails in text format.
> 
> 1) mbox files keep updating so how do notify my script that new data has
> come in? Do I rerun the script with a placeholder where it last
> finished? That would require me to rescan the whole mbox file to find
> the placeholder which is pretty bad design.

It would probably be more efficient to just store the file position (see 
IO#tell and IO#seek).  However, that does not work if the mbox file is 
not just appended to but also removed from.  And the file position 
approach also does not solve the problem of concurrent modifications.

I assume for mbox files there is an established protocol how the file 
locking works.  You'll probably find it somewhere on the web or someone 
more knowledgeable than me posts information here.

> 2) What is the most efficient way to read the emails into memory before
> putting it into a db? Since there are multiple emails in each mbox file
> will I just read one of the emails, store it into memory, dumb it into
> db, then replace the current email in memory with the new one?

Yes, processing one email at a time seems the most efficient approach, 
especially since there is no information about other mails that you need 
  to store a single email.

> ps. I realize there are scripts that already do this. I'm creating this
> for my own learning experience.

Have fun!

Kind regards

	robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/