From: Aredridel <aredridel@...>
Date: 2003-06-13T01:16:50+09:00
Subject: Re: HTML -> list of sentences? (semi-impossible task)

On Wed, 2003-06-11 at 20:38, Hal E. Fulton wrote:
> Hello, all.
> 
> Here's an idea I'm toying with. Suggestions
> are welcome.
> 
> I want to take an HTML document (reasonably
> well-formed, but not guaranteed) and remove
> all the tags from it...
> 
> ...and get a list of the *sentences* in the
> document.
> 
> There are, of course, several things that make
> this difficult:
>   - need to distinguish between end-of-sentence
>     and embedded punctuation, including both 
>     abbreviations and textual references to 
>     Ruby methods such as eof? and split!
>   - need to treat sentence fragments as sentences
>   - need to ignore blocks of code
>   - etc.
> 
> My current approach is to start with htmlsplit
> from the RAA. This is fairly simplistic, but 
> at least it doesn't have any dependencies.
> 
> Not sure whether to do it in two steps or not:
> 1. Convert to text
> 2. Process

I would parse into a tree, process there, then strip tags.  The reason
being, ruby code and other nongramatical entities are likely to be
offset by tags -- <pre>, <code>, <tt>, things like that.  Not always,
but it's a useful heuristic.

It's not a trivial task -- I've done a lot of natural-language work for
the Wiki that I run (it's markup is one of the least code-like of any
wiki).  How good you need the results to be are a big deciding factor in
how to implement, for sure.  Natural language parsing is a big cpu
cruncher.

> 
> Might be just as easy to do it in one step if
> I knew what I was doing.
> 
> Also not sure what is the best tool/library for
> this job.
> 
> Comments welcome.
> 
> Hal
> 
> --
> Hal Fulton
> hal9000@hypermetrics.com
> 
>