From: Justin Bailey <jgbailey@...>
Date: 2006-07-06T00:50:25+09:00
Subject: Re: [SOC] progress on "Automated Wrapper Generation for Information Extraction"

------=_Part_741_21796150.1152114622518
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

On 7/4/06, A. S. Bradbury <asbradbury@tekcentral.org> wrote:
>
> As you may or may not be aware, I am currently working on one of the 10
> Google Summer of Code projects[1] mentored by Ruby Central. I am being
> mentored by Austin Ziegler.


Thanks for the update - sounds pretty interesting. Some thoughts below:


To
> understand my progress, it's probably easiest if I explain something about
> how the system works. A document's structure might be defined in the
> following way: doc_tree = Ariel::StructureNode.new do |r|
>   r.title
>   r.timestamp
>   r.author
>   r.post_body
>   r.comment_list do |c|
>     c.comment_author
>     c.comment_title
>     c.comment_body
>     c.comment_date
>   end
> end


I like the interface,  and the "humane" access it gives to the structure of
the page. It appears to handle single items and lists well.

At the moment you can use Ariel to describe the structure of a document, and
> to generate a start_rule or an end_rule. I am making good progress on the
> code
> to use labeled examples to learn start and end rules for a whole document
> tree.


Will I be able to point Ariel at a set of documents, and have it spit out a
reusable class which I can include in another program? For example, I have a
Bible reference parser (i.e. things like Gen 1:1, etc.)  that scrapes web
pages to get the actual verses. Right now I use hand-built regular
expressions and some patterns to get the right page for a given book,
chapter and verse. Could I use Ariel to generate the "lookup" code instead?

Another very small program I wrote scrapes calculator results from Google (
e.g. enter "2 + 2" into the google search box and get back "4"). Here again
I used StringScanner and some regular expressions to get the result and
transform it slightly. Would Ariel be able to help me with that?

== Questions for the Ruby community ==
> 1. What form would you like extracted data to take?
> YAML and XML output shouldn't be a problem, but I'm thinking about the
> outputted Ruby data structure. Supposing that the doc_tree defined above
> were applied to a document, the extracted structure could be queried like:
> p root.title.extracted_text
> p root.date.year.extracted_text
> p root.comment_list[3].author.extracted_text
> root.children would produce an array of the title object, author, and so
> on. root.comment_list.children[3] == root.comment_list[3]. Any ideas?


You could take a page from the Rails "pluralize" methods and also offer:

  root.comments[3] == root.comment_list.children[3] == root.comment_list[3]

  'root.comments_list' would also work.

Of the chioces you show, though, I like the less verbose
"root.comment_list[3]"
- though if comment_list is more than just a simple array the interface
could get tricky.

2. How should a document be labeled?
> In order to feed the learner, you must save a copy of the type of document
> you
> want to extract information from, and then mark up the information you
> want
> extracted. What markers would be appropriate?
> Something such as <l:comment_list>....</l:comment_list> is a possibility.


Have you heard of microformats? Essentially, its a way to markup existing
HTML pages with added attributes to indicate structure.Its more less
intrusive than adding new tags, etc. You can read about them here

  http://microformats.org/about/

And see what Yahoo is doing with them on their local results pages:

http://ylocalblog.com/blog/2006/06/21/we-now-support-microformats/

3. Which is better?
> (a). doc_tree = Ariel::StructureNode.new {|r| r.comment_list}
> (b). doc_tree = Ariel::StructureNode.new {|r| r.comments :list}
> (c) doc_tree = Ariel::StructureNode.new {|r| r.list :comments}
> It's certainly possible for (a) and (b) to both be allowed.


I prefer (a), though (c) seems intriguing. It sort of implies the interface
for Ariel objects is always the same (get a single item, get a list, etc)
and you just pass different symbols in. That could make it harder to
introspect against, though. For any of these interfaces you should strive to
make sure actual methods are implemented and its not just a lot of
method_missing tricks. Actual implementations are a lot easier to deal with
when meta-programming than hacking around method_missing logic.

Thanks again for the update, hope this feedback helps!

Justin

------=_Part_741_21796150.1152114622518--