From: Peter Szinek <peter@...>
Date: 2006-05-15T17:21:27+09:00
Subject: Simple crawling/HTML extraction

Hello all,

I would like to accomplish the following task:

1) I have a bunch of links. Visit every link one by one
2) For every visited link, extract a string with a regexp from the page

    2) can be recursive: If one/more of the extracted string(s) is
    a link, visit it, extract strings  etc. etc. - of course this should
    stop after a specified level to prevent endless recursion

3) At the end, merge every extracted string into one list

The question is: do i have to do everything by hand or are there some 
higher level APIs for such stuff? What should i look at into (if i would 
not like to reinvent the wheel)?

Cheers,
Peter