From: Aaron Patterson <aaron_patterson@...>
Date: 2006-05-16T03:17:00+09:00
Subject: Re: Simple crawling/HTML extraction

On Mon, May 15, 2006 at 05:21:27PM +0900, Peter Szinek wrote:
> Hello all,
> 
> I would like to accomplish the following task:
> 
> 1) I have a bunch of links. Visit every link one by one
> 2) For every visited link, extract a string with a regexp from the page
> 
>    2) can be recursive: If one/more of the extracted string(s) is
>    a link, visit it, extract strings  etc. etc. - of course this should
>    stop after a specified level to prevent endless recursion
> 
> 3) At the end, merge every extracted string into one list
> 
> The question is: do i have to do everything by hand or are there some 
> higher level APIs for such stuff? What should i look at into (if i would 
> not like to reinvent the wheel)?

I suggest checking out WWW::Mechanize:

http://rubyforge.org/projects/mechanize/
http://mechanize.rubyforge.org/

Or just 'gem install mechanize'

Make sure to check out the examples in the RDoc.

--Aaron