From: Bill Guindon <agorilla@...>
Date: 2005-06-30T11:13:55+09:00
Subject: Re: spidering a website to build a sitemap

On 6/29/05, Gene Tani <gene.tani@gmail.com> wrote:
> i noticed webfetcher in RPAbase, haven't had a chance to play with it:

Should've thought to scan RPA.  Wish it was still being updated, I
sure do miss it.

> http://www.acc.umu.se/~r2d2/programming/ruby/webfetcher/webfetcher.html

Gave it a couple test drives, and it's quite nice.  The following gave
me exactly what I was looking for.

require 'webfetcher'

page = WebFetcher::Page.url('http://www.somedomain.com/')
links = page.recurse.links
File.open('links.txt', 'w+') {|f| f.puts links.uniq}

Thanks much for tracking it down.

-- 
Bill Guindon (aka aGorilla)