From: Bill Guindon <agorilla@...>
Date: 2005-06-30T05:45:50+09:00
Subject: Re: spidering a website to build a sitemap

On 6/29/05, Belorion <belorion@gmail.com> wrote:
> I'll throw my little snippet in, in case anyone finds it useful.
> 
> I just wrote this up to spider my rails app to give me a list of all
> the urls so I can use them later in a stress test.
> 
> Not terribly advanced, but gives you the format of:
> 
> http://www.blah.com/foo.html
> {tab} http://www.blah.com/bar.html
> 
> Where tabbed out children of the foo.html are pages foo.html points to.
> 
> http://snippets.textdrive.com/posts/show/74

Good stuff!  It's missing a couple of features for stock sites
(handling javascript:, mailto:, #name links etc.), but those can
easily be added.

Thanks much for posting it.
 
> -Matt
> 
> 


-- 
Bill Guindon (aka aGorilla)