From: Belorion <belorion@...>
Date: 2005-06-30T05:07:39+09:00
Subject: Re: spidering a website to build a sitemap

I'll throw my little snippet in, in case anyone finds it useful.

I just wrote this up to spider my rails app to give me a list of all
the urls so I can use them later in a stress test.

Not terribly advanced, but gives you the format of:

http://www.blah.com/foo.html
{tab} http://www.blah.com/bar.html

Where tabbed out children of the foo.html are pages foo.html points to.

http://snippets.textdrive.com/posts/show/74

-Matt