From: Bill Guindon <agorilla@...>
Date: 2005-06-23T07:06:27+09:00
Subject: Re: spidering a website to build a sitemap

On 6/22/05, Ryan Leavengood <mrcode@netrox.net> wrote:
> Bill Guindon said:
> >
> > Grabbed it as a gem, trying a simple test.  Oddly enough, had to add
> > it's lib path to the LOAD_PATH to get rid of an error (uninitialized
> > constant WWW (NameError)).
> 
> Hmmm, I didn't have to do that. Do you have rubygems in your RUBYOPT?

Nope, guess it's time to add it.
 
> Mechanize does mess around with the LOAD_PATH itself because it uses new
> features from the Ruby v1.9 net libraries.
> 
> But for me it worked fine, as shown in the code below.
> 
> > Any docs available on this, or any public examples?  Does look like
> > it'll give me a good start.
> 
> Unfortunately the docs are a bit light at the moment. I learned a lot by
> reading the source though, which is well written. Once I get my web-site
> up I was going to write an article on Mechanize, but for now that doesn't
> help you much :)
> 
> It needs to be heavily refactored, but here is the prototype code I wrote
> to help me renew books at my city library's web-site:
 
[helpful code snipped]

Thanks, that gives me a better idea of what can be done with it.  

Now comes the fun part of parsing through relative urls, checking for
base href's, munging similar urls (ie: /some/file.html vs.
some/file.html both called from the root).  Should be interesting.

> > thanks much for the pointer.
> 
> No problem. Hope the above code helps too.
> 
> Ryan
 
-- 
Bill Guindon (aka aGorilla)