From: Brian Denny <brian@...>
Date: 2002-10-23T12:18:42+09:00
Subject: [ANN] Crawler 0.0.1


Hi all,

I recently searched in vain for a web-crawling library in ruby. If
there is one out there already, i'd love to know about it.

In any case, I have written my own, and I hope it proves some use to
somebody besides me.

You can find Crawler at 
http://implementality.com/projects/crawler/

This is the first time I have announced a piece of software (however
small) to a public forum.  I am sure that Crawler has much room for
improvement. (Starting, perhaps, with its name?  Any suggestions?) So
please feel free to provide feedback, criticism, patches, etc.

For the impatient, here is how crawler works:

# instantiate crawler with a callback routine
crawler = Crawler.new do  | url, page_data |
              do_something_with_url()
              do_something_with_page_data()
          end

# crawl to depth 3, invoking the above callback for each page
crawler.crawl ('http://www.rubycentral.org/', 3)   


happy crawling,

-brian