From: Gene Tani <gene.tani@...>
Date: 2005-12-07T06:02:42+09:00
Subject: Re: Screen scraping an html text contents into a file


basi wrote:
> Hello,
>
> I'm looking for a screen scraper that will extract text contents off
> html pages and save the text into files. I have looked at Mechanize and
> Rubyful_Soup, but they are a bit over my head to modify to save just
> the text contents to a file. (I'm a researcher trying to use Ruby for
> real world text analysis tasks, and trying to learn Ruby at the same
> time.)  The levels of usage I'd love to have (choosey beggar):
>
> > program prompts me for url address to scrape and file name to save texts into,
> > or edit program to enter url address and file name
>
> Of course a program that, given a url, would walk down the links, open
> the pages, and save the text contents to a file would be ... that would
> be a commercial product. Is there one?
>
> Thanks!
> basi

i think open-uri and Rubyful_soup are pretty straightforward.  I like
this shows open-uri vs. Net::HTTP:
http://www.zenspider.com/dl/rubyconf2005/open-uri.pdf

There's commercial website downloaders that will follow every link in
every page, hit the server hundreds of times in a few seconds and get
your IP blacklisted pretty quickly (so run them from Starbuck's
wireless).  Look in Oreilly Spidering Hacks, for the right way to do
it. the (perl) examples are straightforward.