From: Peter Szinek <peter@...>
Date: 2006-11-29T16:54:24+09:00
Subject: Re: Simple screen scraper using scrAPI

doog wrote:
> I'm a Ruby novice. Does anyone have an example of a simple screen
> scraper in Ruby that uses scrAPI (and works on Mac OS X)?

Though I don't  seem to understand the intensity of the holy war Paul is 
leading against anything that is not hand-coded on the fly, this time I 
will have to agree with him: the request 'I would like to write a screen 
scraper in scrAPI (or Hpricot, or xxx)' is not always the right way. 
Screen scraping is can be very tedious and complex, and it really 
depends on the input page, the type of the actions you would like to 
perform (fetching the page is trivial? do you need to navigate? (i.e. 
fill forms, lick links) how complex is the parsing?) quality you would 
like to achieve, robustness (i.e. if the underlying page changes, the 
scraper should still perform well) and another 10k things. Some time ago 
I wrote a small article on this:

http://www.rubyrailways.com/data-extraction-for-web-20-screen-scraping-in-rubyrails/

It is a bit outdated now (I am planning to beef it up with FireWatir, 
Hpricot and other sections) but it can help you as a starting point.

Conclusion: it depends on the page and task at hand what/how should be 
used. I suggest that if you have a concrete problem, drop us a mail and 
we will figure out something.

Cheers,
Peter

__
http://www.rubyrailways.com