From: Peter Szinek <peter@...>
Date: 2006-11-20T18:31:12+09:00
Subject: Re: Ruby screen scraping

> I agree completely (see my other post on this topic), but it appears the OP
> was trying to read machine-generated Web content, presumably with reliable
> syntax.

Then you are right of course. I guess the problem is in the definition
of the term 'screen scraping' ( or 'web extraction' or 'web mining' or
'html extraction' - people can not even agree on it's name ).

For me 'screen scraping' means a complex thing: navigating to the
document, parsing it into something meaningful and querying the objects
of the parsed structure. In general, I am assuming that neither of these
steps are trivial - maybe because I am working for a web extraction
company for years now and I have seen every kind of nice tricks of the
other side (a.k.a the anti-scrape camp)

Of course, if you define screen scraping as the last step only (i.e. you
have  a parsed model (e.g. a well formed page) and you need to query
that) - then of course regular expressions are always the first thing to
consider.

Since the OP was referring to a machine generated page, I think the
latter applies - so yep, as far as he need all <p>'s only, regular
expressions are probably the easiest thing to pull out.

Peter

__
http://www.rubyrailways.com