From: Jeff Barczewski <jeff.barczewski@...>
Date: 2006-11-21T14:26:14+09:00
Subject: Re: Ruby screen scraping

------=_Part_45859_28201809.1164086770296
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

On 11/19/06, Chris Gallagher <cgallagher@gmail.com> wrote:
>
> Hi,
>
> I'm looking at creating a ruby script that will firstly access our
> cruise control page on localhost and examin the page to see the values
> on the page, so basically telling us if the build succeeded or failed.
>
> Does anyone have any opinions on what might be the best way to approach
> this task. Ive been looking at a number of different packages including
> Htree.
>
>

Chris,

There are many ways to accomplish this as others have pointed out. When I
approached a similar task three years ago, I was working in the java world
and would have loved to have some of the tools available for Ruby today.
However I believe that the technique I used has merit in some situations
today.

I was screen scraping realtor sites for data (to find the perfect house),
because I was dissatisfied with the searching and data mining capabilities
of the sites. I was mining multiple sites, so the technique had to be
flexible but also resilient because I did not control the source sites (and
they would often change their layout). My first attempt used xpath's to try
and get to data, however that was futile since developers would often change
the site's layout and even small changes would break the logic (ie. changing
nesting of tables, or adding styling around data).

After taking a step back and considering the situation from a fresh
perspective, I scrapped the idea of using xml style data location in
something that seemed too fluid, too fragile.

My second approach was much more resilient, I used simple regular
expressions to zoom in and find the data. After studying the source html I
was able to discover a way to easily get to any data for the sites I was
working on.

The basic approach was this:

1) I would use a regular expression to search into the html for something to
get me close to the data, something that seemed to be consistent and
unlikely to change. (a reference point)


2) I would then extract reasonable number of characters before and/or after
the reference point based on where the data is located. It is not necessary
to know exactly just gather a conservative amount beyond what you think you
need.

3) repeat with step 1 if needed, or use regular expression to extract the
data desired from this subsection of data extracted in step 2.

I wrapped these basic ideas in to a few simple methods to make it easy and
it turned out to be a very successful approach. I found it easy to add new
sites pretty easily and it turned out to be very robust technique that was
very forgiving to changing fluff html. It was pretty easy to find a
reference point in the html that was consistent, and once there the data was
close by, so I'd extract a healthy chunk and then it was pretty easy to
search in this smaller amount of data. Use logging for each step to help you
while you are fine tuning the approach. But once I switched over to this
approach, I never had to revisit the code once I set it up for a site, it
just worked. Low tech, simple, but suprisingly effective.

Of course after many months of daily operation mining the realtor sites, I
eventually found the perfect house and abandoned the code; it had served its
purpose well. So I don't have anything concrete to offer you (and it was in
Java), but if any of the other methods mentioned by the others don't quite
meet your needs or end up being too fragile, you might consider a variation
on this approach for your own data extraction. It is especially flexible for
scraping sites which tend to vary over time. In your case it sounds like you
have control over the source so many methods would work for you, however
don't forget that there may be some variation over time if you ever upgrade
(cruise control).

Hope it helps you or others that are pursuing this task!

Blessings,

Jeff Barczewski
MasterView project developer, http://masterview.org/
Inspired Horizons Training and Consultancy http://inspiredhorizons.com/

------=_Part_45859_28201809.1164086770296--