From: Peter Szinek <peter@...>
Date: 2006-04-17T06:32:05+09:00
Subject: Re: How extract data from a web site?

Ingo Weiss wrote:
> Hi,
> 
> I would like to use Ruby to read the content of a web site, and then
> extract certain data from it. The site is machine generated so the
> format doesnt' change, but unfortunately it is far from being valid
> XHTML or similar.
In order to parse the page, first you need to push it through  some kind
of tidy-up engine, so you can turn invalid to HTML to XML. I recommend
this one:

http://tidy.rubyforge.org/

After this step you have reduced the problem of arbitrary (possibly
invalid) HTML parsing to XML parsing which is definitely easier, e.g.
with REXML.

> What would be the easiest way to get there? I guess I need some kind of
> HTML parser, or? How to I read a web site into Ruby in the first place?
Another possibility would be Rubyful soup:

http://www.crummy.com/software/RubyfulSoup/

You do not need pre-tidying here, just 'use it'.  Examples:

soup = BeautifulSoup.new(page)

# find all <p>'s:
  soup.find_all('p')

# find all tags that have an attribute align="left"
  soup.find_all { |tag| tag['align'] = "left" }

You got the idea.

another possibility (never tried but looks good):

http://rubyforge.org/projects/ruby-htmltools/

There are certainly much more ways to do this, but i think these should
be enough to get you started.

HTH,
Peter