From: Peter Szinek <peter@...>
Date: 2006-04-17T06:49:57+09:00
Subject: Re: How extract data from a web site?

Ingo Weiss wrote:
> Hi,
> 
> I would like to use Ruby to read the content of a web site, and then
> extract certain data from it. The site is machine generated so the
> format doesnt' change, but unfortunately it is far from being valid
> XHTML or similar.
> 
> What would be the easiest way to get there? I guess I need some kind of
> HTML parser, or? How to I read a web site into Ruby in the first place?

BTW the cleanest method i have ever seen (and used/using, both at my
full-time job and for my PhD thesis software)  is unfortunately not
available in Ruby. For this reason i have to use Java (which i do not
like at all, mainly when compared to Ruby) at both places. There i am
using the following method:

JavaXPCOM is a Java wrapper for mozilla native XPCOM, written by Javier
Pedemonte.
W3CConnector (a piece of software i made) generates a wrapper around
JavaXPCOM, implementing W3C interfaces.

This way, using my W3CConnector, you have a full access to the mozilla
DOM through wrapper classes implementing W3C interfaces. (The
W3CConnector wrapping is optional, just this way i can use Xerces and
other packages talking to W3C DOM interfaces.)

Access to the full Mozilla DOM tree means you can write full XPath
queries to extract nodes, and XPath is a really powerful language for
such purposes.

So, the missing layer in Ruby is RbXPCOM. Although it exists, it is
quite unfinished and abandoned since 2001. Does somebody know something
about it?

My PhD is a next generation web extraction engine, which heavily relies
on Mozilla DOM which can not be acquired (AFAIK) in any other way than i
have described here (For example i am using the coordinates of rendered
elements etc).

So does anyone have any info whether RbXPCOM will be finished in the future?

Cheers,
Peter