From: Xeno Campanoli <xeno@...>
Date: 2005-06-13T15:19:22+09:00
Subject: Re: Questions about web crawling in ruby

Eric Hodel wrote:

>
> I have found a good deal of success with http-access2, but I still  
> had to do the things you don't want to do (external error handlers).   
> I don't think there is a good generic way of handling that.

What do you do about Javascript?  My pages, for instance, are 
Javascript.  It seems like in this day and age one could get a package 
to interpret that too, as well as those robots.txt files.

xc

-- 
Xeno Campanoli, xeno@eskimo.com, http://www.eskimo.com/~xeno
Pride before justice      equals destabilization.
Power before truth        equals destruction.
Profit before environment equals death.