From: Eric Hodel <drbrain@...7.net>
Date: 2005-06-14T03:02:35+09:00
Subject: Re: Questions about web crawling in ruby

On 12 Jun 2005, at 23:19, Xeno Campanoli wrote:

>> I have found a good deal of success with http-access2, but I  
>> still  had to do the things you don't want to do (external error  
>> handlers).   I don't think there is a good generic way of handling  
>> that.
>
> What do you do about Javascript?  My pages, for instance, are  
> Javascript.  It seems like in this day and age one could get a  
> package to interpret that too, as well as those robots.txt files.

Google doesn't interpret JS.  It may read it by mistake, and some  
spiders grab things that look like URLs from JS, but too my  
knowledge, nobody crawls a site with JS links by interpreting the JS.

-- 
Eric Hodel - drbrain@segment7.net - http://segment7.net
FEC2 57F1 D465 EB15 5D6E  7C11 332A 551C 796C 9F04