From: James Britt <jamesUNDERBARb@...>
Date: 2004-08-21T15:48:40+09:00
Subject: Re: ruby-doc.org content snarfing

David Morton wrote:
> James Britt wrote:
> 
>> this, and if it were simply due to popularity I'd be thrilled.  But it 
>> appears that someone has been running wget and snarfing the site 
>> wholesale.
> 
> 
> Not sure if this is the problem, but I noticed that 
> http://www.ruby-doc.org/robots.txt doesn't exist. Is it getting hit by 
> search bots?

Crawlers and spiders and such are fine.  They hit the site often.

I've never seem this sort of downloading, though.  And the legit spiders
(i.e., the ones likely to respect a robots.txt file) tend to identify
themselves as such.  This one didn't.

That's not to say that a robots.txt file wouldn't be a bad thing, just
that spiders have never been a problem.


James