From: Andreas Schwarz <usenet@...>
Date: 2004-08-21T23:00:51+09:00
Subject: Re: ruby-doc.org content snarfing

James Britt wrote:

> I received an alert E-mail today telling me that ruby-doc.org had 
> exceeded its alloted bandwidth.  There could be all sorts of reasons for 
> this, and if it were simply due to popularity I'd be thrilled.  But it 
> appears that someone has been running wget and snarfing the site wholesale.

You should look at mod_throttle and set up a few .htaccess rules against 
annoying web spiders:

SetEnvIf user-agent MSIECrawler keep_out
SetEnvIf user-agent ^Teleport keep_out
SetEnvIf user-agent ^WebStripper keep_out
SetEnvIf user-agent ^Offline keep_out
SetEnvIf user-agent HTTrack keep_out
SetEnvIf user-agent Xaldon keep_out
SetEnvIf user-agent WebCopier keep_out

<Limit GET POST >
order allow,deny
allow from all
deny from env=keep_out
</Limit>