From: "Berger, Daniel" <Daniel.Berger@...>
Date: 2005-09-17T03:46:54+09:00
Subject: Re: A big project

> -----Original Message-----
> From: Alexandru Popescu [mailto:the.mindstorm.mailinglist@gmail.com] 
> Sent: Friday, September 16, 2005 12:24 PM
> To: ruby-talk ML
> Subject: Re: A big project
> 
> 
> #: Eric Hodel changed the world a bit at a time by saying on  
> 9/16/2005 8:13 PM :#
> > On 15 Sep 2005, at 12:04, David Heinemeier Hansson wrote:
> > 
> >> http://www.43things.com/
> >> http://www.43places.com/
> >>
> >> The sites all fall within the category of your references 
> and they're 
> >> made with Ruby on Rails. Many of them do more than 400K 
> pageviews per 
> >> day. I know at least one of them is nearing one million.
> > 
> > Our sites (combined) handled over 1 million requests due to 
> a naughty
> > spider for a couple days.
> > 
> 
> Have you solved it? I would be very interested to find out 
> how. Can you post some hints how can you 
> protect against these spiders?

Polite spiders should obey a robots.txt file.  The bad ones, well, you
send the owner a nasty email and/or do your best to block them.

Regards,

Dan