From: Alexandru Popescu <the.mindstorm.mailinglist@...>
Date: 2005-09-17T04:15:20+09:00
Subject: Re: A big project

#: Berger, Daniel changed the world a bit at a time by saying on  9/16/2005 8:46 PM :#
>> -----Original Message-----
>> From: Alexandru Popescu [mailto:the.mindstorm.mailinglist@gmail.com] 
>> Sent: Friday, September 16, 2005 12:24 PM
>> To: ruby-talk ML
>> Subject: Re: A big project
>> 
>> 
>> #: Eric Hodel changed the world a bit at a time by saying on  
>> 9/16/2005 8:13 PM :#
>> > On 15 Sep 2005, at 12:04, David Heinemeier Hansson wrote:
>> > 
>> >> http://www.43things.com/
>> >> http://www.43places.com/
>> >>
>> >> The sites all fall within the category of your references 
>> and they're 
>> >> made with Ruby on Rails. Many of them do more than 400K 
>> pageviews per 
>> >> day. I know at least one of them is nearing one million.
>> > 
>> > Our sites (combined) handled over 1 million requests due to 
>> a naughty
>> > spider for a couple days.
>> > 
>> 
>> Have you solved it? I would be very interested to find out 
>> how. Can you post some hints how can you 
>> protect against these spiders?
> 
> Polite spiders should obey a robots.txt file.  The bad ones, well, you
> send the owner a nasty email and/or do your best to block them.

He, about this 3rd solution I was asking, cause it doesn't seem to be a trivial one ;-).

./alex
--
[.the_mindstorm.]

> 
> Regards,
> 
> Dan
> 
>