From: Phillip Gawlowski <cmdjackryan@...>
Date: 2010-11-23T20:40:38+09:00
Subject: Re: nokogiri ip ban?

On Tue, Nov 23, 2010 at 12:19 PM, Luis G. <l17339@gmail.com> wrote:
> Hi there...
> But I still have the same problem: works in the beginning, but after a
> while stops working.
>
> I can just run the crawler in steps, to not do lots of calls to the
> website in the same moment, but is kinda boring... :)
>
> Any of you face the same problem? Any of you have a solution for this?

Run your crawler in steps (don't crawl the whole site, and only grab
what is new; that's what the Age header is for!), and respect
robots.txt.

Otherwise, well, you get what you deserve, if you hog a server's CPU
cycles and create a Denial of Service attack (nobody cares if it is by
accident or by design).
-- 
Phillip Gawlowski

Though the folk I have met,
(Ah, how soon!) they forget
When I've moved on to some other place,
There may be one or two,
When I've played and passed through,
Who'll remember my song or my face.