From: Andrea Dallera <andrea@...>
Date: 2010-11-23T20:33:25+09:00
Subject: Re: nokogiri ip ban?

Bombing a webserver in the fashion you describe is not advisable in any 
way. They're clearly not happy with what you're doing... so either lower 
the frequency of the requests or ask them directly about your needs - 
they might be willing to let you run your script more often or even give 
you the raw data directly.
-- 

Andrea Dallera
http://github.com/bolthar/freightrain
http://usingimho.wordpress.com




Il 23/11/2010 12:19, Luis G. ha scritto:
> Hi there...
>
> I've been playing with Ruby and Nokogiri to crawl some website to get
> text information, but after a while I realize that some of those
> websites block my access while the script is running. Since the moment
> they block the access, the script keeps running (cause I handled the
> exception) but is getting what is suppose to.
> After the block, if I try to access using the browser, I just can't, so
> I guess they block the IP address, right?
>
> I also tried using TOR, like this:
>
> Nokogiri::HTML(open(url, :proxy =>  'http://(ip_address):(port)'))
>
> But I still have the same problem: works in the beginning, but after a
> while stops working.
>
> I can just run the crawler in steps, to not do lots of calls to the
> website in the same moment, but is kinda boring... :)
>
> Any of you face the same problem? Any of you have a solution for this?
>
> thanks,
>
> Luis
>