From: Rob Biedenharn <Rob@...>
Date: 2008-08-20T01:19:05+09:00
Subject: Re: grabbing html from pages that require login

You might be able to use httpclient for this.  It manages a session  
much like a browser might and can track cookies that the site might  
send.  I was able to use this for doing some pretty ugly manipulation  
of some pop-up windows that was clearly intended to be used only by a  
human with a browser.

   gem install httpclient

-Rob

On Aug 19, 2008, at 8:20 AM, Adam Akhtar wrote:

> Hi ive searched the forums already but couldnt come up with any  
> concrete
> information.
>
> Ive written a script that accesses certain pages on ebay and scapes
> information. Its using open-uri to grab html and just standard regexps
> to grab the data i want. When building the script I used ebay  
> webpages I
> saved to my harddisk. All worked fine.
>
> However when the script attmepts to access the actual online  
> webpages I
> get an "file or directory not found ..." error.
> What I think is happening is that ebay is automatically redirecting  
> the
> script to the user login. Even if im already logged in via firefox i
> guess the fact that the request is coming from something else  
> outside of
> firefox makes ebay suspect its from another location and thus for
> security reasons requests the user to reenter their detials. Im  
> assuming
> this is to do with cookies???
>
> So how do get round this issue? Is there a way to get avoid this login
> requirement or somehow providing my script with the info it needs to
> login and continue from there?
> -- 

Rob Biedenharn		http://agileconsultingllc.com
Rob@AgileConsultingLLC.com