From: Adam Akhtar <adamtemporary@...>
Date: 2008-08-19T21:20:33+09:00
Subject: grabbing html from pages that require login

Hi ive searched the forums already but couldnt come up with any concrete
information.

Ive written a script that accesses certain pages on ebay and scapes
information. Its using open-uri to grab html and just standard regexps
to grab the data i want. When building the script I used ebay webpages I
saved to my harddisk. All worked fine.

However when the script attmepts to access the actual online webpages I
get an "file or directory not found ..." error.
What I think is happening is that ebay is automatically redirecting the
script to the user login. Even if im already logged in via firefox i
guess the fact that the request is coming from something else outside of
firefox makes ebay suspect its from another location and thus for
security reasons requests the user to reenter their detials. Im assuming
this is to do with cookies???

So how do get round this issue? Is there a way to get avoid this login
requirement or somehow providing my script with the info it needs to
login and continue from there?
-- 
Posted via http://www.ruby-forum.com/.