From: Alex Stahl <astahl@...5.com>
Date: 2010-12-03T18:05:52+09:00
Subject: Re: Screen scraping an aspx site with Mechanize

--=-eBp4umhLbNL/cl1JBJiH
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit

Glad I could help... a few things to know:

-Mechanize throws an exception on any response which is not an HTTP 200
or 302.  So the error you're receiving, HTTP 400, is not handled by
mechanize and needs to be by your client.
-#get takes a URL as its parameter, so link should be a URL string.
(Actually, there's more than one way to pass the URL - check the
following link if that's not what you want:
http://mechanize.rubyforge.org/mechanize/Mechanize.html#M000231)
-Starting an xpath with "//*" causes the parser to look at *every*
element until it finds one which has the @id you supplied.  Better to
replace "*" with the actual HTML element.

Based on the xpath in the error at the link, you're not extracting a URL
- you're getting an HTML object (or, more specifically, an XML
node/nodeset).  Instead, what you want is the  "href" property of the
<a> tag located at the xpath.  (In the below example, '//path/to' would
be the unique HTML element(s) which is/are the parent of the anchor
tag).  Access the property like so:

link = page.xpath("//path/to/a/@href").to_s
p link

It's also helpful to output the link prior to using it as a param to
#get to see what you'll ask for.


________________________________________________________________________

Alex Stahl | Sr. Quality Engineer | hi5 Networks, Inc. | astahl@hi5.com
| m: 415.710.6961 

On Fri, 2010-12-03 at 02:27 -0600, Sofie Willander wrote:

> Thank you for your reply! I haven't gotten it to work yet though. I get
> an error on the following:
> 
> Alex Stahl wrote in post #965773:
> > resp = req.get(link)
> 
> The error read:
> Mechanize::ResponseCodeError: 400 => Net::HTTPBadRequest
>         from
> /usr/local/lib/ruby/gems/1.8/gems/mechanize-1.0.0/lib/mechanize.rb:259:in
> `get'
>         from (irb):8
>         from :0
> 
> It seems to find the button (I assumed that the xpath to the link was
> actually the button's xpath. Correct?). I get the button as an object,
> can I use req.get() on it? What am I not doing correctly?
> 
> I've attached a textfile with the output of the two last commands. I
> would be so glad if you could help me once again.
> 
> Attachments:
> http://www.ruby-forum.com/attachment/5509/HTTPBadRequest.txt
> 
> 

--=-eBp4umhLbNL/cl1JBJiH--