From: Alex Stahl <astahl@...5.com>
Date: 2010-12-04T05:30:16+09:00
Subject: Re: Screen scraping an aspx site with Mechanize

--=-GFPEQ5KOuqQA7dogBc4n
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit

Sorry, I haven't looked too closely at the site you're scraping and had
assumed the button was wrapped by a link.  But upon closer inspection
that doesn't appear to be the case.  Looking at the page source, the
button is a form input element which doesn't actually cause a request to
be sent.  In this case, as I noted in my first email, you will in fact
need to generate a click event.  Unfortunately, that's not really what
nokogiri is for.  Since you need that specific event to fire, as was
previously recommended, watir or selenium are the more appropriate
tools.  

Another option would be to use wireshark to sniff for any requests which
are sent, and then try to reconstruct and send those requests via
mechanize.  But this would be a little more complex than just using the
right tools.

Of course, I just checked the mechanize docs again... and there is a
#click_button method in the form object, so that could be a solution as
well.  (http://mechanize.rubyforge.org/mechanize/Mechanize/Form.html)


________________________________________________________________________

Alex Stahl | Sr. Quality Engineer | hi5 Networks, Inc. | astahl@hi5.com
|

On Fri, 2010-12-03 at 05:22 -0600, Sofie Willander wrote:

> Alex Stahl wrote in post #965922:
> > Based on the xpath in the error at the link, you're not extracting a URL
> > - you're getting an HTML object (or, more specifically, an XML
> > node/nodeset).  Instead, what you want is the  "href" property of the
> > <a> tag located at the xpath.  (In the below example, '//path/to' would
> > be the unique HTML element(s) which is/are the parent of the anchor
> > tag).  Access the property like so:
> >
> > link = page.xpath("//path/to/a/@href").to_s
> > p link
> 
> Now I'm even more confused.. Have you got any examples to show me?
> How do I find the href for a button?
> 

--=-GFPEQ5KOuqQA7dogBc4n--