From: brabuhr@...
Date: 2010-05-20T21:49:51+09:00
Subject: Re: Scrape javascript content

On Thu, May 20, 2010 at 1:48 AM, Phil Mcdonnell
<phil.a.mcdonnell@gmail.com> wrote:
> I'm trying to scrape a page that hides some data behind a javascript
> function.  Is there any way to get this data?  I've been using
> Mechanize, but I'm not sure it can do this.  Is there a better library
> to use for this type of thing?

http://celerity.rubyforge.org/
http://watir.com/

> The following is the interesting part of the page:
>
> <td class="colPlus" onclick="fireClick(this,0)">
>    <a id="iroc_0" class="plus" href="#" onclick="return
> false;">&nbsp;</a>
> </td>

The *really* interesting part is what does the Javascript do :-) with
(a potentially large) effort you may be able to "reverse-engineer" the
javascript and emulate manually in mechanize.  I.e. if the javascript
builds a simple HTTP request, you may be able to send the same request
from mechanize (possibly) without much effort.