From: Phlip <phlip2005@...>
Date: 2009-04-11T14:20:05+09:00
Subject: Re: scrape data from Javascript inside HTML source

Yaxm Yaxm wrote:

> is there any scraper that interact with Javascript function inside a
> HTML page?  Sometimes, the data is returned from a  Javascript function
> or a javascript variable.  so I wonder if there's a easy way to get data
> out by evaluating and evaluating the javascript based on the context of
> the page in Ruby?

In test? Or "scraping" a target website to see what it's got?

Either way, I would use Nokogiri to rip the HTML and find <script> tags, then 
use racc and rkelly to interpret the JavaScript and find its variables.

By "would" I mean I already _do_ that. Here's the rkelly calls required:

       RKelly.parse(js).pointcut('TargetMethod()').  # with the ()
           matches.each do |updater|
         updater.grep(RKelly::Nodes::ArgumentsNode).each do |thang|
           p thang
         end
       end

However, if you are attacking other peoples' websites to scrape out their data, 
you might instead try Watir. It just runs a web browser and evaluates its JS 
directly.

-- 
   Phlip