From: Daniel Finnie <dan@...>
Date: 2008-12-21T08:14:49+09:00
Subject: Re: Anyone scraping dynamic AJAX sites?

------=_Part_41180_11517434.1229815384225
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

If the site is truely AJAX, i.e. the data is loaded from an HTTP call from
JavaScript, you could monitor the HTTP requests made by the browser.  On
Firefox, I use the LiveHTTPHeaders extension.  Just load go
view-->sidebar-->HTTP Headers, load the page with whatever data, and look
thru the requests for anything interesting.

I used this method to get Facebook contact info and it worked fairly well.
As a bonus, any data found with this method is usually in a very
machine-understandable format like JSON or RSS.  There are Ruby libraries
for both.

Dan


On Sat, Nov 29, 2008 at 7:25 PM, Becca Girl <cschall@yahoo.com> wrote:

> Hello.
>
> Is there anyone who has successfully found a way to scrape a dynamically
> generated AJAX web site?  If I view the source, it gives me the
> variables.  If I use Firebug to view the DOM, it gives me the actual
> values.  Any ideas?
>
> Thanks.
> --
> Posted via http://www.ruby-forum.com/.
>
>

------=_Part_41180_11517434.1229815384225--