From: Kyle Schmitt <kyleaschmitt@...>
Date: 2008-12-01T11:39:56+09:00
Subject: Re: Anyone scraping dynamic AJAX sites?

On Sat, Nov 29, 2008 at 6:25 PM, Becca Girl <cschall@yahoo.com> wrote:
> Hello.
>
> Is there anyone who has successfully found a way to scrape a dynamically
> generated AJAX web site?  If I view the source, it gives me the
> variables.  If I use Firebug to view the DOM, it gives me the actual
> values.  Any ideas?
>
> Thanks.
> --
> Posted via http://www.ruby-forum.com/.

As gf pointed out, the problem is you need a full DOM and working
javascript for this, sometimes even working css, to really do it
properly, you need a full blown, fully supported, web browser.

Short story, use the WATIR library to interact with your browser's DOM
to do this.

http://wtr.rubyforge.org/

I used to do this all the time for work, in a testing capacity.  I
tried a number of diferent solutions, and found WATIR far superior to
anything else out there, including the very pricey pay packages.  If
you cut through all the marketing BS, half the pay-packages are
functional the same as WATIR, and the other half are more primitive.

--Kyle