From: Peter Szinek <peter@...>
Date: 2008-12-01T18:48:03+09:00
Subject: Re: Anyone scraping dynamic AJAX sites?

--Apple-Mail-2-709906065
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Just for completeness sake: scRUBYt! (since 0.4.05) is using FireWatir  
as the agent (or mechanize - you can choose whether you want scrape  
AJAX or not) so you can do full blown AJAX scraping - but with a  
scraping DSL which usually speeds up the scraper creation, especially  
in the case of complicated scrapers.

Cheers,
Peter
___
http://www.rubyrailways.com
http://scrubyt.org

On 2008.12.01., at 3:39, Kyle Schmitt wrote:

> On Sat, Nov 29, 2008 at 6:25 PM, Becca Girl <cschall@yahoo.com> wrote:
>> Hello.
>>
>> Is there anyone who has successfully found a way to scrape a  
>> dynamically
>> generated AJAX web site?  If I view the source, it gives me the
>> variables.  If I use Firebug to view the DOM, it gives me the actual
>> values.  Any ideas?
>>
>> Thanks.
>> --
>> Posted via http://www.ruby-forum.com/.
>
> As gf pointed out, the problem is you need a full DOM and working
> javascript for this, sometimes even working css, to really do it
> properly, you need a full blown, fully supported, web browser.
>
> Short story, use the WATIR library to interact with your browser's DOM
> to do this.
>
> http://wtr.rubyforge.org/
>
> I used to do this all the time for work, in a testing capacity.  I
> tried a number of diferent solutions, and found WATIR far superior to
> anything else out there, including the very pricey pay packages.  If
> you cut through all the marketing BS, half the pay-packages are
> functional the same as WATIR, and the other half are more primitive.
>
> --Kyle
>


--Apple-Mail-2-709906065--