From: Peter Szinek Date: 2008-04-28T21:09:48+09:00 Subject: Re: help re recording/replaying (i.e. automating) HTTP interactions to a web-site??? --Apple-Mail-5--850847277 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit In the current official release (0.3.4), FireWatir is not yet added. So you have just Mechanize + Hpricot. Mechanize does the navigational part - fill this textfield, then click that button, and if you arrived at the result page, crawl to all the detail pages etc. Once you arrive at the final page from where you don't want to go on further, you start the actual scraping, and in this case that's done through Hpricot. You take the page where you arrived, parse it with Hpricot and collect the results from it. In the development release, it is possible to plug-in other agents than Mechanize - theoretically anything, currently FireWatir is implemented. But if you want to use Mechanize as the agent for crawling, you don't need to install FireWatir at all. FireWatir based scraping has other benefits beyond JS/AJAX - for example more robust HTML parsing (which is done by Firefox in this case). Hpricot is a great parser but it can't beat Hpricot (yet). Firefox-parsed HTML also means you can use XPaths straight from FireBug or DOM Inspector (which is not the case with Mechanize) On the downside, Mechanize-based navigation/scraping is faster (you don't have to wait until the page renders, which is a prerequisite for FireWater-based navigation etc.) Does this answer your question? (If not, be sure to keep asking ;-) Cheers, Peter ___ http://www.rubyrailways.com http://scrubyt.org On Apr 28, 2008, at 1:35 PM, Greg Hauptmann wrote: > I must admit you're managing to overwhelm me slightly with the > number of > libraries / packages here :) > So what does the stick look like you're talking about. Will it be > fronted > by scrubyt then like: > > - Scrubyt > - FireWatir > - Mechanize > - hpricot > > Is this correct? If it were simply could you put in brackets the > key thing > each layer does/focuses on? > > Cheers > Greg > > > On Mon, Apr 28, 2008 at 9:06 PM, Peter Szinek > wrote: > >> >> On Apr 28, 2008, at 12:50 PM, Greg Hauptmann wrote: >> >> thanks Peter - I was starting to look at Mechanize but will focus in >>> at scRUBYt... >>> 2008/4/28 Peter Szinek : >>> >> >> OK, cool. If you don't have JS/AJAX or other trick that Mechanize >> can't >> handle, you should be OK. >> >> On the other hand, if you do have JS/AJAX on the page, you will need >> FireWatir, whether you like it or not :-) the FireWatir-enabled >> version of >> scRUBYt! is not yet officially released - if you want to try it, >> you need >> to d/l it from http://scrubyt.org/scrubyt-0.4.03.gem and install. >> >> Let me know if you encounter any problems! >> >> >> Cheers, >> Peter >> ___ >> http://www.rubyrailways.com >> http://scrubyt.org >> >> --Apple-Mail-5--850847277--