From: Michael Neumann <mneumann@...>
Date: 2007-08-30T23:40:59+09:00
Subject: Re: Getting all google results with hpricot and connecting two gsub statements to just one?

On Wed, 29 Aug 2007 16:10:06 +0200, Gregory Seidman  
<gsslist+ruby@anthropohedron.net> wrote:

> On Wed, Aug 29, 2007 at 09:45:04PM +0900, kazaam wrote:
>> I'm trying to fetch all google results with hpricot. For the first page
>> of results I wrote this here:
>>
>> 	#!/usr/bin/env ruby
>> 	$Verbose=true
>>
>> 	require 'hpricot'
>> 	require 'open-uri'
>>
>> 	google =  
>> Hpricot(open("http://www.google.com/search?name=f&hl=en&q=#{$*}"))
>> 	(google/"h2.r/a").each {|line| puts  
>> line.to_s.gsub(/^.+href="/,'').gsub(/" .+$/,'')}

This doesn't work?

(google/"h2.r/a").each {|line| puts  line['href']}

>> So my first question is can I connect the both gsub statments above in
>> just one gsub which should increase the speed? Or is there even a better
>> way than using gsub for cleaning the results?
>>
>> And the next question is: how can I get all results not just from the
>> first page?

Can't you teach Google to show more results on a page?
If not, extract the "next page" link and fetch that page. Proceed as long
as long want :)

> Look into mechanize or scrubyt for this. They sit on top of hpricot, but
> are much better suited to screen scraping applications than hpricot  
> alone.

Nope. For this purpose, Hpricot is the best you can use!!! And it is  
really great!
Mechanize doesn't give you much advantage if you just want to parse pages
(and I say that as the author who wrong the initial version of Mechanize  
:).

scrubyt never worked on my machine, but I also find it too complicated or
maybe I am just too stupid :).

Regards,

   Michael