From: "Jesús Gabriel y Galán" <jgabrielygalan@...>
Date: 2010-09-04T23:55:43+09:00
Subject: Re: Using Nokogiri to scrape multiple websites

On Sat, Sep 4, 2010 at 4:24 PM, Ryan Mckenzie <ryan@souliss.com> wrote:
> Hi everyone,
>
> I'm looking to use Nokogiri to scrape about 10 websites for their anchor
> texts and output them on screen. What would be the best way to achieve
> this?
>
> I have tried doing something like this without much luck...
>
> def index
>  sites = Array.new("site1.com","site2.com","site3.com")
>  sites.each do |site|
>    @textlinks << scrape(site)
>  end
> end
>
> def scrape(website)
>  require 'open-uri'
>  require 'nokogiri'
>
>    doc = Nokogiri::HTML(open(website))
>  return doc.xpath('//a')
>  end

What is exactly the problem? You need to write the full url starting
with http:// so that open-uri works correctly. After that, scrape will
return an array of nokogiri elements each of them representing a link.
You are then putting each of these arrays into another array called
@textlinks. In orther to output the links to the screen, take a look
at the to_html method of Nokogiri::XML::Element.
This worked for me:

sites = %w{http://www.google.com http://www.yahoo.com}
links = []
sites.each {|site| links.concat(scrape(site))} # the scrape method is
the one you wrote above
links.each {|link| puts link.to_html}

Hope this helps,

Jesus.