From: "Jesús Gabriel y Galán" <jgabrielygalan@...>
Date: 2010-09-07T18:54:59+09:00
Subject: Re: Using Nokogiri to scrape multiple websites

On Tue, Sep 7, 2010 at 11:42 AM, Ryan Mckenzie <ryan@souliss.com> wrote:
> Jesús Gabriel y Galán wrote:
>> def index
>>   keywords = %w{accounts resources membership}
>>   sites = %w{http://www.google.com http://www.yahoo.com}
>>   links_by_site = Hash.new {|h,k| h[k] = {}}
>>   sites.each do |site|
>>     keywords.each do |keyword|
>>       links[site][keyword] = scrape(site, keyword)
>>     end
>>   end
>>   links
>> end
>>
>> def scrape(website,inputtext)
>>   require 'open-uri'  #these could maybe go at the start of the script
>>   require 'nokogiri'
>>
>>   regex = /#{inputtext}/
>>   links_that_match = []
>>   doc = Nokogiri::HTML(open(website))
>>   doc.xpath('//a').each do |link|
>>     if  regex =~ link.inner_text
>>      links_that_match << link.to_html
>>    end
>>   end
>>   links_that_match
>> end
>>
>> Untested, but it can give you some ideas. The resulting hash will have
>> something like:
>>
>> {"http://www.google.com" => {"accounts" => [<some links containing the
>> word accounts>], "resources" => [<idem for resources>]
>> ...
>> }
>>
>> Jesus.
>
> That works great! Thank you.
>
> Instead of having to pull the items from a hash though I would really
> like to try pull them from a database for when the list gets extremely
> large. I've tried using the hash to pull from a variable but it produces
> an error which says the hash is an odd length.

I don't understand what you mean here.

> It is only going to be a
> flat table database so all of the data will be called under
> @backlinks.title (the keyword(s)), @backlinks.permalink (for the site)
>
> def index
>  @links = Hash.new { |ha,lnk| ha[lnk] = {} }
>  @backlinks = Backlink.find(:all)
>  keywords = %w{@backlinks.concat(title)}
>  sites = %w{@backlinks.concat(permalink)}

irb(main):004:0> keywords = %w{@backlinks.concat(title)}
=> ["@backlinks.concat(title)"]

You probably mean:

keywords = @backlinks.map {|bl| bl.title}
sites = @backlinks.map {|bl| bl.permalink}

but I don't know exactly what @backlinks is (probably an ActiveRecord?)

>  links_by_site = Hash.new {|h,k| h[k] = {}}
>  sites.each do |site|
>    keywords.each do |keyword|
>      @links[site][keyword] = scrape(site, keyword)
>    end
>  end

This code produces the error: "hash is an odd number length"?

Jesus.