From: Chris Gallagher <cgallagher@...>
Date: 2006-11-20T01:40:40+09:00
Subject: Re: Ruby screen scraping

thanks guys I'll look into both of them.

Another question I would have is how would i then get this scraped info 
to insert into a mysql database called say "build" and a table called 
"results".

For now if you could base answers on the following htree code?

require 'open-uri'
require 'htree'
require 'rexml/document'

url = "http://www.google.com/search?q=ruby"
open(url) {
 |page| page_content = page.read()
 doc = HTree(page_content).to_rexml
 doc.root.each_element('//a[@class="l"]') {
        |elem| puts elem.attribute('href').value }
}

which is returning a result of:

C:\>ruby script2.rb
http://www.ruby-lang.org/
http://www.ruby-lang.org/en/20020101.html
http://www.rubyonrails.org/
http://www.rubycentral.com/
http://www.rubycentral.com/book/
http://en.wikipedia.org/wiki/Ruby_programming_language
http://en.wikipedia.org/wiki/Ruby
http://www.w3.org/TR/ruby/
http://poignantguide.net/
http://www.zenspider.com/Languages/Ruby/QuickRef.html

Cheers.

-- 
Posted via http://www.ruby-forum.com/.