From: Chris Gallagher <cgallagher@...>
Date: 2006-11-21T04:04:39+09:00
Subject: Re: Ruby screen scraping

Turns out I actually ended up abandonning HTree and the rest. I used 
net/http in order to fetch the page and then took the table of the page 
that I was interested in examining and converted that using rexml. I 
have now been able to grab the values that I wanted using XPath :)

require 'net/http'
require 'uri'
require 'rexml/document'
include REXML
def fetch(uri_str, limit=10)
  fail 'http redirect too deep' if limit.zero?
  puts "Trying: #{uri_str}"
  response = Net:: HTTP.get_response(URI.parse(uri_str))
  case response
  when Net::HTTPSuccess
    response
  when NetHTTPRedirection
    fetch(response['location'], limit-1)
  else
    response.error!
  end
end

response = fetch('http://10.37.150.55:8080')


scraped_data = response.body

table_start_pos = scraped_data.index('<table class="index" 
width="100%">')
#puts table_start_pos

table_end_pos = scraped_data.index('</table>') + 9
#puts table_end_pos

height = table_end_pos - table_start_pos

gathered_data = response.body[table_start_pos,height]

converted_data = REXML::Document.new gathered_data
#puts converted_data


module_name = XPath.first(converted_data, "//td[@class='data']/a/]")
puts module_name

build_status = XPath.first (converted_data, "//td[2]/em")
puts  build_status.text

last_failure = XPath.first(converted_data, "//tbody/tr/td[3]")
puts last_failure.text

last_success = XPath.first(converted_data, "//tbody/tr/td[4]")
puts  last_success.text

build_number = XPath.first(converted_data, "//tbody/tr/td[5]")
puts  build_number.text

-- 
Posted via http://www.ruby-forum.com/.