From: brabuhr@...
Date: 2007-09-21T03:54:07+09:00
Subject: Re: Screen Scraping Advice

> > give me advice? Should I just do keyword searches for those 20+
> > products? Or is there a better approach?

On 9/20/07, Glenn Gillen <glenn.gillen@gmail.com> wrote:
> I'm slightly biased, but scrubyt should be able to do most of the
> remaining heavy lifting for you
>
> http://scrubyt.org/

On that note:

require "rubygems"
require "scrubyt"

froogle_data = Scrubyt::Extractor.define do
  fetch "http://www.google.com/products"
  fill_textfield "q", "WIC-1T"
  submit

  info do
    product "WIC-1T"
    vendor "NEW2U Hardware from ..."
    price "$40.00"
  end
  next_page "Next", :limit => 10
end

puts froogle_data.to_xml

(tons of improvement needed, but):

<root>
  <info>
    <product>WIC-1T</product>
    <vendor>NEW2U Hardware from ...</vendor>
    <price>$40.00</price>
  </info>
  <info>
    <product>WIC-1T</product>
    <vendor>ATS Computer Systems...</vendor>
    <price>$353.95</price>
  </info>
  <info>
    <product>WIC-1T</product>
    <vendor>eBay</vendor>
    <price>$49.95</price>
  </info>
  <info>
    <product>WIC-1T</product>
    <vendor>eBay</vendor>
    <price>$149.99</price>
  </info>
  <info>
    <product>WIC-1T</product>
    <vendor>PCsForEveryone.com</vendor>
    <price>$337.07</price>
  </info>
  <info>
    <product>WIC-1T</product>
    <vendor>COL - Computer Onlin...</vendor>
    <price>$149.00</price>
  </info>
  <info>
    <product>WIC-1T</product>
    <vendor>eCOST.com</vendor>
    <price>$297.14</price>
  </info>
  <info>
    <product>WIC-1T</product>
    <vendor>eBay</vendor>
    <price>$45.00</price>
  </info>
  <info>
    <product>WIC-1T</product>
    <vendor>ATACOM</vendor>
    <price>$291.95</price>
  </info>
  <info>
    <product>WIC-1T</product>
    <vendor>Express IT Options</vendor>
    <price>$216.44</price>
  </info>
</root>