From: brabuhr@...
Date: 2007-09-20T05:24:46+09:00
Subject: Re: scraping web pages for cisco products

On 9/19/07, Chuck Dawit <chuckdawit@gmail.com> wrote:
> One idea I was given was to
> split the pages into ones with forms and those without forms. Those
> without forms probably wont have anything for sale so I can eliminate
> those. But then I really don't know how to handle after that.

Here's a naive implementation of binning by forms:

> cat sites
www.cnn.com
www.usedcisco.com
www.rubyforge.org
slashdot.org
technocrat.net
bk.com

> cat firstbin.rb
#!/usr/bin/env ruby

require 'rubygems'
require 'mechanize'

agent = WWW::Mechanize.new

sites = File.readlines("sites")
bin1 = []
bin2 = []
bin3 = []

sites.each do |site|
  site.chomp!

  page = agent.get "http://#{site}"
  forms = page.forms
  search_forms = forms.select{|f|
    (f.name   and f.name.match        /search/i) or
    (f.action and f.action.to_s.match /search/i)
  }

  if search_forms.size > 0
    bin1 << site
  elsif forms.size > 0
    bin2 << site
  else
    bin3 << site
  end
end

p bin1
p bin2
p bin3

> ruby firstbin.rb
["www.cnn.com", "www.rubyforge.org", "slashdot.org"]
["www.usedcisco.com", "technocrat.net"]
["bk.com"]