From: Glenn Gillen <glenn.gillen@...>
Date: 2007-09-20T18:33:47+09:00
Subject: Re: Screen Scraping Advice

> I work for Cisco Systems in San Jose Ca. I proposed a project to  
> perform
> a screen scrape/spider hack to go out and look for websites with the
> Cisco name in its domain name (ex. usedcisco.com, ciscoequipment.com,
> etc.) and see if those companies are selling Cisco equipment. I  
> want to
> look for specific products (ex. WIC-1T, NM-4E, WS-2950-24) on these
> websites and see if they are being sold for under 60% of their  
> MSRP. We
> are trying to track down companies that are selling counterfeit
> equipment. So I started by downloading the DNS list of all domain  
> names
> so I could read through that and extract all domain names with  
> Cisco in
> it. Once I do that I want to go to each page and search/scrape for  
> these
> products, but I don't really know the best approach to take. Can  
> anyone
> give me advice? Should I just do keyword searches for those 20+
> products? Or is there a better approach?

I'm slightly biased, but scrubyt should be able to do most of the  
remaining heavy lifting for you

http://scrubyt.org/

Glenn