From: John Joyce <dangerwillrobinsondanger@...>
Date: 2007-09-18T03:35:46+09:00
Subject: Re: Screen Scraping Advice


On Sep 17, 2007, at 12:25 PM, Charles Pareto wrote:

> I work for Cisco Systems in San Jose Ca. I proposed a project to  
> perform
> a screen scrape/spider hack to go out and look for websites with the
> Cisco name in its domain name (ex. usedcisco.com, ciscoequipment.com,
> etc.) and see if those companies are selling Cisco equipment. I  
> want to
> look for specific products (ex. WIC-1T, NM-4E, WS-2950-24) on these
> websites and see if they are being sold for under 60% of their  
> MSRP. We
> are trying to track down companies that are selling counterfeit
> equipment. So I started by downloading the DNS list of all domain  
> names
> so I could read through that and extract all domain names with  
> Cisco in
> it. Once I do that I want to go to each page and search/scrape for  
> these
> products, but I don't really know the best approach to take. Can  
> anyone
> give me advice? Should I just do keyword searches for those 20+
> products? Or is there a better approach?
> -- 
> Posted via http://www.ruby-forum.com/.
>
Doesn't sound like much scraping, just searching text for a string.  
You could even do a lot of that work with Google.
but just download the file and search for a string. create a data  
file of your own that tells you what line you found the string.
Scraping is really for getting data from other sites, using the DOM  
structure they have to get (for example) the weather report.