From: franco Date: 2007-09-18T10:40:04+09:00 Subject: Re: Screen Scraping Advice On Sep 17, 1:25 pm, Charles Pareto wrote: > I work for Cisco Systems in San Jose Ca. I proposed a project to perform > a screen scrape/spider hack to go out and look for websites with the > Cisco name in its domain name (ex. usedcisco.com, ciscoequipment.com, > etc.) and see if those companies are selling Cisco equipment. I want to > look for specific products (ex. WIC-1T, NM-4E, WS-2950-24) on these > websites and see if they are being sold for under 60% of their MSRP. We > are trying to track down companies that are selling counterfeit > equipment. So I started by downloading the DNS list of all domain names > so I could read through that and extract all domain names with Cisco in > it. Once I do that I want to go to each page and search/scrape for these > products, but I don't really know the best approach to take. Can anyone > give me advice? Should I just do keyword searches for those 20+ > products? Or is there a better approach? > -- > Posted viahttp://www.ruby-forum.com/. Hpricot http://code.whytheluckystiff.net/hpricot/ is a great screen scrape library for ruby. scraping might not be the best approach because each site/page uses a different layout, therefore the same scrape recipe probably won't work for another page. you could scrape froogle (google products?) or some other aggregate consumer sales site. it will have one interface and probably a lot of data. you might want to see if there are web services for froogle, usually better than scraping.