From: Brad Phelan <phelan@...>
Date: 2007-09-20T16:30:05+09:00
Subject: Re: scraping web pages for cisco products

Chuck Dawit wrote:
> unknown wrote:
>>> With this method do I need to know the name of the form to use it? With
>>> mechanize I thought you had to look at the form name first before you
>>> could use it?
>> It helps to know someway to distinguish the form you're looking for
>> from the other forms on the page.  It would be possible to iterate
>> through all the forms on a page, entering some text into the text
>> fields in the form and submitting them; but, most of the time the
>> script would probably be in either the wrong form or the wrong field
>> in the right form (and, of course, there are other issues, e.g. forms
>> that require multiple fields to be edited).  I don't see anyway to
>> avoid customizing the code for each site (though, if you get a good
>> framework built the effort per site should decrease?).
> 
> I agree but I have around 2000 sites to look at and I can't look at each 
> and every form, that would take way to long. Do you think a better 
> approach would be to use a search engines API to search for the products 
> on each site? I've never used any search engine API, if I know the 
> website name and the product name and a price I want can I use those 
> parameters in the search to find results?


This query seems to work

site:solecentral.com.au OR site:xtargets.com  AND crocs

I advertise my brothers e-commerce site on my site and they both contain 
the same keyword "crocs". Google returns all the pages from my site and 
his site that contain the word "crocs". However I am not sure how high 
the query scales as I think Google truncates the search string after 
some length so adding in 2000 sites for the query string might break.

Not sure if the same query trick also works in froogle as well as
vanilla google.

Hope this is somewhat helpful.

--
Brad
http://xtargets.com