From: Konrad Meyer <konrad@...>
Date: 2007-09-20T03:12:51+09:00
Subject: Re: scraping web pages for cisco products

--nextPart2034462.32gXqZMtcQ
Content-Type: text/plain;
  charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Quoth Glen Holcomb:
> On 9/19/07, Glen Holcomb <damnbigman@gmail.com> wrote:
> >
> > On 9/19/07, Chuck Dawit <chuckdawit@gmail.com> wrote:
> > >
> > >
> > >
> > > I submitted a post a few days ago about scraping the web for Cisco
> > > products. I didn't receive that much input so I thought I would ask
> > > again. Here are the requirments. I have a list of 2000 urls that all
> > > have Cisco in its domain name.
> > > (ex. http://www.soldbycisco.net
> > > http://www.ciscoindia.net
> > > http://www.ciscobootcamp.net
> > > http://www.cisco-guy.net
> > >
> > > and I want to scrape through them and determine which websites are
> > > selling new cisco products, I'm actually looking for around 20 or so
> > > products (ex. WIC-1T, NM-4E, WS-G2950-24). One idea I was given was to
> > > split the pages into ones with forms and those without forms. Those
> > > without forms probably wont have anything for sale so I can eliminate
> > > those. But then I really don't know how to handle after that. Does
> > > anyone have a different/better approach? Any help would be appreciate=
d.
> > > --
> > > Posted via http://www.ruby-forum.com/ .
> > >
> > >
> > Not to make your problem worse but you will need to differentiate betwe=
en
> > new and used equipment too.
> >
> > --
> > "Hey brother Christian with your high and mighty errand, Your actions
> > speak so loud, I can't hear a word you're saying."
> >
> > -Greg Graffin (Bad Religion)
>=20
>=20
> I don't remember who but someone suggested using Froogle and parsing that
> output.  Froogle and a few other sites like Pricewatch might be a far less
> complicated approach, you won't find all of them but then again I don't
> think you can possibly find everything anyway.
>=20
> --=20
> "Hey brother Christian with your high and mighty errand, Your actions spe=
ak
> so loud, I can't hear a word you're saying."
>=20
> -Greg Graffin (Bad Religion)

That was me. Seems to me you shouldn't parse froogle so much as just use it.
Writing a script is a lot more work and won't get you what you want; froogle
will.

=2D-=20
Konrad Meyer <konrad@tylerc.org> http://konrad.sobertillnoon.com/

--nextPart2034462.32gXqZMtcQ
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQBG8WaaCHB0oCiR2cwRAoxRAJ0YfUzjSQTl3uY5425bHwB+FUFbmwCdG1yO
lorZNLwuPVwewROHM8c3u0c=
=EJGr
-----END PGP SIGNATURE-----

--nextPart2034462.32gXqZMtcQ--