From: rking <rking@...>
Date: 2007-08-16T04:24:59+09:00
Subject: Re: HTTP headers and source

Here's my suggestion:

Firefox + LiveHTTPHeaders - http://livehttpheaders.mozdev.org/installation.html

LHH shows all HTTP chatter, so there's nothing that a server can see
that you can't. From there it's just a matter of imitating the headers
with Net::HTTP.

Remember, though, that you have some vague sort of obligation to
maintain netiquette. If a server rejects automated requests, they may
have a good reason to, and you're going against their wishes to mimic
a real browser. I doubt the Feds are going to come kicking your door
in over it, but it's still worth trying to be respectful.

Google, for example, has an API that they encourage for automated
usage. Here are some details: http://code.google.com/apis/soapsearch/api_terms.html

-rking