From: Michael Libby <michael.c.libby@...>
Date: 2008-11-06T02:54:56+09:00
Subject: Re: What's the Best Way to Mimic an HTTP Request?

On Wed, Nov 5, 2008 at 11:08 AM, Daniel Miessler <daniel@dmiessler.com> wrote:
> Thanks, much, Michael. Unfortunately I'm not quite tracking on why that
> was necessary. It just seems a bit elaborate given what I thought was a
> simple problem.
>
> But I totally appreciate it...I just wish it were something simpler.

The site you're hitting makes heavy use of redirects (and not really
for their intended purpose). What this means is that you submit your
request for a given URL and the server responds with a redirect and a
new URL. If you are working in a browser, your browser automatically
requests that URL, and the server again responds with a redirect and a
new URL. Again, a web browser handles requesting that next URL
automatically. This URL is the actual results page with the data you
want. It's the web site making you jump through hoops to get where you
want to go.

Net::HTTP does not have a built in facility for following redirects
the way your browser does. So you have to write code to follow
redirects by submitting new requests until you get to one that is not
a redirect, which is what the fetch() method from the Net::HTTP
example does.

-Michael