From: Michael Libby <michael.c.libby@...>
Date: 2008-11-06T01:56:57+09:00
Subject: Re: What's the Best Way to Mimic an HTTP Request?

On Wed, Nov 5, 2008 at 10:24 AM, Daniel Miessler <daniel@dmiessler.com> wrote:

> The problem is that I keep getting a redirect
> (#<Net::HTTPMovedPermanently:0xb7c35ffc>), which doesn't happen when I
> make the request from a regular browser.

That site makes heavy use of redirects. Watch closely while running
queries or check your browser history.

> So I sniffed the regular request with wireshark, and a browser sends a
> bunch of additional headers when it makes the request. Cookies,
> referrer, etc.
>
> Are any of these headers more necessary than others, and is there a
> preferred way to send the headers using Ruby?

Headers probably have no effect here.

What you probably want is code like this:

    require 'net/http'
    require 'uri'

    def fetch(uri_str, limit = 10)
      # You should choose better exception.
      raise ArgumentError, 'HTTP redirect too deep' if limit == 0

      response = Net::HTTP.get_response(URI.parse(uri_str))
      case response
      when Net::HTTPSuccess     then response
      when Net::HTTPRedirection then fetch(response['location'], limit - 1)
      else
        response.error!
      end
    end

    resp = fetch('http://www.ruby-lang.org')
    puts resp.body

(from http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/index.html --
"Following Redirection")

regards,
Michael Libby