From: Xavier Del Castillo <xdelcastillo@...>
Date: 2011-02-18T13:19:30+09:00
Subject: Problem: open-uri blocking redirection from http to https

Hello,

I was working on a small script to verify the presence of an element 
through a list of URL, some of these URLs have a redirections from http to 
https, when the script crawled into them I got the following error:

/usr/lib/ruby/1.9.1/open-uri.rb:216:in `open_loop': redirection forbidden: 
http://beta.carsdirect.com/auto-loans/finance-app -> 
https://beta.carsdirect.com/auto-loans/finance-app (RuntimeError)

I understand that this is intentional, as per the comments on open-uri.rb

     # This test is intended to forbid a redirection from http://... to
     # file:///etc/passwd.
     # https to http redirect is also forbidden intentionally.
     # It avoids sending secure cookie or referer by non-secure HTTP 
protocol.
     # (RFC 2109 4.3.1, RFC 2965 3.3, RFC 2616 15.1.3)
     # However this is ad hoc.  It should be extensible/configurable.

This mentions that "https to http" redirects are forbidden intentionally, 
but redirections from "http to https" are also blocked.

Is there a way to override this security check? currently I had to change the
following line in the library to allow "http to https" re-directions:

(/\A(?:http|ftp)\z/i =~ uri1.scheme && /\A(?:http|ftp)\z/i =~ uri2.scheme)

to

(/\A(?:http|ftp|https)\z/i =~ uri1.scheme && /\A(?:http|ftp|https)\z/i =~ 
uri2.scheme)

Thanks,
Xavi