From: Daniel Choi <dhchoi@...>
Date: 2008-09-24T09:15:10+09:00
Subject: Re: open-uri fetches outdated content vs. curl


I used net/http to do the same thing, but this time I printed out the
redirect locations. The result is very interesting. If it don't set
the "User-Agent" header, it get redirected to one proxy -- the one
with outdated content. If I set the "User-Agent" header to "Mozilla/
5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/XX (KHTML, like
Gecko) Safari/YY" (faking Apple Safari), I get redirected to another
proxy, with the up to date content.

I didn't know that servers redirected requests to bad or good proxies
depending on what the User Agent header is. But this seems to be the
case here.