From: Daniel Choi <dhchoi@...>
Date: 2008-09-18T22:12:28+09:00
Subject: Re: open-uri fetches outdated content vs. curl

On Sep 18, 2:26 am, Robert Klemme <shortcut...@googlemail.com> wrote:
> On 18.09.2008 02:05, Daniel Choi wrote:
>
>
>
> > Try running the following program:
>
> > ================
> > require 'open-uri'
>
> > feed_url = "http://www.slate.com/rss"
>
> > result1 = open(feed_url).read
> > puts "Saving result1.xml:"
> > File.open("result1.xml", "w") {|f| f.write(result1)}
>
> > result2 = `curl -L #{feed_url}`
> > puts "Saving result2.xml:"
> > File.open("result2.xml", "w") {|f| f.write(result2)}
>
> > command = "diff result1.xml result2.xml"
> > puts system(command)
> > ================
>
> > result1 should be identical to result2, but it turns out that the feed
> > thatopen-urifetches is outdated content (by over a month), while the
> > feed that curl fetches is up-to-date. Can anyone please explain what
> > is going on?
>
> Reasons I can think of:
>
> i) Both approaches use different paths to the server, namely a different
> (or no) proxy.
>
> ii) There is something in the request that makes the server send
> different data.
>
> Can you try to obtain HTTP headers from both approaches?  That might
> clear up a few things.  Also, on Unix type systems check for environment
> variables and ~/.xyzrc files which might affect proxy settings.
>
> Another good idea might be to try a different tool, e.g. a web browser,
> to see what that turns up.
>
> Kind regards
>
>         robert


Thanks for these suggestions. The problem actually just cleared itself
up, after several days where the open-uri fetch was getting outdated
content. I think it was a problem is upstream proxies. I'll try to
look at the headers out of curiosity.