From: Aaron Patterson <aaron_patterson@...>
Date: 2006-09-13T01:38:21+09:00
Subject: Re: Fetching an URL using cookies

On Tue, Sep 12, 2006 at 06:51:40PM +0900, Zouplaz wrote:
> I'm trying to fetch an url which needs several cookies to be set in 
> order to properly return a result.
> 
> I've found a page in the website from which I can get the session 
> cookies (instead of posting cookies set by myself I prefer use the ones 
> coming from the server)
> 
> So,
> 
> def http_get(url, url_before = nil)
>     headers = Hash.new()
>     headers['User-agent'] = "Mozilla/4.0 (compatible; MSIE 6.0; Windows 
> NT 5.1)"
>     unless url_before.nil?
>       response = @http.get(url_before)
>       cookies = response.response['set-cookie']
>       headers['Cookie'] = cookies
>     end
>     response = @http.get(url, headers)
>     raise "url #{url} not accessible on host #{@host}:#{@port} - code 
> #{response.code}" if not ['200','302'].include?(response.code)
>     response.body
> end
> 
> 
> The problem is that I'm not sure it the way I repost the cookies is 
> right or not. The cookies retrieved by the unless block ARE OK but when 
> the second @http.get occurs, the remote web server ignore them and send 
> a redirect to a default page.
[snip]

Why write all this yourself?  WWW::Mechanize will handle storing and
sending cookies for you.  Then you can concentrate on getting the data
from the web page.

  http://mechanize.rubyforge.org/

You can even set a custom user agent string!  Hope that helps.
-- 
Aaron Patterson
http://tenderlovemaking.com/