From: Zouplaz <user@...>
Date: 2006-09-12T20:50:42+09:00
Subject: Re: Fetching an URL using cookies

le 12/09/2006 13:09, William Crawford nous a dit:
> 
> Another thought, however.  Perhaps the page has additional requirements 
> that you haven't met.  Cookies that don't exist on the other page, but 
> were set at login or somewhere else.  Headers that you aren't sending 
> and it expects.  A specific refering page.  (Or something else I've 
> momentarily forgotten.)
> 

I don't why but suddenly it worked... I presumed I've missed something 
somewhere..

Now, I've rewritten the code and I use a "write-once" cookie mechanism 
which is generic for every "scrapping" class that I use - It's 
sufficient for now

def http_get(url)
     headers = Hash.new()
     headers['User-agent'] = "Mozilla/4.0 (compatible; MSIE 6.0; Windows 
NT 5.1)"
     headers['Cookie'] = @cookies unless @cookies.nil?
     response = @http.get(url, headers)
     raise "url #{url} no access on host #{@host}:#{@port} - code 
#{response.code}" if not ['200','302'].include?(response.code)
     @cookies = response.response['set-cookie'] if @cookies.nil?
     response.body
end

Just for my own education, could this code be rewriten in a more elegant 
way ?


Thanks