From: Wyss Clemens <WYS@...>
Date: 2002-01-16T00:43:20+09:00
Subject: RE: grabbing stuff from web pages

a) the following snippet is from the "Programming Ruby" book (just before
http://www.rubycentral.com/book/lib_network.html#Net::HTTP.new):
"
The code below illustrates the handling of an HTTP status 301, a redirect.
It uses Tomoyuki Kosimizu's URI package, available in the RAA. 

h = Net::HTTP.new(ARGV[0] || 'www.ruby-lang.org', 80)
url = ARGV[1] || '/'

begin
  resp, data = h.get(url, nil) { |a| }
rescue Net::ProtoRetriableError => detail
  head = detail.data

  if head.code == "301"
    uri = URI.create(head['location'])

    host = uri['host']
    url  = uri['path']
    port = uri['port']

    h.finish
    h = Net::HTTP.new(host, port)

    retry
  end
end
"

The 302 error also seems to provide "head['location']" which can be used for
the "next try". I have not found the herein mentioned uri-module though.

Clemens

> -----Original Message-----
> From: Ron Jeffries [mailto:ronjeffries@REMOVEacm.org]
> Sent: Dienstag, 15. Januar 2002 15:33
> To: ruby-talk@ruby-lang.org
> Subject: grabbing stuff from web pages
> 
> 
> Part of my web site has recommended books. I use the cover jpegs from
> Amazon as part of my page. The rule with Amazon is that you have to
> download the picture and put it on your site, not link to theirs. 
> 
> So I was starting out with Dave and Andy's example:
> 
> require 'net/http'
> 
> h = Net::HTTP.new('www.xprogramming.com', 80)
> resp, data = h.get('/index.htm', nil)
> if resp.message == "OK"
> 	puts data
>   data.scan(/<img src="(.*?)"/) { |x| puts x }
> end
> 
>  figuring that I could then come up with some way to get and save the
> image ... I remember doing it once before as an experiment, though I
> can't find the code now.
> 
> But when you set up to hook to the amazon book page, which can be
> accessed this way in a web browser
> 
> http://www.amazon.com/exec/obidos/ASIN/0201708426/
> 
> so that the code looks like this:
> 
> require 'net/http'
> 
> h = Net::HTTP.new('www.amazon.com', 80)
> resp, data = h.get('/exec/obidos/ASIN/0201708426/', nil)
> if resp.message == "OK"
> 	puts data
>   data.scan(/<img src="(.*?)"/) { |x| puts x }
> end
> 
>  You get an error 302. I suppose that it's all that magic redirection
> that they're using to get you to the real page, but I don't 
> know enough
> to be sure.
> 
> So I'd appreciate help on
> 
>   a) getting the right page address given that I only know the number
> that comes after ASIN, and
>   b) saving the jpeg once I get its src address.
> 
> Thanks!
> 
> Ronald E Jeffries
> http://www.XProgramming.com
> http://www.objectmentor.com
> I'm giving the best advice I have. You get to decide whether 
> it's true for you.
>