From: Daniel Harple <dharple@...>
Date: 2006-03-30T00:28:07+09:00
Subject: Re: Waiter, there's a noob in my soup!

On Mar 29, 2006, at 1:54 PM, Dimitri Aivaliotis wrote:

> Perl??
>
> WWW::Mechanize is a Ruby library

It is a port of a Perl library.

On 3/26/06, Jeff Pritchard <jp@jeffpritchard.com> wrote:

> I was wondering if anyone could point me to some example code that is
> using RubyfulSoup to parse a sitemap to get links to all the pages on
> that site and request each page and grab things from it.

Here's an example that finds all the links:

require 'net/http'
require 'rubygems'
require 'rubyful_soup'

uri = URI.parse('http://ruby-lang.org/en/')
soup = BeautifulSoup.new(
   Net::HTTP.get_response(uri).body
)

soup.find_all('a').
   map { |link| link['href'] }.
   reject { |link| link.nil? }


You will get links with a host, and some with just relative/absolute  
paths. You can do (I didn't test this):

link_uri = URI.parse('./20050820.html')
link_uri.host = uri.host unless link_uri.host

and fetch them. You will also want to check for redirections and  
other errors. See the [Net::HTTP docs][1] for fetching a page when  
redirected.

[1]: http://www.ruby-doc.org/stdlib/libdoc/net/http/rdoc/

Regards,
-- Daniel