From: Niklas Frykholm <niklas@...>
Date: 2001-11-01T06:01:41+09:00
Subject: [ruby-talk:24034] ANN: WebFetcher

I've uploaded a new module to the RAA. It is available at

    http://www.acc.umu.se/~r2d2/programming/ruby/webfetcher/

Comments are most welcome.

----------------------------------------------------------------------

    WebFetcher

    Niklas Frykholm (niklas@kagi.com)
    0.5.0, Sunday 21 October 2001


WHAT IS IT?

    WebFetcher is a module designed to facilitate fetching large
    ammounts of documents and images from the web. With WebFetcher it
    is easy to do such things as downloading all the images on a page.
    You can also do more complex tasks, such as fetching all the pages
    linked to by a certain page and all the images on those pages. You
    can save the documents in a tree structure mirroring the server
    layout or flat in a single directory. In either case, WebFetcher
    can translate the links on the pages so that their internal
    relationships are preserved.

    It is released under the same license as Ruby. If you have any
    questions, contact me at niklas@kagi.com.


SAMPLE USAGE

    Downloads all the pages in the Programming Ruby book and saves
    them to your hard drive.

        require 'webfetcher'
        include WebFetcher

        book = Page.url('http://www.rubycentral.com/book/')
        pages = book.recurse(10)
        pages.save('programming-ruby')


INSTALLATION

    Run

        ruby install.rb

    or copy the file webfetcher.rb to the desired destination
    manually. You may need to be root to install the program.


MORE

    More documentation is found in the file webfetcher.html. In
    examples.rb you find some sample scripts.

----------------------------------------------------------------------

// Niklas