From: Josef 'Jupp' Schugt <jupp@...>
Date: 2003-06-08T02:44:50+09:00
Subject: Re: Pkg for getting web pages

Saluton!

* Gennady; 2003-06-06, 20:14 UTC:
> But anyway, Ruby is great for interfacing to other programs and if
> something is being done well by some specialized tool it is worth
> using it from Ruby instead of reinventing a bicycle in Ruby ;-),
> especially when the tool is readily available.

The question is still open to discussion: *Which* tool. Many people
know w3 but that is not the only tool that fits. There are others,
most notably *curl* - it comes with a library (for C IIRC).

*curl *
is a client to get documents/files from or send documents to a
server, using any of the supported protocols (HTTP, HTTPS, FTP,
GOPHER, DICT, TELNET, LDAP or FILE). The command is designed to work
without user interaction or any kind of interactivity. curl offers a
busload of useful tricks like proxy support, user authentication, ftp
upload, HTTP post, SSL (https:) connections, cookies, file transfer
resume and more.

http://curl.haxx.se/

*snarf*
can transfer files through the http, gopher, finger, and ftp
protocols without user interaction. It is small and fast and can
force active FTP (default is passive), resume downloading a partially
transferred file and spoof MSIE us well as Navigator user-agent
string and checks the SNARF_PROXY, FTP_PROXY, GOPHER_PROXY,
HTTP_PROXY, and PROXY environment variables.

http://www.xach.com/snarf/index.html


Gis,

Josef 'Jupp' Schugt