From: Florian Ebeling <florian.ebeling@...>
Date: 2008-01-23T20:46:28+09:00
Subject: Re: Parsing HTML / following links etc

> I've was pushed towards ruby from by a friend. I'm used to the usual
> shell scripting and was told this will be much more powerful /
> gracefully / easier.
>
> It does look all very exciting but I'm having a problem looking for  
> the
> easiest way to implement something quite simple.
>
> I'm used to using wget to crawl some of my sites to a certain layer.
>
> wget http://www.digg.com -r -l 2 will digg down two layers from the
> front page (follow the links).
>
> I can't find an easy way of doing this. Open-uri doesn't seem to
> supporting recursive following. I've looked at pulling down the HTML  
> and
> parsing it back to open-uri but there doesn't seem to be an easy way  
> of
> doing this.
>
> Another thing I would like to do it pull down other elements from the
> html such as  images so I explored html-parsing but they all seemed to
> be geared towards manipulation rather than downloading information for
> manipulation later :(

you might want to try the hpricot (!) gem, but that covers only
html parsing. then you use the standard html client and pull
the documents you fancy.

a regular http client library does not typically include getting
all referenced objects, this is rather a higher-level 'application'  
feature.


>
>
> Thanks for your help if you can
>
> Dan
> -- 
> Posted via http://www.ruby-forum.com/.
>