From: Eric Hodel <drbrain@...7.net>
Date: 2002-11-08T17:13:35+09:00
Subject: Re: IE leech


--9A1A73/U17WN0PFw
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Andi Weiss (miner@arcor.de) wrote:

> Hi out there,
> does anybody know how to leech all the links from a previously opened IE
> Window. I want to do this with IE not with NET/HTTP. Would be nice if any=
one
> can help me.

This is very easy with xmlscan (htmlscan) and net/http.  Assuming you
save the following as 'leech.rb', do ./leech.rb http://path/to/page

You can create your own file-naming method if you want and pass it as a
method object into Leech.

#!/usr/bin/env ruby -w

require 'uri'
require 'net/http'

require 'htmlscan' # part of xmlscan

class Leech < HTMLScanner
  def initialize(url, filename_fun =3D nil)
    super(*[])
    @base =3D URI.parse(url).normalize unless url.kind_of? URI

    @filename_fun =3D (filename_fun || self.method(:default_filename_fun))
  end

  def leech
    parse(fetch(@base).body)
  end

  private

  def default_filename_fun(url)
    url.path =3D~ /\/([^\/]+)$/

    filename =3D $1 ? $1 : ''

    return "#{url.host}-#{filename}"
  end

  def cleanup_url(base, link)
    link =3D URI.parse(link).normalize unless link.kind_of? URI

    return link if link.scheme =3D=3D 'mailto' or not link.relative?

    if link.relative? && link.path[0].chr !=3D '/'
      path_ary =3D base.path.split('/')
      link.path =3D path_ary[0...path_ary.length-1].join('/') + '/' + link.=
path
    end

    link.scheme =3D base.scheme unless link.scheme
    link.host =3D base.host unless link.host
    link.port =3D base.port unless link.port

    return link
  end

  def fetch(url)
    response =3D nil
    url =3D cleanup_url(@base, url)

    case url.scheme
    when 'http'
      begin
        url.normalize!

        Net::HTTP.start(url.host, url.port) { |http|
          response, =3D http.get(url.path)
        }

      rescue Net::ProtoRetriableError =3D> err
        url =3D URI.parse(err.response['location'])
        retry if url.host

      end
    else
      # implement other handlers here
    end

    return response
  end

  def on_stag(element, attrs)
    return unless element =3D=3D 'a'
    page =3D nil
    url =3D cleanup_url(@base, attrs['href'])

    $stderr.puts "fetching: #{url}" if $DEBUG

    File.open(@filename_fun.call(url), 'w') { |fp|
      begin
        page =3D fetch(url)
        fp << page.body if page
      rescue Exception =3D> err
        $stderr.puts "Caught #{err} while fetching #{url}"
      end
    }
  end
end

if __FILE__ =3D=3D $0

  Leech.new(ARGV.shift).leech

end

--=20
Eric Hodel - drbrain@segment7.net - http://segment7.net
All messages signed with fingerprint:
FEC2 57F1 D465 EB15 5D6E  7C11 332A 551C 796C 9F04


--9A1A73/U17WN0PFw
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.0 (FreeBSD)

iD8DBQE9y3ItMypVHHlsnwQRAt3NAKCIDJReIhy1+MHoxX1Pur6R9jovzACffRWF
JH256p9OwFWYIlep7oaq2jk=
=Gysv
-----END PGP SIGNATURE-----

--9A1A73/U17WN0PFw--