From: jbritt@...
Date: 2003-02-13T15:51:16+09:00
Subject: Re: Web Automation


>      I can almost figure out how to do this. I know how to
>      use "require CGI" and I understand how to use eruby.
>      I suspect that this is not really a CGI problem. Maybe
>      it is more akin to downloading mail using pop.


Not sure where cgi would come in, as this appears to be a client-side application.

It sounds like you want to make an HTTP request to fetch a page, parse the returned (possibly malformed) HTML (and possibly cookies,
too), create a new HTTP POST or GET, and submit the new HTTP request to proceed to the next page.

>
>      Basically, I can find examples which use CGI to field
>      responses to request s sent to a another url. But
>      I have not seen a case of an automated interaction
>      with a url. It seems an obvious thing to have. Am I
>      missing something? Can some point me to examples of
>      scripted queries?


While I can't point to any examples offhand, I believe this has come up before on this list, so you can perhaps check the archives
at Google groups, or maybe somebody else can provide code/reference.

A simple example of a basic fetch-n-parse is dictionary.rb.  It shows how to fetch a web page and parse out specific info,  but does
not create a new request based on any content:

#!/usr/local/bin/ruby
#--------------------------------------------------------------
# Script to fetch a definition for dictionaty.com, strip the HTML, and display
# the results in the cosole window.  Original code written by Ralph Mason:
# http://www.geocities.com/masonralph/ruby.html
#--------------------------------------------------------------
require 'net/http'

class WebDictionary

  def look_up( word )
    site = "www.dictionary.com"
    page =  "/cgi-bin/dict.pl?term=#{word}"

    a = Net::HTTP.get( site , page  )

    res = ""
    if ( a=~/No entry found for/ )
      puts "No entry found for '#{word}' - Suggestions:\n\n"

      while $' =~/<a href="\/cgi-bin\/dict.pl\?term=\w+">(\w+)/
        res << $1
      end
    else
      while a =~ /<!-- resultItemStart -->/
      $' =~ /<!-- resultItemEnd -->/
      a=$'

      res <<  $`.gsub(/<.*?>/m){ |i|
        case i
          when "<b>"
            ""
         when "</b>"
          "\n"
         when "<p>"
          ""
         else
           ""
          end
        }
      end
    end
  res
  end

end

if __FILE__ == $0
  if  ARGV.size == 0
    puts  "Usage: #{$0} <some word> "
    exit
  end
  puts WebDictionary.new().look_up( ARGV[0] )
end