From: Ta St <stockwet@...>
Date: 2008-11-20T04:48:58+09:00
Subject: Help with Screen Scraper!

Hi,

I need some help with a simple screen scraper app. Here's how it works:

1) I open a connection to a url
2) I look for a specific string
3) When I get the string, I do something with it
4) Important: I'm polling a url looking for data changes.

Problem:
Everything works fine, until the website is updated with new content. At
that point, my code throws an "private method `scan' called for
nil:NilClass (NoMethodError)"

I can get the code to work using
begin
....
rescue
  retry
end

But, the problem is that it causes the connection to close and re-open.
This causes a further delay in grabbing the data.

Is there a way to handle this? Again, the polling works fine until the
server updates the new page that I'm scanning. At that time, the error
is thrown and I either have to re-open a connection or fail on error. My
Java application seems to be handling this more gracefully, but, Ruby is
faster on the polling. Thanks for any insight.

Regards.


******************
require 'open-uri'
  pval = "14"
  url="http://www.someurl.com"
  stop = false

    while stop == false
        #Create timestamp
        sts="S " << Time.now.strftime("%m/%d/%y %H %M
%S.")<<Time.now.usec.to_s
        puts sts
        lines=open(url).read.split("\n")
      #Get Lines of data and find the SearchValue
      tdate=lines.detect {|line| line =~ /SearchValue/}.scan(/\d+/)
      #Compare SearchValue to pval
      if tdate[1] != pdate

        puts tdate[0]
        ts="F "<<Time.now.strftime("%m/%d/%y %H %M
%S.")<<Time.now.usec.to_s
      else
        puts ".."
      end
    end
-- 
Posted via http://www.ruby-forum.com/.