From: Gavin Kistner <gavin@...>
Date: 2005-09-15T23:00:11+09:00
Subject: Re: REXML screen scraping questions

On Sep 14, 2005, at 10:51 PM, daz wrote:
> If you just want to search the data with regexen, you
> might make a method which yields the data to its block
> at the point where it's been collated.
> (Post back if you need any help with that)

Following is a sample of what I do for screen scraping - net::http  
and regex only. Just look for an indicative message on the screen,  
abstract it appropriately, and use it as an anchor for the data you  
want.

The following undocumented script hammers the WUnderground.com server  
to get min/max/average temperatures for a given city (airport code)  
for a given date range (optionally across many years). I used it (and  
Excel) to create
http://phrogz.net/tmp/BoulderTemperatures_LateSeptember.pdf
and
http://phrogz.net/tmp/CopperTemperatures_LateSeptember.pdf
(Trying to give a bunch of family members coming into town for a  
wedding a feel for the potential temperature ranges, and the  
variations possible within a given 5-day period.)


require 'net/http'
require 'date'

def get_temperatures( airport_code, date_range, year_range=nil )
   if year_range
     d1 = date_range.first
     d2 = date_range.last
     dates = year_range.collect { |year|
       ( Date.new( year, d1.mon, d1.day )..Date.new( year, d2.mon,  
d2.day ) ).to_a
     }.flatten
   else
     dates = date_range.to_a
   end

   Net::HTTP.start('www.wunderground.com', 80) { |http|
     dates.collect { |date|
       url = "/history/airport/#{airport_code}/#{date.year}/# 
{date.mon}/#{date.day}/DailyHistory.html"
       html = http.get( url ).body
       stats = { :min=>'Min', :max=>'Max', :mean=>'Mean' }
       stats.each { |key,val|
         if str = html[ %r{#{val}(?: Temp(?:erature))?</td>.+?<td[^>] 
*>(.+?)</td>}im , 1 ]
           temp = str[ %r{(\d+).+?\&#176;F}i, 1 ]
           stats[ key ] = temp ? temp.to_f : nil
         else
           stats[ key ] = nil
         end
       }
       if stats[ :min ] && stats[ :max ]
         DayTemperature.new( date, stats[ :min ], stats[ :max ], stats 
[ :mean ] )
       end
     }.compact
   }
end

class DayTemperature
   attr_accessor :date, :min, :max, :mean
   def initialize( date, min, max, mean=nil )
     @date = date
     @min = min
     @max = max
     @mean = mean
   end
   def to_s
     "%s\t%3i\t%3i\t%4i" % [ "#{@date.year}-#{@date.mon}-# 
{@date.day}", @max, @mean, @min ]
   end
end

temps = get_temperatures( 'KBJC', Date.new( 2000, 8, 15 )..Date.new 
( 2000, 9, 15 ), 1990..2005 )
puts temps.join( "\n" )